Test Failure Analysis With AI-Critical To DevOps

Software testing is an extremely important process to ensure that all quality requirements are met before any application is released in the market. With DevOps, the release cycles have become shorter, but expectations are still the same, i.e. high-quality end product. But, it is a fact that the tests fail sometimes and testing may hit a roadblock. Usually, a knee-jerk reaction is to pass the buck to the development team for fixing the issue. However, the ideal approach is to understand the root cause of the failure.

Test failure analysis is a systematic process to analyze and identify the underlying causes for a failed test and to prevent them from occurring again.

Test failure analysis is an important exercise that should be conducted for all critical test failures. Continuous testing generates a lot of data in terms of test results which can be utilized for test failure analysis which can help in sifting, identifying the cause, and fixing the issue. It helps in improving the overall QA Ops process by helping in managing and addressing the root causes of any issue.

Key elements of Test failure analysis

Test failure analysis is a good quality control measure that provides you with insights on what exactly went wrong, at what point, and understand the reason behind it. This helps the teams in improving the testing strategy based on the findings by identifying whether the problem was a testing issue or a flaw in design and development.

Test failure analysis is tailored as per the organization’s QA process and the application under test. However, certain key elements are common across the board and are mentioned below.

Root cause analysis

Root cause analysis of the defect holds the key to understanding what exactly went wrong. 

Was it because of issues with the software or there was a problem with the test itself? 

Once this basic question is answered, it is time to trace the origin of the problem. The problem could be at any point in the whole development cycle: requirement gathering and understanding, or design and development, or in some cases environmental issues while testing. Once the origin is verified, the teams need to work on it for corrective action.

If there is an issue with the test, then the whole test plan needs to be re-examined and corrected before further testing can take place.

Addressing false failures

False failures are the bane of test automation. These are the cases when the automation system should have been able to correctly identify whether it is a Pass or Fail. Instead, it is incorrectly specified as a Fail. We have covered this in detail in our blog, click here to read more. False failures can lead to unnecessary delays because every failed test case  needs to be triaged and based on its priority needs to be addressed.

Detailed Reporting

Good reporting aids the teams to

• Understand test results
• Saves time in going through a huge amount of data that was generated while testing
• Helps in distinguishing between actual defects, script errors, feature change and the noise.
• Helps in addressing the key question of how many known defects are in the build, and can the shipping be delayed, while the test team works on classifying the automation failures as actual bugs or a re-test scenario.

Addressing the root cause

Once the actual defects have been identified and traced to their origins, the concerned team needs to first segregate the ones that need immediate attention and address them as per the priorities. Depending on the magnitude of the change, the re-test can be done within the same test cycle or a new sprint can be conducted just for testing.

Acing the test failure analysis with Webomates

In a typical test cycle, on average 30%-40% of tests fail. Though 93% of these failures are NOT related to defects and are false failures, they need to be triaged by the testing team to understand the underlying reason for these failures. Once the impacted test scripts are fixed, then the automated test run can be deemed a success with either true fails(i.e. defects) or true passes(scripts passed).

Webomates provides solutions to handle such scenarios with ease.

AI Test Package Analyzer

Webomates’ AI Test Package Analyzer identifies all the test cases which are impacted due to a feature change, and traces them from impacted user stories/epics/requirements to identify the impacted test cases and test scripts. This aids in understanding the root cause of the failed test case, whether it is due to feature change or any other reason, and appropriate action can be taken.

AI defect predictor

Our ingenious AI defect predictor can identify false positives with 99% accuracy. For example, 300 test cases are executed and results are reported as stated in the figure below:

Out of 300, 30% of test cases failed. Out of these failed test cases, 92% were false positives. Those are the ones that need to be addressed. Defect predictor can easily identify multiple root causes for such failures, like

• Locator ID changes 
• Timeout errors 
• Test Data changes 
• Feature changes and additions

Prior to building these interlinked AI systems Webomates measured the average time that it took for following tasks with respect to the 90 failed test cases 

• Identifying false positives
• Identifying the root causes for the failure
• The healing process to fix the test case and the test script

All these tasks would typically take 12 hours. Today, with the current generation of the AI systems this has dropped to 2 hours only.

Detailed reporting

Test reports and analysis are critical components of the testing process. They aid in improving the quality and accelerate the releases.

Webomates has a comprehensive process when it comes to defect triaging and reporting. The defect report contains:

1. Defect Summary
2. Steps for replicating the bug
3. Video of actual bug instance
4. Priority suggested for the defect
5. Test cases mapped to the defect
6. Additional notes

Our unique process presents a walkthrough video of an “actual instance of bug” happening so that it is easier for the development and testing teams to understand what exactly transpired during testing. It guides them in reproducing a defect.

The triaging reports are shared across the board with all the stakeholders giving them a detailed insight into what can be improved in terms of quality strategies.

AI Defect Creator

AI Defect creator automates the process of gathering the data for a defect, including identifying the key 20 seconds of video that shows the defect occurring. The process of creating a “good” defect is time consuming as described in this article that talks about defect triaging in detail. However, there is a dramatic reduction in the effort with the  AI Defect Creator. 

Webomates CQ has an option of executing mini test suites that saves effort and a significant number of man-hours.

Webomates provides intelligent automation( AI Testing)  solutions with intelligent analytics. It leverages the power of data processing, analysis, reasoning, and machine learning to provide an end-to-end testing solution for your business. We have a capable team of analysts and engineers to aid you along with the power of intelligent automation.

If this has piqued your interest and you want to know more, then please click here and schedule a demo, or reach out to us at info@webomates.com. If you liked this blog, then please like/follow us Webomates or Aseem.

Tags: AI Testing, Artificial Intelligence, Defect prediction, Defect Triage, DevOps, Intelligent automation, Test Automation