Artificial Intelligence and QA: Hype From Reality

Artificial Intelligence (AI) is at the peak of inflated expectations in the Hype Cycle.  There is a law called Amara’s law that states “We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.“

I think most people in AI would agree that we are currently over-estimating the effect of AI on technology. Looking through the lens of AI in the Quality Assurance space I thought it might be beneficial to look at what areas of Quality Assurance seem ripe for automation (the business type of automation) via AI.

In the sections below I review various areas in Test case based execution and then provide my personal estimate of the impact that using AI would have in the near term. I quantify this prediction with two probabilities:

1. Assistance:. The AI system will help the QA tester or QA Automation engineer.
   1. A rating of “1” is no assistance
   2. While a “10” is a high amounts of assistance – such as a 50% reduction in work
2. Automation:. The AI system will do all the work.
   1. A rating of “1” means there is no automation and
   2. While aA “5” means 50% of the work will be fully automated (that is, no human assistance is needed)
3. This forecasts is for technologies that are available today and will be widely available in the next 2 years or less

Execution of Test Cases

There are several companies that are focused on creating software automation systems based on Artificial Intelligence. These systems typically reduce effort with the intent to eventually completely automate the following capabilities:

1. Test case creation (Text based test case descriptions)
2. Test case script creation (automation scripts such as Selenium Java code)
3. Self-healing scripts that modify themselves when the UI changes

Test Case Creation

Test case creation is really challenging. The AI based system needs to be able to analyze a UI that it is asked to target and identify a series of test cases that implement user behavior when using the system. Since in most User Interfaces, particularly ones with dynamic data, a very large number of test cases can be created, the next question becomes is to identify the test cases that are a priority and select the ones to actually execute. It also needs some level of domain expertise.

Assistance: 3/10           Automation: 1/10

Test Case Script Creation

Many companies have been focused in this area and rapid advancements are being achieved. Often times the focus here is to convert and regenerate test cases to test scripts. Or alternatively to re-code test cases based on execution or on user behavior and then to generate the execution scripts.

Assistance: 7/10           Automation: 4/10

Self-Healing Scripts

To me this category really falls in the area of rocket science. Self-healing scripts adapt to the changes that occur in the UI and re-generate the automation scripts. Such a system needs to be able to detect a variety of changes:

1. Changes in the UI that have caused scripts to break due to technical issues such as element Locator ID changes.
2. Changes in the functionality of the User Interface.
3. Self-healing must occur in minutes, that is minutes after getting access to the latest software build.

Combined, these two sets of requirements are a major technical challenge. In order to detect changes in functionality, complete (100%) script generation is a necessary precursor. And if the system is generating test scripts based on test cases then complete test case generation is needed as well too. The third requirement is that self healing must occur in minutes only compounds the technical challenge.

Assistance: 5/10           Automation: 1/10

Method of Execution of Test Cases

These are systems that determine which is the best method for executing a set of test cases. Execution systems could be:

1. Multiple different AI test automation systems
2. Multiple different software test automation systems
3. Multiple different crowdsourcing channels
4. Multiple different manual QA pools of talent
5. Multiple different pools of devices (for different browser/OS/Mobile platforms)

And combinations of each of the pools above determine the optimum execution based on quality, execution time, setup effort etc.

Assistance: 4/10           Automation: 2/10

Execution Output

After test case execution is complete only half the job is done. There is a lot more work that goes into analysis of test case execution and defect prediction.

Execution Analysis

Analysis of test case execution revolves around determining how many of the “fails” in a Pass/Fail report are true fails and how many are false fails. So what does that mean? For example, if the result of a test case execution is that the test case failed, a True Fail would be to determine that the test case has in reality failed. A False Failure would be, if on analysis, you determine that the cause of the failure is for example an automation error such as a Time-Out Error. Then, in reality, you have no idea whether the test case actually failed or passed. All you know for sure is that the Automation system failed the test and that failure is an automation failure. Unless you update your script, or manually verify the test case, you don’t know if the test case has in fact failed. Thus, this is a case of a False Failure. Automation, in particular, is notorious for the number of False Failures it tends to generate.

Similarly, a True Pass and False Pass need to be looked at more carefully – especially False Passes as they are incredibly important to identify should they ever occur. Fortunately, they tend to be rather infrequent.

Assistance: 4/10           Automation: 2/10

Defect Prediction

Defect prediction, on the other hand, centers around the probability for a True Fail or False Pass to translate into a new defect.

The key here is in satisfying the requirements of both words:

1. “new” – as in, not seen before.  
2. “defect” or bug – defined as an unexpected or unintended behavior.

Assistance: 3/10           Automation: 2/10

Table for Quick Lookup

Below is a table that summarizes the “scores” that I have assigned to various facets of QA with an AI focus.

Webomates is focusing on all of the above areas of AI in QA and is aggressively partnering with other leaders in this space. By using our TAAS (Testing as a Service) you automatically start using the best AI tools in the market!
If you are interested in learning more about Webomates’ CQ service please click here and schedule a demo or reach out to us at info@webomates.com.

Tags: Artificial Intelligence, Software Testing