Guide To Why Audio Testing Is Important

Audio testing is an integral part of media testing. The world of audio is much more than just some sound files. It also encompasses the soundwaves transmitted over a network, voice commands, AI voice assistants like Alexa, Siri, Cortana, etc. There is a lot of science involved and intense testing needs to be conducted behind the scenes, in order to render a crystal clear sound. 

The past few months have seen a meteoric rise in the usage of web conferencing platforms. Imagine a scenario, when there is an important topic being discussed in a video-conference and the voice quality is abysmal. Another case could be, misinterpretation or delay in the response to the voice commands while using AI-powered voice assistants. The whole idea of using these assistants is to save time, which is a precious commodity these days, and issues like these defeat the whole purpose. The entire media streaming experience can go for a toss if the audio is not in sync with the video. Audio testing proves useful in preventing such problems.

Metrics for Audio Testing

Audio quality testing involves checking the application objectively, as well as subjectively. This will become clearer once we take a look at the metrics that have to be tested for determining the audio quality. The figure below lists down the test metrics, followed by brief explanations in subsequent paragraphs.

• Audio fidelity
  Audio fidelity requires significant human involvement. It depends on the human ears’ capability of identifying sound lapses, distortion, echo, or any other obvious inconsistencies. Not all distortions/echoes can be deemed as bugs, hence it is important that the tester is fully aware of what kind of audio he/she is testing. One such example can be, when one audio app is playing and a sound related to another app starts in the background, like app notifications or ringers. How both sounds play together as output and whether it is a “good” output will depend on the judgment call made by the tester.

• Network Conditions
  Be it for any broadcasting service, or internet calling service, the audio quality is heavily dependent on the network conditions.
  Ideally, the audio streaming should be tested for all possible scenarios, like low/high bandwidth, peak-time data transfer, network congestion, etc. Few important metrics related to network conditions are mentioned below.

• Audio delay
  Audio delay refers to the time taken by the audio stream to reach the end-user. It depends on the network speed/performance. The unit of measurement for the audio delay is in seconds.

• Jitter
  Audio jitter is the variation in the delay of packets received over the network due to issues like congestion. In other words, it is any deviation from the periodicity of receiving audio signals over the network.
  Preferably, the delay between receiving each packet should be constant. In case of variations, the output sound is jittery. Jitter configurations can have custom settings as per the application under test.
• Packet loss
  Packet loss refers to a situation when one or more data packets transmitted over the network do not reach the intended destination.  Issues like network congestion, due to heavy traffic, may result in packet loss.
• Switching the networks
  In real-world scenarios, different users use different networks or Wi-Fi setups. There could be link drops and reconnections while switching between the networks. All such possibilities should be taken into account while testing. Emulators can be used to simulate such situations, however, the results may vary in real-world situations.

• VISQOL aka Virtual Speech Quality Objective Listener
  This is a signal-based metric that has been specifically designed to test quality issues associated with Voice over IP (VoIP) transmissions. 
  Multimedia conferencing applications like Google Hangouts and Skype are used worldwide on a variety of hardware, mostly on computers and mobiles. The quality of transmitted speech is tested via VISQOL, which compares the reference signal with the test signal. Its results are solely based on the difference between a clean and a distorted signal.
• POLQA aka Perceptual Objective Listening Quality Analysis 
  POLQA is the global standard for benchmarking the voice quality of fixed, mobile, and IP-based networks. It is widely used by 3G/4G/VoLTE and other networks that deliver HD voice quality services. It can assess speech with better accuracy than its predecessor PESQ.  
  It predicts speech quality by analyzing digital speech signals, by comparing the reference signal(source) and the degraded signal(receiver).

Issues with the traditional way of audio testing

Why we need to automate the audio testing process

Audio quality should undergo continuous testing for functionality, regression tests must be performed with every sprint, and performance/load tests should be an integral part of the testing process, just before the deployment. This calls for automating the tests wherever possible. Let us take a look at the top reasons to automate the audio testing process.

Minimizing human involvement
Human ears are not tuned to catch the full spectrum of sound frequencies, hence the non-audible ones may go untested, thus affecting the test results. Besides this, repetitive listening gets monotonous and the tester may falter. Also, the perceptions and judgments vary from person to person, leading to inconsistencies. Defining a testing yardstick becomes a challenging job.

Regression testing
Any changes to the base software module need to be tested for all possible aspects. For example, any changes made to an IVR system have to be verified, but human involvement in listening to the same clip can lead to fatigue, making it prone to lapses. In such cases, the automated process of using acoustic fingerprinting comes handy.

Increasing test coverage
There are high chances of missing a few test scenarios if audio testing is done only manually, especially when it comes to inaudible frequencies. Automating the tests and leveraging on AI can cover more scenarios, leading to better test coverage.

Performance testing
It is always prudent to conduct performance testing in the real-time scenario and not rely completely on the emulators. It should be done for all target platforms/devices. Performance can degenerate in real-time situations, so monitoring audio quality after deployment helps in gauging situations that could not be simulated during testing. Automating these steps will reduce the chances of error.

How Webomates can help you

A good testing tool should be able to comprehensively test above mentioned metrics with aid of test automation. Webomates provides automated audio testing services that test for latency, buffering, packet loss, or audio frame loss issues for a range of networks 2G/3G/4G/5G, etc. We also test audio applications to verify the quality of playback for various scenarios, like play/pause/forward/rewind. 

Besides the above mentioned services, Webomates provides comprehensive media testing services, which covers the following:

• Multi-device testing (Mobile/ Web/TV )

• API and UI testing
• Performance testing
• Exploratory testing

• Elastic scaling of test cases

• Smoke testing is made more efficient by elastic scaling i.e. quick ramping up of test cases if required.

• Guarantees

• Our services come with execution guarantees (15 mins / 24 hours)

• Leveraging AI value
  We have a human-assisted AI setup for covering all possible aspects of media testing

• Defect prediction – AI defect predictor
• AI analytics services

• Reporting

• We have a comprehensive triaging process, which aids developers and testers in analyzing and fixing bugs quickly.
• Outage detection
• Real-time notifications are sent out as pager alerts, in case any issue is detected.

Our media testing services aid in giving high-quality results in less time, thus significantly improving the feature velocity, which is the need of the hour.

Webomates CQ can make life easier for organizations looking for comprehensive TaaS tools. If this has piqued your interest and you want to know more, then please click here and schedule a demo, or reach out to us at info@webomates.com.

If you liked this blog, then please like/follow us Webomates or Aseem.

Read Next –

Api testing

Shift left testing

DevOps testing

Intelligent test automation

OTT media testing services

Requirements traceability matrix

Black box testing

Regression testing

Software Testing Life Cycle

Test Automation vs Manual Testing

Selenium Testing Automation

Exploratory testing in software testing

Tags: Audio testing, Automation, Media testing