The Curious Case of Video Testing

Video services have gained immense popularity in the past few years. They can be broadly classified into two categories: Video Conferencing and Video Streaming aka broadcasting. Their demand has skyrocketed in the last few months when the world is under siege due to the prevailing pandemic.

Live video conferencing tools, like MS teams, Zoom, Google Hangouts, Skype, etc, are being used extensively these days for conducting meetings and scheduling web-based conferences.

The education field has started depending highly on various eLearning platforms, besides leveraging the video conferencing tools.

Equipped with smartphones and access to the internet, the rate of consumption of video on demand is overwhelming too, be it sports, music, movies, news, or documentaries. It gives the flexibility to watch videos on the move at the time of your choice.

The surge in demand and usage, in general, puts a high load on the content delivery and management infrastructure. In order to appease the current customer base and entice new ones, the product and service providers need to maintain & upgrade current infrastructure, besides introducing new measures to provide consistent and high-quality services. This gives rise to the need of having a good video testing process in place.

Why do we need video testing?

Imagine a scenario when an important point is being discussed in a meeting/class and the video gets paused, or the picture gets pixelated, or the voice is not clearly audible. It adversely impacts the whole user experience. 

The performance of the whole ecosystem of video services is vital for continued customer satisfaction and loyalty. Like every other technology, video testing is extremely important to ensure that the end-user gets maximum out of his/her investment in the tech he/she paid for. This task is considerably tough keeping in mind the rate at which the hardware (mobiles, televisions, tablets, etc.) are upgraded and the content demand keeps going up.

The video content is dynamic in nature, coupled with the wide range of receiving hardware makes it a challenging task. Hence, it is imperative to have a well-outlined video testing process in place.

Video streaming performance metrics

Video testing also aids in analyzing the impact of actual usage during peak times by measuring various performance metrics, thereby ensuring that the end-users can have an uninterrupted experience. These metrics form the baseline for video testing.

• Bit Rate: The pixelated picture and impeded speech are classic symptoms of low quality video streaming. Take a look at the following snapshot of stills from a video. It is easy to spot the difference in quality of both.

Ever wondered why this happens? The culprit is “Bit Rate”. The video quality is directly proportional to the bit rate. Higher the bit rate, better the quality, consequently larger the video file size.

Bit rate is measured as the number of bits transmitted per second. A higher bit rate does not necessarily translate into better quality viewing on the receiver’s end if the hardware is not equipped to process that.While bit rate can determine the quality of audio-visuals in terms of looks, it has another comrade called, Frames per second (FPS), which determines the smoothness of the video. Just imagine watching an intense sports live and the player moves around in spasmodic choppy motion because the frame rate was not ideal. Higher frame rate means smooth videos with crisp detailing.

• Buffer time/fill: Quite often we see the video buffering message on our screens before it can start playing. 
  Have you ever thought about why this happens? Internet speed and reliability of connection is the prime reason. If the internet connection is slow, then the local buffer is filled with data before playing the video, so that there is no lag in the stream.Buffer fill is the time taken for the video file to fill the buffer. Longer the time is taken for buffering, a higher chance of having a dissatisfied end user.

• Lag length: So now the buffer is full and the video starts playing. All goes well as long as the download rate matches pace with the bit rate.

But the moment, buffer starts getting drained, the video stalls and re-buffering starts. This leads to a lag in streaming.
Commonly known as time taken for re-buffering, this phenomenon is measured by lag length. Lag length is the time taken to refill the buffer.

• Play Length: Play length is the measurement of data streamed by the data center and received by the end-user. There might be users who consume volumes of video data. Play length helps in understanding the capacity of the infrastructure, enhancement planning, and streaming demand patterns.

• Lag Ratio:

This is the ratio of waiting time: watching time. The moment this ratio is skewed with waiting time exceeding watching time by a high degree, it is time to revisit and fine tune the setup.

Nuances of Video Testing

Several scenarios have to be considered while planning for video testing. The following points are just a few examples that can be used as verification guidelines while embarking on the journey of test case planning.

These are just some examples. A lot more scenarios can be conjured up for video testing. Fiddle as much as you can with the various functionalities and come up with test cases. 

Keeping in mind the above sample scenarios, manual testing and visual analysis with human intervention becomes an obvious choice.  However, manual testing is time-consuming and has its own limitations when it comes to repetitive testing. That’s where automation helps. Tools like Selenium, Ruby, and Jscript can be used for video testing automation. However, it is vital to ensure that the stakeholders get the best ROI from automation.

The success of automation testing is highly dependent on the infrastructure generating and transmitting the content as well as the recipient. Testing for recipients can be done on the basis of the target audience and sample data generated from it. Moreover, the test environment can emulate the real setup to a certain degree. Device and target environment diversity may pose a challenge in automation since video testing needs to address a variety of endpoints with different configurations. Also, it is important to ensure that the automation scripts are reusable and maintainable.

Adaptive bitrate streaming is a way of adjusting the video quality depending on the user’s device and network connectivity. This may pose a challenge while testing the system and may lead to inaccurate testing results.

Conclusion

The rate at which video content delivery services are growing and evolving to provide the best, fast, and clear content culminates into a direct need for having high-end testing services. Eventually, it all boils down to the end-user experience. 

Webomates uses black-box testing technique to test a variety of video streaming, as well as proctored based online examination applications which covers above shared video performance metrics verification. These tests combine Automation and Manual testing techniques to check real-time issues related to buffering, bitrate, frame loss, latency or functional issues. This approach ensures that all these said issues can be caught during regression and the user doesn’t face any unwarranted experience while streaming the videos.

At Webomates, we continuously work to evolve our platform and processes in order to provide guaranteed execution, which takes testing experience to an entirely different level, thus ensuring a higher degree of customer satisfaction.

Webomates offers regression testing as a service that is a combination of test cases based testing and exploratory testing. Test cases based testing establishes a baseline and uses a multi-execution channel approach, to reach true pass / true fail, using AI Automation, automation, manual, and crowdsourcing. Exploratory testing expands the scope of the test to take the quality to the next level. Service is guaranteed to do both and even covers updating test cases of modified features during execution in 24 hours.If you are interested in learning more about Webomates’ CQ service please click here and schedule a demo, or reach out to us at info@webomates.com.

Tags: buffering, Latency, On Demand, OTT, Videoperformance, videotesting