Overcome the Challenges of Crowdsourced Testing

Crowdsourced testing is becoming popular in the industry, and it is a fairly new trend in the software engineering community. In crowdsourced testing, crowd-users are rewarded to perform testing tasks and submit their results, and because crowd-users are often paid per task, there is a financial incentive to complete the tasks rapidly. These submissions of the tasks are composed of answers in simple language with relevant screenshots.

Crowdsourced testing enables organizations of any size to access the power of curated or non-curated crowd-users to strengthen the test coverage especially across multiple geographies who have access to virtually unlimited devices at any time. It can be scaled up or down at a moment’s notice, at a very low cost. This might be otherwise impossible to mobilize within an in-house test effort. To learn more about the value of adding crowdsourcing to your testing strategy read this article. However, there are a lot of challenges with Crowdsourced testing. Let’s start looking at the disadvantages involved with Crowdsourced testing.

Challenge: Confidentiality 

Creating and maintaining confidentiality and reliability is one of the most critical challenges that an organization faces while working with CrowdSourced testing. It is important not to share or expose any information with crowd-testers which is confidential to the product owners.

How does Webomates handle it?

Webomates carefully assesses each challenge and creates bespoke solutions like custom plugins, secured virtual machines with continuous monitoring of activities performed by the crowd-users, HTTP proxy, etc for facilitating testing on the system under test.

Challenge: Effective Communication

Communication is another challenge while working with crowd-users. As Crowdsourced testing is carried out by anyone across the world, the person may not be immediately available for communication and clarification.

How does Webomates handle it?

Webomates overcame this challenge firstly by providing very high quality written test cases in a simple natural language. Each test case is divided into multiple validation points to which crowd-users provide answers.

Additionally, Webomates communicates with workers using multiple forms like email, chat messenger, discussion forums, etc to resolve any concern faced by the crowd-users. Conducting a regular Open Session using Video Conferencing tools is also an effective way to resolve the concerns which aren’t possible by other means of communication. Finally, prompt reply to the users’ queries is critical.

Challenge: Quality of Execution

Another major issue that arises is ensuring the quality of testing or the quality of defects raised. Quality of submission is another disadvantage generally faced with Crowdsourced testing. This becomes even more challenging when working with non-curated users.

How Webomates handles it?

Webomates overcame this challenge by creating Qualification Tests and by providing training to the users on complex scenarios like API testing and judging their understanding and knowledge with Qualification Tests. This way we converted non-curated users to semi-curated users.

It’s a challenge to manage crowd-users across the globe as they work in different time zones. Webomates always notifies the users whenever tasks are available to work. It is also important to publish tasks at the scheduled time and day of the week, making habitual for crowd-users when the tasks will be available. With regular work, they also become familiar with the application under test. It is also essential to provide feedback for each task.

Challenge: Ability to scale up

Ability to scale up on demand is another challenge with crowd-users. Crowd-users are spread across the globe and available to work in different time zones. Whether they have the right device on test cases that are supposed to be executed and whether they feel encouraged to work on a daily basis when they are not obliged to? So their availability to perform time-boxed testing can be a challenge.

How Webomates handles it?

Webomates does profiling of crowd-users on a regular basis to determine their QA skills, their availability according to the time zones they belong to and devices on which they can execute the test cases.

Webomates created four different pools of crowd-users on their ability to execute test cases. The performance of crowd-users is measured on multiple criteria like quality of submission (i.e. accuracy of the test cases submitted), the speed with which test cases are executed, the number of test cases executed, etc. The user is moved successively to a higher level when there is a significant improvement in his performance.

All these factors help Webomates to maintain a talent pool of crowd-users community which can scale-up quickly to execute large numbers of test cases.

Conclusion

Crowdsourced testing can give any organization a new software testing technique for carrying out feature regression testing or exploratory testing. However, setting up and running a crowdsourcing platform can face challenges like confidentiality, effective communication, scaling and quality of execution.

At Webomates, we refine our Crowdsource testing each day to continually improve our results with techniques ranging from using algorithms to identify which test case to crowdsource to maintaining the high quality of execution and effective communication with crowd-users while respecting the customer’s confidentiality. All these factors together help us to scale and provide speed to test the customer’s application.

If you would like to see the Webomates platform in action you can always sign up for a free trial. Or if you would like to learn more about Webomates CQ click on the link or schedule a call.

Tags: Crowdsource Testing, Software Testing