Top 5 Challenges in Crowdsourced Testing

Jeff Howe and Mark Robinson coined the term “crowdsourcing” in 2005. In today’s fast paced and dynamically changing technical world, the customer requirements change quite frequently. This leads to an exponential rise in testing requirements, since everyone wants more complex scenarios to be tested in a shorter period of time. Organizations need to be well equipped to handle this kind of demand, without having to compromise on the quality of the end product. There may be times when the inhouse team is pressed for time and resources, leading to a bottleneck in the development-test-delivery pipeline. Crowdsourced testing helps the organizations in managing the testing process in a cost-effective manner.

While the advantages of crowd sourced testing are well known, please be aware that “Crowd sourcing is not a Silver bullet”. It has its own challenges and limitations.

This blog talks about top five challenges in crowd-sourced testing and also recommends certain solutions to overcome these challenges. Following table gives a quick overview followed by a detailed explanation in the entailing text.

Detailed Analysis of Crowdsourcing Challenges

• ‘Whether to’ and ‘What to’

Crowdsourced testing is not a single stop solution to all testing problems. It is a crucial strategic decision, that requires  proper analysis of the reasons for opting it over inhouse testing, in terms of time, cost and quality of work. It is also important to identify the risks involved and keep a track of progress on a regular basis. Rarely does it make sense to send all testing to just crowd sourcing. 

Solutions:

1. Establish a lucid strategy on objectives to be achieved using crowdsourced testing.
2. Perform an appropriate DAR(Decision Analysis & Resolution) to ensure that all possible alternatives have been explored and a conscious decision has been made.
3. Explore some of the commonly used crowdsourced testing platforms like MTURK from Amazon.

• Cultural Change

Moving from a regular dedicated team-based structure to a hybrid model that involves crowd sourced testing resources, is a big cultural change for organizations. This shift raises concerns in the teams and may impact productivity if not handled properly. This gives rise to the following issues.

1. Unclear/ambiguous requirements can lead to wrong/delayed outputs from crowdsourced testing teams.
2. Employees react differently to the introduction of external resources. Insecurities can demotivate the team.
3. Team building (between existing resources and crowdsourced resources) becomes a daunting task.
4. Query resolution/communication with the crowdsourced team can’t be done the way local / small permanent teams can be done.
5. Entire ecosystem of project execution gets impacted and has to be handled delicately. 

Solutions:

1. Institutionalize the concept of crowdsourced testing within the existing team.
2. Define clear Scope, Requirements, Communication methods, and channels, etc.
3. Take appropriate judgment on what to automate and what to crowdsource. Refer to our article Crowdsourced Testing Vs Automation Testing.

• Quality (Execution, Monitoring & Control)

It is of essence to identify the right resources for the job in order to achieve good quality testing,. Resources with appropriate expertise in ‘testing’ and ‘domain’ are difficult to identify. Further:

1. Generally people try to follow the most cost-effective resource approach, which may not necessarily lead to the talented resources.
2. Knowledge transfer to the crowdsourced resources becomes a challenge.
3. Identifying the methodology and appropriate tools for monitoring the progress and quality is difficult.
4. Ready availability of the resource can pose a challenge in case an interaction is required to handle queries. 

Solutions:

1. Establish a clear selection criteria. Conducting an online test to set a minimum threshold for hiring proves helpful.
2. Deliver adequate knowledge sharing sessions for quality work.
3. Select appropriate tools for monitoring the progress and quality of execution.
4. It’s important to be aware of Do’s and Don’ts of Crowdsourced testing.

• Knowledge Management

As mentioned earlier, it’s challenging to get SMEs and resources with appropriate knowledge of the application under test. This gives rise to the need of conducting knowledge sharing sessions. Also, the resources can learn about AUT while working on it giving them better insights for testing better. However, there is always a risk of losing this knowledge as the crowdsourced testing resources are not bound to any particular organization.

Solutions:

1. Build a strong and easy to understand knowledge repository. 
2. Have a good monetary and non-monetary incentive process in place to earn loyalty of  crowdsourced resources.

• Security and Confidentiality

Last but not the least, security and confidentiality are of prime importance for organizations. Sharing the details of application under test may lead to security and confidentiality issues.

Solutions:

1. Do not crowdsource any work which has high business sensitivity.
2. Have appropriate online NDAs signed from crowdsourced testing resources
3. Avoid sharing insights that are not related and mandatory for the task in hand.
4. Avoid any unmonitored communication of crowdsourced teams with existing teams.

Conclusion

If used appropriately, Crowdsourced testing can add a lot of value to the whole testing process. Having said that, Crowdsourced testing is not a wonder drug that can cure any problem. One needs to vigilantly identify appropriate usage areas and establish apt implementation techniques and procedures.

At Webomates, we use AI to continuously improve the confidence level of the results achieved. We also take care of customer confidentiality while taking care of several challenges mentioned above. To know more you can see our article “The Webomates Way: Overcoming the Challenges in Crowdsourced Testing”

If you would like to see the Webomates platform in action you can always sign up for a free trial. Or if you would like to learn more about Webomates CQ click on the link or schedule a call.

Tags: Crowdsource Testing, Manual Testing