Canary Testing- Ensure Safer & Faster CI/CD

Canary birds were once used as an alert system by British coal miners for warning about toxic gases in mines. The birds were kept in the mines and if the gases started reaching dangerous levels, the canary bird would die after inhaling them. As the human threshold for toxic gases was higher than the canary, the bird’s death signalled the danger, prompting the evacuation of the mine. 

In a similar way, a small subset of users are used as an early warning system to flag any issues in the latest release having new features or bug fixes, before those features become accessible to the entire user community. Canary Release is a practice that allows partial rollout of new features to a limited audience while currently functioning software is still available to everyone else.

Understanding Canary Release

Canary Release testing strategies are gaining importance, especially as organizations are moving towards micro services architectures, agile processes and Continuous Integration/Deployment. It is an excellent method to reduce the risk of a software defect impacting the entire user base and bringing the software release to a standstill

Canary release reduces the risks to a great extent by:

1. Significantly reducing the exposure of changes to just the subset of users
2. Deployment in the production environment
   • Validating final deployment configuration 
   • Validating production data usage
3. Auto rollback is triggered if expected metrics are not achieved by the subset of users, such as page load time, latency, scalability etc.  

Canary release allows testing of new features in parallel. Since these features are available to a fraction of the actual number of users, the impact is minimal in case things go south.

Different organisations customize and tailor Canary Releases to fit their organization goals. Some such instances are stated below:

1. Some organisations use it for Dev, QA and UAT testing in production. The traffic is diverted as per the type of users testing the release. Once the signoff is received for a stage, the version moves up the chain to the next stage and is then finally released to production. 
2. Some organisations divert 10% of user traffic to the new version and remaining continue using the older version. This whole process is carefully monitored to identify issues due to new features or bug fixes. In case of any issue in the new version, it is rolled back to the previously functioning version. 
3. It is also used as a feature flag or A/B testing. However, this is not an ideal approach for A/B testing, since normally the focus is to capture analytics to prove a hypothesis and with a canary release only a subset of users are exposed. 

A separate production environment is used for Canary testing keeping the existing production environment as the baseline. Maximum users of the software continue using the existing baseline production environment, but a subset of users are directed to the test environment once the changes are implemented. The subset of users then “test” the modified software, thus serving the purpose of a Canary. Any issues in this version are flagged and appropriate action is taken. The baseline environment continues to function in parallel.

Once the Canary release stabilizes and meets the quality requirements, the remaining users are redirected to the new environment and the baseline environment is archived for stipulated time frame, in case it is needed for a rollback at a later date.

Canary Release in a non-micro services environment

In a non micro services environment, canary deployments are applicable to an entire application, thus limiting its actual utility. Any new/modified feature cannot be immediately tested against real world traffic until the full build is created.

Take a look at the following figure for a better understanding. Team B made some changes to the service they were working on, but in order to release that particular service to production, the application as a whole has to be built(with no other changes, except service B), and then released to production.

Micro services architecture takes care of this issue.

Canary Release in micro services environment

Micro services architecture aid in faster release cycles and can easily scale up to meet the market demands. This obviously leads to the need of having a robust CI/CD process in place. Different components/services may have different release cycles. But the net result should be a seamlessly functioning end product. 

Individual services can upgrade, test and release their versions without impacting the functioning of an entire software. It also gives the flexibility of maintaining and controlling the versions for the individual services.

Canary releases aid the development teams in releasing micro services/functions effectively in the real environment.

When to perform Canary Testing

• Any application which has multiple micro services that keep changing as per the user requirement should be preferably tested as a Canary Release.
• When there is a high risk to operations when a functionality changes. In such cases, it is prudent to check the changes in a production environment.

Benefits of Canary Testing

• It provides flexibility of testing the new version with a rollback strategy in place, in case there are any issues.
• Load testing can be done efficiently. It becomes easier to monitor the performance metrics by slowly ramping up the load.
• Risk is mitigated by exposing new changes to a limited number of users.

Drawbacks of Canary Testing

• To facilitate Canary testing, it is important to maintain multiple versions of the software to manage rollouts and rollbacks (in case of issues). Version control can be a daunting task if not managed properly.
• In case of mobile/desktop versions, pushing the user to update the software to the latest build is a subjective activity. This creates ambiguity in the total number of users actually under review for Canary testing.
• Managing database schema changes to reflect the canary testing environment, while maintaining the integrity of the current database may prove challenging.

Conclusion

Webomates has a gating mechanism for various environments used for production and testing. 

Our CI/CD toolchain triggers following process:

• Code build process
• Deployment
• Regression testing
• Invokes canary setup 

Once the setup is operational, the Webomates canary module tracks issues faced by canary users. This is done on the basis of predefined rules.

The results of Canary testing are reported back and appropriate actions are taken as per the results.

If you are interested in learning more about the services offered by Webomates then please click here and schedule a demo, or reach out to us at info@webomates.com. You can also avail a free trial by clicking here.

Tags: Canary Testing