DEVOPSdigest

It’s the big day, and your team is excited to implement the new features on your company’s grocery shopping app. You’ve been working on the code for weeks and ensuring that everything will run smoothly by planning for a canary release.

The service that allows your customers to search and compare groceries will now have two new features whereby customers can search for products that are organic and products that are locally sourced. This is going to make their lives a lot easier, and maybe even win you that coveted customer satisfaction award. Your stakeholders are eagerly awaiting the feedback and press coverage from the launch of these new features — no pressure!

The features go live, and all is running smoothly. The routing of a select few of your users to the new features is working excellently. Your team begins to cheer and celebrate, and a huge smile stretches across your face. But that smile soon disappears and the cheering soon stops. The app is crashing when customers try to access a long-awaited new feature and they’re angry-tweeting about not being able to use it. A bug has been identified — one that wasn’t detected during your tests.

You get a report on another new feature. It’s working great and the feedback is amazing. That smile stretches across your face again. But then you remember, you used a canary release. Now you have to halt both features to fix the bug.

You have to explain to your stakeholders why the entire rollout has stopped. Now your weekend plans are ruined because you’ll be stuck fixing the bug and implementing the full rollout again. You kick the nearest trash can, and you mumble some not-safe-for-work words under your breath.

Your frustration and anger could have been avoided if you had opted to do a feature flag rollout instead of a canary release. You’d be halting only one of the features instead of the entire rollout.

I know what you’re thinking … “But both are common feature release strategies for testing in production, increasing the safety of continuous delivery, and deploying faster. Both aim to minimize the fallout of unforeseen problems and build confidence in a release. Both also gradually expose new code to users on the production infrastructure.” And you’re right! But, that’s where the comparisons end.

The differences between canary releases and feature flag rollouts are significant enough to impact development velocity — and your weekend plans.

A canary release operates within the infrastructure networking layer to deploy the new features to a small subset of production machines. The canary phase starts with most machines running the version without the new features, and a smaller subset of machines running the version with them. A traffic router (usually a load balancer) will route users to the two sets of machines, with stickiness turned on to ensure that each user has a consistent experience. Users routed to the canary release will keep getting access to the new features while the rest will not.

Canary releases are orchestrated by your infrastructure and DevOps team. They also have limited targeting capabilities. They either target a percentage of your incoming traffic and/or whitelist certain IP addresses.

While canary releases do make a lot of sense when the thing you are changing is a piece of infrastructure itself or an app that is effectively a single black box, they are not the way to go when you have a manifest of multiple features contained in a single deployment package.

A feature flag, meanwhile, operates within the application code and deploys the new features to all of the production machines. Exposure to the new features is limited to a targeted subset of users by hiding them behind a flag. The traffic router now lives in the application code and is controlled by an if/else statement. This if/else statement is the feature flag. A simple function call made by the if/else statement decides which code path gets executed for each user. The rules for that if/else can be changed on the fly at any time.

Feature flag rollouts are orchestrated by your engineers, product managers, or agile testers, and have an infinite flexibility of targeting, allowing any group of users to be affected.

See the difference yet?

But the biggest and most important differentiation is the way these feature releases operate with roll backs. Canary releases operate at the level of deployment, which means that if there are any issues or bugs, the entire deployment is rolled back. So, if you’re implementing two new features at the same time, it means both have to be rolled back and launched again on another day.

However, feature flag rollouts operate at the most granular code level, which means that if two new features are being implemented and an issue or bug is found, one feature can be rolled back independently of any other, and without a hotfix or redeployment.

Not having to roll back the entire deployment means: avoiding an all-hands fire-drill, not delaying the rest of your team from completing their part, not having an awkward conversation with your stakeholders, and being able to continue with your weekend plans.

Feature flag rollouts make for happier and faster-moving engineering teams due to the fact that they can simply make the bug-infested feature fade to black and allow the working feature to shine.