How We Deploy Product Releases at xMatters
With Halloween behind us and the holiday shopping season fast approaching, engineering and product teams know what that means: code freezes! At xMatters, code freezes are a part of our product release process in anticipation of the busiest—and most important—time of the year for many of our customers.
But code freezes are just one piece of the puzzle in how we ensure our customers have the most reliable experiences. The way our product releases are designed is much more than that.
How xMatters Deploys Product Releases
Deploying new releases requires a careful balance of speed and reliability, and making sure that users aren’t taken by surprise when the product changes. In other words, we need to deliver important enhancements while not moving the cheese too much.
Here at xMatters, we deploy small releases multiple times a week for reasons ranging from customer requirements, third-party changes, fixing bugs, and other operational concerns that need fast attention. We deploy frequently to keep our “change increment” small so it is easy to ensure that all of the various services in xMatters are properly interacting with each other (and there are fewer variables to debug when they don’t).
Feature Toggles
A key to implementing frequent deployments is decoupling features and deployments. This allows us to deploy code to address issues or provide minor enhancements without having to worry about the status or readiness of a feature in progress. We do this using something our customers never see: feature toggles.
New features are in our deployments but are effectively disabled until we “toggle them on” in our feature toggle system. With feature toggles, our service teams can focus on fearlessly deploying their latest code, and our product owners can decide the best time for a feature to be toggled on for our customers. Not only does this enable frequent deployments, but it also allows our product owners to run beta programs (more on this later) and a progressive non-production to production release process—all while using the same deployed code.
Canary Release Strategy
Deployments are rolled out using a phased approach called a canary release, where the new changes and functionality are initially only available to a small subset of users or early adopters. Doing so allows us to catch any issues with the new code before releasing it to a larger audience.
Conventional wisdom would say that more deployments equal more changes, and more changes equal more incidents. But using these, and other modern release engineering processes has shown that to be a myth. We have now quadrupled our deployment frequency, and the results have been rewarding: customer-impacting incidents and customer-reported issues have been reduced by 50%.
Quarterly Releases
Now, what about our exciting quarterly releases like the Ninja release? Great question! Each quarter, xMatters reserves and delivers an extensive list of features and functionality enhancements curated by our product managers. Updates reserved for quarterly releases tend to be larger or more impactful to customer experiences, like interface updates to managing incidents or sending and receiving alerts. We announce these updates pre-release on our Support site so xMatters users can prepare themselves for the changes. We also enable the new features in customer non-production environments first so they can evaluate the new features and update any internal training or playbooks they maintain.
At least two weeks later (and sometimes more), we “toggle on” the new features in production environments. Our customers appreciate that this approach approximates some of the old school “promote to production” behaviors they rely on for their own change management.
Hard Freezes vs Soft Freezes
During the most critical and busiest times for our customers, the last thing they want is to have issues with their incident response and management system. That’s why we incorporate two types of code freezes to prevent any incidents from occurring: hard freeze and soft freeze.
Hard Freeze
There are no deployments except for addressing major changes or services outages during a hard freeze. Any lower priority bug fixes or updates will have to wait. At xMatters, our hard freeze time frames in 2021 are:
- November 22 to November 29
- December 18 to December 25
As you can see, those time frames are during the peak holiday shopping seasons—the week of American Thanksgiving and Black Friday, and the week leading up to Christmas.
Soft Freeze
We avoid major changes during a soft freeze but bug fixes and other important changes can still go out. At xMatters, our soft freeze time frame in 2021 is:
- December 26 to January 1
It’s the week after Christmas, meaning that retail stores will be busy with shoppers quickly spending their holiday earnings, and our xPerts may still be enjoying the holiday festivities with their loved ones.
xMatters Early Access Program (EAP)
Want to try out xMatters latest features before they go to production environments? We offer customers an opportunity to opt their non-production environments through our Early Access Program (EAP) to see new functionality before it’s released. Typically, customers enrolled in the EAP will see new features every month, or as often as every week in the buildup to the quarterly release.
Calling All Old-School Gamers!
In case you didn’t notice, xMatters quarterly releases are all code-named alphabetically after vintage video games. We’ve just celebrated Ninja, the classic hack-and-slash platform game, and in the previous quarters, we’ve covered other iconic games like Missile Command, Lunar Lander, Krull, and more. Out Run is up next, but what should our “P” release be named? Share your suggestion with us on social!
And there you have it! A detailed walkthrough of our product release process at xMatters. Without the hard work from many teams across the organization, we wouldn’t have the smooth sailing ship that we have now. If you’re interested in learning more, be sure to follow our latest product news and updates here.