Jurij Tokarski

Jurij Tokarski

Q&A7 minUpdated:

How Do We Release Without Breaking Things?

Every release takes a day, a freeze and a night deploy, and still breaks. Small changes behind flags make a release one command and a rollback take minutes.

Every release takes our team a full day: a code freeze, a manual QA pass, then someone deploys at night. Something still breaks every other time. How do teams ship often without this?

Small changes, released often, behind flags and watched from the first minute. Each part of that day-long release has a smaller version.

A release becomes one command, and a bad change rolls back in minutes. The freeze and the night deploy go away, and QA moves earlier.

The breakage every other time is the clue I'd start from. It comes from the size of the batch, not from the people running it.

Big Releases Hide the Bad Change

A code freeze gathers every change since the last release into one batch. When something breaks after the night deploy, it's buried among unrelated changes, and nobody knows which one did it.

Rolling back means taking all of them back, the good with the bad. So teams in that spot usually fix forward, at night, on code that changed in many places at once.

Then the next release gets put off, and its batch grows. Production Is Done names that loop: if deploying is painful, it gets avoided, and work piles up.

Small Batches Point at the Bad Change

The way out is smaller batches. Under Continuous Flow, work runs top-down from a ranked list, and each piece deploys as it's ready. A failure then points at the one change that just went out.

Feature Flags make small deploys safe. Unfinished work ships to production behind a disabled flag, so main stays ready to release. When a feature misbehaves, the flag turns off: a surgical rollback, not a full revert.

Quality Gates test in proportion to risk: the payment flow gets more scrutiny than a settings page.

Watching closes the loop. Monitor Day One puts error tracking in place from the first deploy. A bad change then shows up in minutes, not in a user's complaint.

Teams usually call this DevOps, or CI/CD.

DevOps Is the Habit, CI/CD the Pipeline

CI/CD is the machine. Continuous integration builds and tests every change as it merges. Continuous delivery takes it to production at the push of a button, without a release day.

DevOps is the way of working around that machine. It means small changes in a continuous flow and a rollback in minutes. The team owns the path from idea to users together.

Patrick Debois coined the word in 2009 for exactly that. Later it also became a job title for cloud administration.

The difference matters to this team. A faster pipeline would barely shorten their day, because the freeze and the QA pass would still wait in front of it. Why does my team ship so slowly? follows those waits.

The release steps make a natural first fix, because someone already runs them by hand.

Write the Steps Down, Then Script Them

Start by writing the steps down, in order, as the person who deploys at night runs them. Often that list lives in one head.

Then turn it into one command, a step at a time. Each step becomes a script or a CI job. The command runs them in order and reports where the team already talks.

Two decisions stay with people: which version goes out, and whether it's good enough to go.

On a product team I helped, a release took 16 manual actions across three services. I automated them into one Slack command. It opened a release pull request per service and waited for QA to approve in a Slack thread. Then it merged, tagged the version and announced each deploy.

The team went to one or two releases a day, with no incorrect release in almost two years. The bigger change was that the team stopped being afraid to release.

Undo belongs in the same design. A release that takes one command to ship and a night to undo is only half automated.

Scripting removes the steps. It can't remove a test phase that still sits inside them.

QA Moves to the Change

This team's manual QA pass tests the whole batch at once, on release day. It finds problems late, when they're already mixed with everything else.

So QA moves to the change. Each change gets tested while it's still one change, on a preview or on staging, before it joins the rest. Testing belongs before the release, not inside it.

A product team I helped worked this way: staging stayed ready to ship, and the release only published what QA had already passed.

That holds while staging matches production. Config, secrets and data drift apart, and then staging passes while production breaks. So staging gets matched to production on purpose, and the last check runs in production, behind a flag.

None of this asks for a new tool.

A Small Team Needs Fewer Tools

A release needs three kinds of tool, and most teams already have them.

A CI runner such as GitHub Actions builds, tests and deploys. A host such as Vercel, Firebase Hosting or Cloud Run serves what it built. Error tracking such as Sentry says what broke before users do.

Use the ones the team already has. A script in the CI runner does most of what a dedicated release tool would. Every new service is one more failure point.

Flags need no tool at first either: a boolean check in the code is enough until a gradual rollout earns a service.

The real work is in each host's rules. Four failures of mine passed every local test and broke only after deploy. Among them were a file Vercel left out of the bundle and a path that moved on Cloud Run.

Kubernetes Is Rarely the Answer

The largest tool question is Kubernetes, and under ten people the answer is rarely yes. It runs containers across a cluster of machines, and running the cluster becomes work of its own.

The freeze, the night deploy and the QA day all sit outside the cloud, so a cluster fixes none of them.

Often the better fix is less infrastructure. My RSS reader ran on Firebase plus a VPS for background workers: two systems to deploy and monitor. One of four fixes moved the workers into Cloud Functions. Together they left one deployment pipeline and a bill of $0.75 a month.

Clusters, private networks and access rules across cloud accounts are cloud engineering. When a product needs them, a cloud engineer is the right hire.

The Fastest Teams Also Break Least

DORA's 2024 State of DevOps report, from October 2024, found that speed and stability typically move together. The best performers do well on all four of its delivery metrics, and the lowest do poorly.

Its top 19% of respondents deployed on demand, several times a day. Their change fail rate, the share of deploys that need a hotfix or a rollback, was 5%. They recovered in under an hour.

The bottom 25% deployed between once a month and once every six months. Their change fail rate was 40%, and recovery took between a week and a month.

A release that breaks every other time fails more often than that bottom group. Among DORA's respondents, the teams that released least broke most.

How I Fix a Release Path

My one-week DevOps audit reads the whole path, from how an idea becomes a task to what alerts in production. It ends in a ranked fix list and the top fix.

After the audit, the team works down the list, or I do it by the week, one task at a time. The price is on the homepage.

I script my own releases too. Publishing a GitHub Release runs the tests, the build and the npm publish in GitHub Actions, with no npm token stored.

What the Team Should Do Now

At the next release, write down every step as it runs, in order, with who runs it. Then script the first step, and the next one at the release after.

Move the QA pass to each change as it lands, so staging stays ready to ship. Put the next unfinished feature behind a flag, and make sure an error alert fires before a user writes in.

Each scripted step takes something off release day, until the night deploy is a command anyone can run in daylight.

Shipping without breaking things takes small changes, a release anyone can run, and an undo that takes minutes. A bigger platform adds none of them.

Got thoughts on this? Reply by email.

Jurij Tokarski

Jurij Tokarski

I run a solo business and produce software. Usually I'm shipping for clients or building for myself. Sometimes I pause to write about it.

x.comlinkedin.comjurij@varstatt.comRSS