Wolf&Grayson EditorialOrganizations still shipping like it’s 2010 face a structural problem. Multi-year release cycles allow going dark for stabilization, heavy re-architecture, and fundamental changes during beta phases. Continuous deployment requires making changes while the system is live.
Mike L. Swafford has led engineering teams through this transition at cloud scale, including rewriting Exchange Online’s entire database layer from C++ to C# while the service was live and shipping continuously. He has learned that the hardest part is not the tooling. It is the mindset shift required when you cannot put the ship in dry dock.
“With multi-year release cycles, you get to put the ship in dry dock and do a bunch of work on it,” Swafford explains. “Your ship is on the ocean, your plane is flying. You need to make the changes while it’s flying, while it’s floating.”
You Cannot Go Dark
The Exchange Online team rewrote the storage layer in a new language (C#) while the service was live and shipping continuously. This required vigilance about testing, side-by-side deployment strategies, and careful data migration plans. The alternative would have been going dark for three years, which might have cost less in engineering resources but would have introduced massive schedule risk. While this may sound like an extreme example, it’s not. Big architectural changes are an essential part of service innovation. If you don’t have the capability to innovate, you will be left behind.
“While this approach may cost more than going dark for three years, it prevents schedule slippage,” says Swafford. “When you’re shipping continuously, you develop rigor around validation, monitoring, production availability, and customer feedback. The schedule doesn’t get away from you.”
Individual features may take longer than anticipated, but the organization is always improving because shipping discipline prevents the debt accumulation that derails multi-year projects. Instead of planning three-year rewrites, teams design incremental migrations that deliver value continuously.
Where CI/CD Transformations Break
Most organizations break their CI/CD transformation by focusing exclusively on making developers happy without considering the bigger picture. Happy developers get to choose their own tools, work on narrowly scoped items, and remain isolated from mistakes made elsewhere.
Swafford has seen cases where developers validate small changes in 10-15 minutes on their machines, but it takes months for those changes to reach the actual product because there is no rigor about how individual work fits into the larger system.
“You want to have happy developers, but you want them using standardized tools and acting as professionals who discuss interactions between software modules,” Swafford explains. “You want to ship as a team, not as a collection of individuals.”
Systems That Enable Speed Without Risk
Moving fast without increasing production risk requires robust validation before production, safe deployment processes, and comprehensive telemetry working together.
Safe deployment means gradual rollout. Deployment speed is determined by how quickly you can get reliable statistical signals from collected data.
“You need to have a story and a plan about how you gradually deploy things,” says Swafford. “The speed of your deployment is determined by how quickly you can get a good and reliable signal statistically from the data that you collect.”
Dark deployment capabilities allow statistical sampling to verify expected behavior before features go live. Experimentation frameworks test whether new functionality steals user attention from revenue-generating activities.
Enterprise services cannot treat paying customers as testers. New functionality either needs to work when it goes live or needs to be scoped to specific scenarios where it is known to work.
High-Performance Developer Experience at Cloud Scale
At cloud scale, high-performance developer experience requires maximum standardization. Large products require distributed build and distributed test experiences because product size makes local development impractical.
Cloud scale also demands careful cost management. When 100+ microservices each provision for peak loads independently, massive unused capacity accumulates.
“If you have 100 microservices and each one is responsible for availability and cost, they’re going to over-provision,” Swafford explains. “You will have an incredible amount of unused capacity, which is great for surging but not great for cost management.”
The solution is elasticity or capacity sharing, so services run at comfortable levels while handling peaks without expecting everything to peak simultaneously.
First Practical Steps
Organizations still shipping like 2010 should start by ensuring release branches are always healthy through frequent internal deployment and heavy investment in automation that validates code at check-in and before deployment.
The DORA metrics measure progress such as deployment frequency, lead time from check-in to deployment, system availability, deployment failure rate, and time to recover from failures.
Developer productivity metrics provide additional insight. Track failure rates during code check-in, distinguishing between self-induced failures and system flakiness.
“Also listen to your developers,” says Swafford. “We use subjective metrics where we ask developers how happy they are. If you’re below 100 on a 0-200 NSAT scale, there’s something you need to look at.”
The Speed Imperative
There is a huge difference between shipping every three years and shipping every two weeks. Organizations should aim to go as fast as possible to the first feedback ring.
Swafford’s team moved from every two weeks to every week to multiple times per day. Each transition required significant engineering work, including designing custom SKUs for multiple builds in flight, multiple database copies for deployment testing, and failover/failback capabilities.
“Fast inner ring deployment provides feedback on the entire ecosystem, not just individual products,” Swafford concludes. “For complex products, integration with ecosystem components through production deployment enables cross-service functionality development. Rapid deployment against other production services provides valuable feedback for developing functionality spanning multiple services.”
Connect with Mike L. Swafford on LinkedIn for insights on transitioning from box software to continuous deployment successfully.
