If you have spent any amount of time in DevOps, you know continuous deployment is hands-down the most efficient and effective approach for software companies to maintain and upgrade their products. You may also know that the vast majority of software companies don’t practice legit continuous deployment or any other form of continuous integration/continuous delivery (CI/CD).
Let’s talk about what real CI/CD looks like in practice, some potential reasons for low rates of adoption, and the dramatic benefits of CI/CD.
What Is CI/CD?
CI/CD is simply the process of pushing out all validated changes in software products as soon as they’re ready to go. It’s an idea that has generally been accepted as the “right” way to deliver software for almost 15 years now, but less than 20% of tech companies actually practice it. The Continuous Delivery Foundation’s 2021 State of Continuous Delivery Report confirmed that “while 44% of developers use either continuous integration or deployment (CD/CD), less than 18% employ both practices to fully automate their delivery processes.”
For most organizations, CI/CD translates into several updates per day just to keep up with routine maintenance. In fact, LeanIX’s explanation of DORA Metrics states that “more successful companies tend to do smaller and much more frequent deliveries — or in the world of DevOps, more frequent but smaller deployments.”
Yet, per the statistics above, less than half of software companies are even trying.
What gives?
A Misperception of Risk
I can only think of a few reasons why so many companies choose to avoid continuous updates, and I don’t feel any of them are compelling. In fact, I think the vast majority of companies that are late to the game on CI/CD have a misunderstanding of the actual risks involved in deploying software.
The reasons given often amount to a fear of something going wrong — basically, fear of failure. But here’s the problem: Failure is certain to happen at some point because that’s the nature of building and maintaining anything, especially something as complicated as software.
Instead of trying to opt out of a failure by having less frequent updates, these companies should focus on optimizing themselves for recovery. The best way to optimize DevOps processes for recovery is to make the failures as small and inconsequential as possible — which is one of the major benefits of CI/CD.
Outside of misunderstanding the risks involved or just have never heard of CI/CD (which is hard to imagine), the only other rationale I’ve seen given for holding back updates, fixes, and new features for weeks at a time amounts to organizational politics and bureaucratic dysfunction.
Some companies are still clinging to outdated development processes involving overly centralized development functions with bottlenecks and gatekeepers who insist on signing off on each update pushed out the door.
While it’s not good to be in this situation, the good news is that these companies can realize major benefits relatively quickly if they can successfully move toward real CI/CD.
The Benefits of CI/CD
If we stay with the theme of optimizing for recovery, the biggest benefit is that the inevitable misfires and failures that will happen will be much more manageable.
To put it into perspective, if a company practices CI/CD and rolls out multiple updates a day, each of those updates pushes a much more granular amount of code — possibly just a few days of work from one or two contributors.
So when there’s a glitch, the haystack the DevOps team must sift through is infinitely smaller than it would have been for a single monthly update that contains hundreds of hours of work and thousands of lines of code. The ripples caused by the glitch are much smaller, and it’s so much easier to zero in on the problem and fix it.
Your customers are another good reason to push updates and new features as quickly as possible. Is it wise to make your customers wait weeks or even a month for new innovations that will make their lives easier and businesses more profitable?
Timeliness can be particularly critical for security patches, for example. When a vulnerability in the software is identified, shouldn’t software companies go the extra mile to make sure their customers are protected as quickly as possible?
Recruiting is yet another reason to move your organization to a legit CI/CD. Basically, developers and engineers don’t like dumpster fires and complicated hotfixes, and with CI/CD, every fix is a tiny hotfix that prevents dumpster fires.
Pretty much everyone in DevOps knows continuous deployment is the gold standard, yet there are thousands of talented developers and engineers who are stuck pushing massive monthly updates and dealing with all the stress and trouble that follows. Many of those people would consider working for your company just for the ability to work in a more stable and low-drama environment.
CI/CD will also attract early career recruits, as many of them will have learned about the value of this approach.
And let’s not forget the cost and organizational demands. Big updates are expensive and inefficient because orchestrating them sucks so much air out of the organization. Developers, engineers, and quality assurance staff often have to work late or on weekends when major updates go through, and the glitches and bugs that pop up draw even more people into the process.
There can also be problems with the primacy/recency effect. When a company waits a month to push updates through, the engineers who wrote some of the code likely wrote it weeks before and it will be harder for them to zero in on any problems.
With CI/CD, engineers are more likely to see the updates through quickly because they just wrote the code days — or even hours — before.
If It Hurts, Do It More
I think we can agree that big deployments are generally painful experiences. There’s a free-floating level of anxiety that comes with them as a large number of people hold their breath in hopes that their cumulative work over the previous weeks fits together well. And even if nothing goes wrong, everyone is relieved when it’s over.
A mentor of mine used to say that if something hurts in the development process, you should do it more. This is particularly true of software deployments. If a company updates every day, every day cannot be a crisis. It’s just another day.
Stoplight has practiced full-blown CI/CD since the company’s inception, and it’s had a profound effect on the culture of our development and engineering teams. Deployments are just another daily task we complete to keep our products updated and make sure our customers have nearly instant access to new features.
For me, the most gratifying part of our approach to CI/CD is that it empowers our developers and engineers to take small, smart risks whenever new benefits are possible. We can sneak new wrinkles into features when there are strong indications that they will work. And if they don’t, we can quickly pull them back without much disruption.
It’s a fun, creative way to develop software, and I sincerely hope more companies will embrace continuous deployment for the betterment of our industry and my fellow engineers and coders.
I’m the VP of Engineering at Stoplight, and I hope to change the dialogue and culture of software engineering the same way that Stoplight is changing the dialogue and culture around APIs.
