The History of DevOps Reports

by Nigel Kersten

2021 marked the 10-year anniversary of our industry-defining State of DevOps research. You’d think after 10 years that there wouldn’t be much more to say about DevOps, but practices and technologies continue to evolve, the best keep getting better, and we keep unearthing new topics to research.

We’ve been lucky to collaborate with some of the best thinkers in the DevOps space. When we started this project in 2011, there were no vendors doing independent, vendor-agnostic research of this magnitude. Alanna Brown deserves all of the credit for the idea of creating a survey and report to investigate the state and actual impact of this burgeoning movement, and this space wouldn’t be the same without her advocacy and insight. Apart from myself, she also enlisted the incredibly prolific and erudite James Turnbull, a critical contributor to the early days of Puppet.

In 2012, we reached out to Gene Kim who had been a supporter and friend of Puppet since the early days. We knew he was about to publish his groundbreaking novel on IT and DevOps, The Phoenix Project, and couldn’t think of a better person to collaborate with. Gene brought in Jez Humble, the guy who literally wrote the book on Continuous Delivery in 2010 (2010!). Over 4,000 technical professionals completed our 2012 survey — an unheard of response rate for any IT survey at the time. We launched our first State of DevOps report together in 2013 and the appetite for the data blew us away.

In 2013 we had a chance encounter at LISA (Large Installation System Administration Conference) with an academic researcher who specialized in IT impacts. Dr. Nicole Forsgren joined our scrappy little team and took the research to another level. We produced the 2014, 2015, 2016 and 2017 State of DevOps reports together.

By that time, Nicole, Jez and Gene had formed DORA (DevOps Research and Assessment) and wrote Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations based on these reports and other research. It’s a must-read for anyone looking to modernize their technical environment.

We broke new ground once again in the 2018 State of DevOps Report, bringing on Michael Stahnke and Andi Mann from Splunk as authors, with a focus on providing pragmatic and prescriptive guidance for organizations seeking to evolve their DevOps practice. We developed the DevOps evolution model that shows the sequencing of key practices across five stages as a common adoption pattern for scaling DevOps across an organization. The model is far from perfect, and there’s a lot that isn’t included in it, but it matches our experience working with large enterprises working out how to modernize their large and complex IT environments.

In 2019, we examined how organizations are integrating security into the software delivery lifecycle and in 2020, we focused on two themes: applying DevOps practices to change management and adopting a platform approach to software delivery.

By Alanna Brown

When we started the State of DevOps Report, the landscape looked very different than it does today. The organizations we were working with were struggling with deployments, the functional boundary where Dev and Ops painfully collide. The pressure to deploy faster and more frequently was a big driver behind DevOps.

As more organizations started adopting DevOps and practices became codified and broadly shared, we saw the low performers start to catch up with the high performers when it came to deployment frequency; a good sign that this was a solved problem. But of course, deploying faster and more frequently doesn’t necessarily mean you’re deploying better. And on the flip side, many organizations we work with today actually can deploy much faster and more frequently than the business allows.

Last year, we broke new ground with our report by focusing on the DevOps journey itself. We wanted to answer two burning questions that we kept getting asked over and over again: “How do we get started with DevOps?” and “How do we expand our pockets of success?” From those core questions — and a lot of statistical analysis — emerged the DevOps Evolutionary model (or simply, the DevOps Model, as it’s more commonly known). The model deepened our understanding of how organizations adopt and scale DevOps practices. From there, we set out to provide prescriptive and pragmatic guidance, based on the real-world experience of our authors (both in the trenches and from working directly with IT organizations), to help organizations evolve on their journey.

Normalization and Standardization

In the 2018 State of DevOps Report, our analysis revealed five stages of DevOps evolution and the critical practices at each stage that help you achieve success and progress to the next phase of your journey. The first two stages of that journey are normalizing the technology stack and standardizing and reducing variability. Normalization is about getting your arms around the chaos and understanding what you have so you can eliminate the one-offs that create management complexity. Standardization is about placing bets on the technologies you’ll use in the future.

Nigel and I have been meeting with and conducting DevOps Workshops with very large, very complicated enterprises and the majority of them are stuck somewhere in between Stage 1 and Stage 2. Standardization is always a sticky topic because everyone recognizes the value of standardization, but often, no one person in the organization really has the power to enforce the standards. And then there are always those pesky dev teams that want to use their own special stack. We advise starting small with the tools and platforms within your direct purview before attempting to standardize your entire software delivery toolchain across hundreds of teams.

Free Resource

To help you do that, we’re offering a new resource to help rationalize and standardize your toolkit. It’s a simple spreadsheet that lists key capabilities so you can see where there are overlaps between the tools you’re using today.

You can access the spreadsheet and then either make a copy of the Google sheet (File < Make a copy) or download it into a Microsoft Excel file (File > Download as). Then it’s all yours to fill in the tools you use, mark what their capabilities are, and note the overlaps where there may be opportunity for standardization.

A lot of our customers have found this to be useful so we wanted to make it available to everyone. This one happens to focus on infrastructure automation, but we plan to build out more resources in the future and would love to hear your ideas on what would be most useful. Feel free to email us with your ideas!

Jump to 2018 report >>

By Alanna Brown

Every year, we look at IT performance to see how high-performing organizations compare to the rest of our sample population. This year, we found that high performers are accelerating away from the pack in terms of throughput. They deploy 200 times more frequently, which means deploying multiple times per day on demand, versus just a few times per year. High performers also have 2,555 times faster lead times, which means they can deploy a change in less than one hour compared to once every few months. We found that the high performers also continue to maintain high levels of stability — they have a three times lower change failure rate and recover from failures 24 times faster.

Throughput and stability matter, because when you’re able to deploy more frequently, you can experiment more and deliver value to customers faster. Instead of having one or two chances per year to get it right, you have multiple opportunities to validate your ideas, gather customer feedback, learn, and improve. By speeding up your delivery, you can increase your rate of learning.

Speed without stability causes other problems, though: Websites and other services break, disappointing customers, suppliers and fellow employees. However, as we’ve shown year after year, moving faster doesn’t have to come at the expense of stability, reliability, security, or quality. In fact, DevOps practices — for example, version control for all production artifacts, deployment automation, and automated testing — actually predict IT performance, which in turn predicts organizational performance.

By segmenting survey responses according to whether respondents' organizations are high, medium or low performers, we've been able to compare these groups across different dimensions to see whether their attitudes, behaviors, and practices are substantially different from each other. For example, we wondered if high performers had higher employee engagement, and learned that yes, they were 2.2 times more likely to recommend their organization to a friend as a great place to work, compared to low performers. Everyone wants to be part of a winning team, and those who do feel their teams are winning tend to be more engaged and loyal. Other studies have shown that employee engagement correlates with better business outcomes, such as higher customer engagement, revenue growth, and stock market performance.

A big focus of this year’s report was looking at the entire product development lifecycle, starting with the initial product or feature idea, and extending all the way to the customer, where that idea can deliver value. We found that when product teams take a lean-manufacturing approach to product design and delivery — decomposing features into small batches, making the flow of work visible throughout the delivery process, and using customer feedback to inform product design — both IT performance and organizational culture improve.

Another key idea that DevOps borrows from the lean movement is “shifting left,” or identifying and fixing defects early on, rather than inspecting quality at the end. By shifting quality to the left, you’re able to detect problems earlier when it’s much cheaper and easier to fix them. This in turn leads to less unplanned work and rework later on. We found that high performers spend 22 percent less time on unplanned work and rework than low performers, and as a result, they’re able to spend 29 percent more of their time on new, value-adding work. That's a great investment of team time.

Quality isn’t the only thing shifting left. DevSecOps, rugged DevOps, or whatever you want to call it, is all about integrating security early and often throughout the software development lifecycle. That means treating security concerns as a design constraint, getting continuous feedback from the security team, and building security requirements into the automated testing suite. We found that high performers spend 50 percent less time remediating security issues than low performers, because they integrate security testing and controls into the daily work of development, QA, and operations.

Jump to 2016 report >>