We launched our 2012 Devops Survey back in December, in collaboration with Gene Kim and Jez Humble. It was our intent to gain better understanding of the state of DevOps today and share it with the world.
We wanted to see whether people cared about DevOps, and what they were doing about it. Our goal was to benchmark the performance, benefits & barriers, demographics, tooling and community across the spectrum of Devops adoption.
While we're still crunching some of the more advanced (and more interesting) data, we wanted to share some of the demographics of the survey population. At the end of this post, we’ll share some a sneak peak of some of the analysis, and you'll have some idea of why Gene, Jez and I are jumping up and down with excitement.
Who took the survey?
Your response to the survey was astonishing, with nearly 6x the number of respondents we saw in 2011, from over 90 countries. In fact, this is the largest DevOps survey ever conducted, with responses from over 4,000 IT Operations and Engineering professionals.
While we did a pretty good job getting a lot of operations and development professionals surveyed, we're already thinking of ways to survey more QA folks in the future.
As you can see, the vast majority of respondents were hands-on technical people, which is great! Most of the DevOps practices we asked about are performed by the team member, not the C-level executive.
We set out to include a wide variety of organizations, from tiny start-ups to established companies. We also wanted to learn about a good cross-section of team sizes.
Design of the Survey
When conducting a benchmark or survey like this, there are two primary phases: survey design and survey analysis. In the design phase, we create a survey instrument based on the question we want to answer or the hypothesis we want to test. In the analysis phase, we use tools like SPSS, R and Google Refine to tabulate the data, analyze it and prove or disprove the hypothesis.
The DevOps Survey Of Practice was modeled after many of the techniques that Gene used in his 5+ years of benchmarking over 1300 IT organizations with the IT Process Institute. You can read more about the methodology and results in this blog post.
Our survey design goals were to better understand:
- The cultural perceptions and organizational penetration of the DevOps movement
- The relationship between DevOps practices and business performance
- The requisite skills and tools for DevOps implementation
We’ll discuss how we measured #2, as it was Jez and Gene's primary area of interest. What are the high performing DevOps shops doing, and what were they doing that differentiated their performance?
We created two sets of questions. The first were the independent variables—these were the DevOps behaviors, processes and practices that we hypothesized would correlate with performance.
- Who performs code deployments? (Dev, Ops, or both?)
- Who is on the hook for production support (i.e., when something goes wrong during or post deployment, who gets paged)?
- Are environment and infrastructure changes (everything except for code) checked into revision control?
- Is there an automated process to deploy environment and infrastructure changes?
- Does some compliance requirement (e.g., regulatory, security) restrict your ability to make changes more frequently?
The other set of questions were the dependent variables, which are performance measures that we believe are dependent on the behaviors, processes and practices. They are listed below:
- How often does your organization deploy code? (Daily, weekly, monthly, quarterly, or yearly?)
- What percentage of your changes require rollbacks or hotfixes?
- How long does it take to restore service when something goes wrong?
- What is your lead time for changes (i.e., how long does it take to go from code committed to code successfully running in production)?
Gene and I have been using R to analyze the data, and holy crap! There’s signal! There’s definitely early indications that there is strong correlation between specific DevOps practices and improved performance!
Here’s a sneak peek at a (deliberately blurry) diagram that we’ll explain in the next article:
Here’s how Gene, Jez and I felt about it:
And we're looking forward to sharing the information with you! We'll be back on the blog sometime next month with our more robust results.