Improving the DevOps Survey: How and Why

The 2014 State of DevOps report is a significant expansion of an ongoing research project we’ve undertaken for the past three years with the help of two prominent thought leaders, Gene Kim and Jez Humble. In our 2013 report, we identified key measurements for IT performance. We wanted to take our research a step further and understand how IT performance impacts overall organizational performance, using metrics that matter to the business: productivity, market share and profitability. We also wanted to understand how the cultural aspects of DevOps affect both IT and organizational performance.

This year, Dr. Nicole Forsgren Velasquez joined our team, bringing great experience and rigor to our survey design and data analysis. She’s a professor in management information systems and accounting at the Huntsman School of Business at Utah State University, and her work focuses on knowledge management, IT impacts and analytics. Based on her academic work and her experience as a consultant in the IT industry, Nicole suggested inclusion of specific measures of organizational performance, plus questions targeting a company’s climate for learning in the December 2013 DevOps survey. Nicole is also a former sysadmin, making her the perfect addition to our research efforts.

Our analysis of the 2012 data yielded breakthrough performance statistics for high-performing IT organizations, showing that the best among them shipped code as much as 30 times faster than their lesser-performing peers, with 50 percent fewer failures. That was great to discover, and opened up more questions for the team — for example, how the increase in code throughput and quality related to organizational performance.

We realized we hadn't designed the survey in a way that let us find statistically significant predictors for organizational performance. Given the bias towards continual improvement here at Puppet Labs (is obsession a cultural attribute?), we were determined to confirm whether such a connection existed.

Our goals were:

  • To understand how IT performance and DevOps practices impact organizational performance.
  • To understand how cultural elements such as the climate for learning, organizational culture and job satisfaction affects both IT performance and organizational performance.

One major improvement we made to the survey was changing the yes/no questions to Likert-type questions — scaling from “strongly agree” to “strongly disagree” — so we could get a more nuanced measurement. By gathering responses with greater granularity, we were also able to capture a more meaningful spectrum of practices and opinions, and apply more rigorous, predictive analysis. Additionally, our research team decided to leverage a number of previously validated scales in measuring several cultural factors.

  • Organizational performance. To measure organizational performance, we used a previously validated scale that relies on self-reported assessment of performance, relevant to organizational goals. There's a large body of research that says self-reported measures are better indicators of organizational performance than actual return on assets (ROA), because ROA typically lags tech activity by 2 to 3 years. Using this scale was a more accurate measure of how DevOps affects organizational performance, and it also allowed us to capture a measure of organizational performance from all survey respondents, not just those whose employers publicly report their financial information.

  • Job satisfaction. It’s not news that employees who are more satisfied perform better. We wanted to confirm this was still the case for IT, and needed this information to make sure our models were as complete as possible. We also wondered about the relationship between job satisfaction and different aspects/components of IT and organizational performance.

  • Climate for learning. We wanted to know about organizations that regard investing in learning as necessary and valuable for improvement. Did these see superior organizational performance and IT performance?

The improvements we made to this year’s survey made it possible for us to say definitively yes: A company’s culture has a real impact on its IT performance, and therefore on its overall organizational performance. We’ll be carrying these insights forward into our research this fall, and into the design of next year’s survey, too. Stay tuned!

Last but not least, we want to thank the 9,200-plus respondents who participated in the 2013 State of DevOps Survey. We appreciate the time and thoughtfulness you invested in helping us perform our rigorous evaluation of the IT industry. We hope you’ll participate again next year, and we invite those of you who haven’t participated yet to respond when we launch the new survey this December.

Wally Zabaglio is the business optimization manager at Puppet Labs.

Learn More

Puppet sites use proprietary and third-party cookies. By using our sites, you agree to our cookie policy.