DevOps Solves Business Problems: Gene Kim’s Top Aha Moments
Interested in DevOps? Then you have probably already read or listened to Gene Kim. The author of The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win, Gene is also the co-founder and former CTO of Tripwire. He writes and speaks widely about how IT operations and DevOps can improve software delivery.
Gene will be speaking at PuppetConf next month about how to better sell DevOps to skeptical or risk-averse executives. We chatted with Gene last week about the DevOps “aha!” moments that have inspired him to spread the word.
“I wanted to share the top moments I’ve had, as well as those from other people who I respect tremendously. What they all have in common is realizing that DevOps solves some of the biggest business problems we’ve faced,” Gene says. “Whoever they are — the VP of Development, head of UX — they solved not only their business problem, but their boss’s problem, and their boss’s boss’s problems, too.”
Gene recalls a time in 2008, when he was working with Eric Passmore, the senior vice president of global engineering at AOL. In his role as CTO of Tripwire, Gene and his team were helping AOL improve deployment flow.
“Eric was concerned about the increasingly turbulent installs that kept taking longer and longer,” Gene said. “Because it took quarters for the ops team to update the Linux kernel from 2.4 to 2.6, it was as debilitating for dev as a code freeze.” The development team had their code ready, but without the multi-threading support provided by the 2.6 kernel, they couldn’t deploy.
“His “aha” moment was, ‘that’s not an ops problem, that’s not a dev problem, that’s a business problem,’” Gene said. “He started thinking about all the customers they weren’t able to capture because of the delayed functionality.”
The story illustrates how much pain an organization — or a business executive — has to be in before there’s enough motivation to change. In Eric Passmore’s case, the pain of losing business drove him to change and standardize the operational environments of some important projects.
“In the old way of doing things, developers would put WAR [web application archive] files in a network share, and then open up a ticket with IT operations for them to deploy,” Gene said. “This was problematic, because ops had to pick through all that stuff, and figure out exactly how things needed to be deployed, and where.”
Eric shifted that step to the quality assurance team, which took over responsibility for packaging. Now what the IT team received was actually deploy-ready. Time to deployment went from six hours to 45 minutes.
“It wasn’t a huge million-dollar project,” Gene said. “It was just resequencing two steps. My big aha moment was, amazingly great outcomes can come from really small changes.”
DevOps yields business resultsAfter 14 years of benchmarking high-performance IT organizations, Gene views DevOps as a natural continuation of that work. He defines DevOps as the means of increasing the flow of features into production while preserving world class reliability, security and stability. “It’s about linking behavior and controls to performance,” he said.
“Behavior” covers DevOps practices such as delivering smaller changes more frequently; getting dev and ops teams to plan changes together; and reviewing deployments regularly so practices can be improved. “Controls” covers tools such as version control, configuration management and automation.
Changing behavior can be a lot harder than changing tools. As Gene observes, it’s human nature for us to avoid activities that are painful, and to do them less frequently than perhaps we should. This is as true for software deployment as it is for, say, cleaning out your garage.
In the waterfall method of software development, deployments are large and infrequent - and they are also pretty painful for the IT ops team. “You have to clear the week, clear the weekend, when it’s time to do a release,” Gene said.
That’s because large releases are likely to yield a lot of errors, Gene pointed out. “When you’re doing one big deploy per year, it can be tens or hundreds of thousands of lines of code touching multiple databases, modules, interfaces and networks. What are the odds of all these things going right?” Hence, everyone involved in the deployment is working, or on standby, for days or even weeks, having to fix business-threatening errors at any hour of the day or night — assuming they can figure out what caused the errors, a more difficult task in large deployments.
DevOps practices involve deploying smaller changes more frequently, substantially lowering the risk of business-threatening errors. It’s also much easier for IT to find the source of errors when they do occur. When deployments occur multiple times per day instead of once or twice per year, they’re no longer a potentially catastrophic, all-hands-on-deck event — they become business as usual.
Working the DevOps way can also make it much easier to include new features as the team discovers customers want them.
“When you’re stuck with large painful releases, getting into the next release becomes extremely political,” Gene said. “You have to escalate up eight levels of management, all the way to the CEO, to get your idea included.”
Frequent deployment also makes it far more likely you’ll be releasing new features and improvements in a timely fashion, while the market is ripe for them, rather than falling behind more agile competitors.
DevOps skills are prized — and rareFor IT people, success in a more agile world means mastering the skills and tools that enable frequent deployment. Understanding of the cloud is extremely helpful to DevOps. The ability to use development tools to configure and manage development, testing and production environments is critical. Automation is a key skill for IT people, allowing them to manage frequent deployments safely.
Though some in IT fear automation as a job-killer, product teams are hiring people with automation skills as fast as they can find them, whenever they can find them. “I hear it all the time: ‘If I could find 40 ops engineers I’d hire them on the spot,’” Gene said. “ It shows that there’s a narrative saying you’ll never have enough ops engineers.”
- Getting devs and ops folks on the same side is half the battle. Here's a success story: 25% Tech, 75% Culture: Build a Window in the Wall
- Editor's note: Read all about the state of DevOps today in the 2015 report, and learn how DevOps can boost overall organizational performance, and how it depends strongly on IT leadership for success.
- Don't stress about changing everything - just learn to automate one small thing with Puppet Enterprise.