DevOps: There’s No Free Lunch
The good news about implementing DevOps processes and practices is that they allow you to do things faster, at greater scale, and with a higher degree of self-service for internal customers.
The bad news about implementing DevOps processes and practices is that giving people the ability to self-service results in sprawl. People deploy new environments and stand them up isolated from other environments, consuming more resources. After a while, you end up asking yourself, could we not used or adapted existing infrastructure, instead of building new?
This issue is something people don’t usually consider when they decide to do DevOps. A lot of people in DevOps push self-service, so you end up with a lot more people empowered to bring up systems and provision new environments. Now instead of having a handful of servers, you have dozens of virtual machines, or hundreds. The management of all those resources, and tracking them, takes up a lot of time and energy. Plus all these systems take up CPU, memory and I/O, potentially impacting the performance of other resources.
You need to put controls in place, and to do that, you have to model your infrastructure. When people first try to do DevOps, modeling infrastructure is extremely difficult. When you try to implement something the first time, you’re actually in the early stages of learning it. You go pretty far down that road, and you make a large investment in your custom code. Then you begin to realize you made some decisions early on that were mistakes.
That’s pretty painful when it comes to infrastructure, because when you are maintaining infrastructure while you’re rebuilding and refining your code, you are likely to break something, and need to back out changes. People who are good at DevOps can do that without breaking systems, but they are also people who have been doing it for some time. They’ve learned because they have already made their mistakes.
So why should any team shift to DevOps? Because once you get through the learning stage, it really does work. The gains in efficiency are big. My team — the quality engineering team — couldn’t do what we’re doing now, at the scale we are doing it, if not for DevOps. And by that I mean using code to provision, deploy and manage our testing infrastructure and test fixtures.
With the best will in the world — and I’d say we have that here at Puppet Labs — the interests of developers and IT operations will clash from time to time. When our need to test outstrips available internal infrastructure (and available time and attention from the IT ops team), we are able to easily point our tools at external cloud services. That’s the kind of flexibility that DevOps enables.
One big change you have to deal with when you go down the DevOps road — and the main reason for all the push-and-tug between teams — is that when you are running your infrastructure as code, ownership is a lot less clear. It was easy when management of systems was very manual: It was the ops team that logged in and changed things on systems, and physical access to the data centers was tightly controlled. Now we set up environments and virtual machines and manage them all with code. So when a piece of code degrades network services or systems, you at least have your infrastructure code in version control; you can roll back quickly, isolate the change that caused the issue, and identify the person who made the change. This isn’t about hanging someone out to dry: It’s about quick resolution. Getting to the heart of the issue quickly — and to the person who is best equipped to fix it — is a real advantage of using DevOps methodologies.
To do DevOps successfully, you have to acknowledge that now the network, services and systems can be affected by more people. You have to set ownership, boundaries and rules, and put in rigorous processes for auditing and reviewing code. It really is a paradigm shift, one that sparks spirited discussion. And that discussion, too, is part of DevOps.
So yes, DevOps is awesome, but it isn’t less work, especially in the beginning. DevOps requires you to think about the problem you are trying to solve with code. If you are doing it right, you’ll break down the problem into pieces and processes, with an engineering mindset.
The teams that can make this huge leap to infrastructure-as-code will break out of the firefighting-and-hacking cycle. In its place, you’ll have a repeatable, well-documented and scalable process that enables your team to work flexibly and efficiently.