Joshua Zimmerman has done nearly anything you can imagine at the University of Wisconsin - Madison libraries over the past decade — everything from help desk support to web development to Windows system administration. Most recently, Joshua has been part of the team that's architecting and maintaining a growing Linux server environment for both in-house applications and those supplied by vendors.
Joshua's talk at PuppetConf, How to Succeed in Relearning Puppet Without Really Trying, will focus on how his team identified anti-patterns in its Puppet code and infrastructure, and why the team chose to relearn Puppet, rather than to learn what had changed in Puppet. We're hoping for some fun slides like the ones from his Ignite talk at DevOpsDays Chicago, along with the more practical advice.
We invited Joshua to share a few thoughts on DevOps and system administration in advance of his talk at PuppetConf.
Aliza Earnshaw: How (and when) did you first start using Puppet?
Aliza: Have you worked on a team that transitioned to DevOps? If so, what went well, what went badly, and what advice do you have for a team that wants to make the transition?
Joshua: I’d say that I’m in a department that is transitioning to DevOps. My opinion is that you never really finish transitioning to a DevOps culture. At its core, DevOps is about learning to continually empathize and help others, whether they're your technical colleagues, colleagues in your larger organization, or people who consume your software. Sure, the tools we do that with tend to be automation tools; but, that’s the reason we use them.
I feel that things go wrong when a team or an organization views this as a project. It’s not a box that you check off and never worry about again. The organizations where DevOps is considered successful don’t view it as a project. Many of them don’t even call it DevOps. They’re concerned with how their teams work, both as a team and within their organization. They’re focused on both having happy teams and happy software consumers. They succeed because they continually look to improve their organizational processes and culture.
Aliza: DevOps principles seem to make so much sense. What do you think stands in the way of teams adopting them?
Joshua: They’re hard to adopt. We like to think of things like empathy and emotions as inherent skills and therefore easy; however, they’re challenging skills, and frankly a lot of people are bad at employing them. So, we take these teams of individuals that undervalue these skills, in an organization that undervalues them, and we tell them to implement various new technologies. Many of these technologies have been developed with the idea of empathy for others in mind, but using the technology does not fix the underlying issue in the organization. Many people and organizations get this backwards — they feel the tech will fix the organizational and cultural problems they have. When you implement more complex technology and ignore your cultural problems, you wind up with a more complex version of your current problems.
To paraphrase many amazing people (Bridget Kromhout, Katherine Daniels, etc.), “Technology won’t fix your shitty culture.” You have to train people to care in a society that devalues empathy. You have to get people to work well together and actually collaborate, not just cooperate. When you have people who are doing this and you begin to look at “DevOps tools,” you can then understand how they’ve been created to work in ways that leverage cross-functional collaboration. They’ve been created to help your organization put humans before technology, and that is where the strength of these tools lies.
Aliza: How does Puppet help teams make the transition to a more DevOps-ey culture?
Joshua: Any tool, especially those designed around automation, exists to make people’s lives easier. Automation tools like Puppet can allow teams to ease inter-team friction. Puppet can help provide simpler abstractions on top of what were historically complex tasks: configuring and long-term management of systems and application environments. Teams can use tools like Puppet to increase collaboration, by giving teams a common tool for how to configure systems and different ways to communicate about these systems, through things like pull requests and code reviews.
Aliza: What are the most common problems you see when teams are trying to improve their configuration management?
Joshua: One of the biggest problems that I’ve seen — and I’m going to specifically call out people using Puppet — is that they put the automation before the state they’re trying to end up in. I’ve seen manifests written by others (heck, I even wrote some myself early on) that contain exec resources chained to other exec resources, or they deploy a shell script with a file resource and then use an exec resource to run it. Puppet was designed so that you could describe your desired state and converge to it. So, in my opinion, your main manifests, your roles and profiles — if you subscribe to that way of doing things — should reflect that as much as possible.
Another large issue is Not Invented Here syndrome , when people or an organization refuse to use something that was created elsewhere. This is definitely one that we’ve struggled with over the years. There are wonderful modules out there for any CM tool that are approved or maintained by the organization or the community.
Puppet and other CM tools exist to help abstract how you maintain your servers. Someone needs to care about how a specific thing is done, but that doesn’t need to be everyone who touches your code base. It maybe doesn’t even need to be you. There are people at Puppet, Chef, etc., and in their communities, who probably care about how an individual thing is automated more than you do — so let them care. Contribute if you want to, or if your organization needs you to do so. But most of us have better things to do at work than argue about how a specific thing is automated, and we need to let the tools make our lives easier.
Aliza: Is there a mindset or approach to configuration management you think is most helpful?
Joshua: For mindset, I think it is best to approach things in an optimistic way, believing that you’re going to do something right while simultaneously knowing that you’re going to do it wrong. Even if it isn’t wrong now, it will likely be wrong eventually when underlying technologies change, and things are done in a significantly different way. When you approach things in this mindset, you start doing things like writing good tests and good documentation that help explain why you did something, and help you and others refactor and change what you did later, when it's necessary.
Aliza: What's a fun fact about you that we might not know, but would like to?
Joshua: This week, I have watched Disney’s Hercules three times.
Aliza: What's the most interesting thing you've read recently?
Joshua: I sometimes try to take advantage of the fact that I have access to academic journals because I work for a university. I recently spent some time catching up on a few linguistics journals, and there was this amazing article that explored when and how in their development children might map phonological changes on to mental representations of words.
Aliza: Go ahead, tease us: What can we look forward to in your talk about anti-patterns in Puppet code and infrastructure?
Joshua: I think the most important word in the title of my talk is “relearn.” Oftentimes, the anti-patterns that we deal with in tech are self-inflicted, but they happen for reasons that at the time appeared legitimate to us. When we approach these problems, it can be very easy to return to the same or similar answers that we found before. So, rather than approaching the problem from scratch, we approach it with our biases that we learned, no matter how much the tooling or systems have changed since that first time.
My talk is going to have several different components. Part of it will be me sharing my experiences of the past year or two, working through an aged codebase, deciding how to proceed, and updating not just our code, but our processes and policies around it. Part of it will be me rambling on about how we learn, both in general and specifically with regards to tech tools; cognitive biases; and how to let go of the pride we have in our work. I’ll round it out talking about some of the features we found in Puppet 4.x that we like, and how those same features may bite us in the future if we haven’t changed how we approach things.
Aliza Earnshaw is the editorial director at Puppet.