Season 1 — Episode 4

One of Puppet’s big strengths is declarative automation – you model the desired end state of your infrastructure and Puppet does the hard work of enforcing it. This concept is quite powerful and has become the industry standard for configuration management. But sometimes your workflow doesn't quite fit into that pattern. Sometimes you really do need to deploy point-in-time changes, or mix imperative and declarative approaches together in a single workflow.

With Bolt, we've learned from the lessons of Puppet and used those to design the next generation of infrastructure automation. Listen as Eric Sorenson and Yasmin Rajabi explain the tools of the future.

Join the Puppet Community on Slack

Transcript

Ben:Hi, my name is Ben. I am our developer advocate here on Puppet's community team. One of my jobs is to sort of connect the dots between people using our tools and the people building our tools. And part of that is bringing this line of technical podcasts to you. Here with me today is Eric who has been our product manager and sort of look like idea guy around the Puppet platform for many, many years. And Yasmin who is our product manager for Bolt. They're going to be talking about maybe you could call it the next generation of what configuration management might be.

Eric:As you mentioned manager Eric Sorenson and for most of the time I've been here I've been working on Puppet and we introduced Bolt back in 2017 and the project has really taken off since its initial introduction.

Yasmin:I'm also super excited to be here like Ben mentioned. My name's Yasmin and I'm the product manager and I work on Bolt. I've been at Puppet almost two years now so it's been a lot of fun and it's super cool to be here with Eric because when I told someone that I got a job at Puppet their first response was "Oh, you get to work with Eric Sorenson?" And so I do actually get to work with Eric Sorenson so that's cool.

Eric:That's terrifying actually, but it really gratifying as well. That's awesome. Bolt's been a really interesting project. We started off with an idea that we should have something that would work alongside of Puppet that would complement the things that Puppet does really well by filling in some of the things that it doesn't do super well or that people have to hack around in order to make it work. Largely that's around tasks. It's really hard to make Puppet the tool do something once and never do it again. It's really hard to make Puppet change the state of say a service twice in one catalog run. It's just kind of like fighting against the way the tool was designed which you know as I think most people know is intended to enforce one particular state and to keep that state in compliance with what your manifest says all the time.

Yasmin:The way I kind of look at it is Puppet's really good at telling systems to be a certain way. What our community asks for are tools that allow them to do things a certain way. So puppet's great at making your systems *be* a certain way and Bolt's great at making your systems *do* a certain thing.

Eric:Yeah, I think that's a great way to look at it. When I thought back over the history of configuration management - and I've been doing this kind of a terrifyingly long time - I gave a talk at Configuration Management Camp in Belgium in early February and called it fourth generation configuration management because there really have been a number of iterations and evolutions of how we think about and perform configuration management in our systems. The fourth generation is... maybe it's a little arbitrary. I had to come up with something that fit the title of the talk and I don't know whether it's completely accurate but the way I thought about it was first we had shell scripts before we had any kind of configuration management people were doing config management they just didn't call it that. You wrote shell scripts, and if you got a new machine which was not a very frequent occurrence you might get a new like a Sun Server or something like that once a year and so you didn't really have to spend a lot of time building repeatable configuration for the systems because it wasn't just something that happened all the time. But when you did, you wanted to pull something out of your home directory or someone else's home directory and run a shell script to do kind of baseline configuration on it, and that sort of set the stage for the first generation of actual tools that were purpose built for doing configuration management. I think a lot of people are familiar with CFEngine and its author Mark Burgess who did a lot of really important early work in both the theory of systems administration and the theory of configuration management as well as writing and developing CFEngine, which was a pretty groundbreaking tool at the time. It did things that no other tool in the open source ecosystem could actually do. And it was pretty complicated to use. It was tough, but once you got your CFEngine stuff up and running it was like "Oh my gosh, I have superpowers now. I can do things that used to take me weeks and weeks and I can get them done in just a few hours." And that was kind of the first iteration of it. Have either of you poked at CFEngine, have you ever used it?

Ben:Only in anger.

Yasmin:I think that was pre-my time in working with computers. Recently [Michael] Stahnke sent me an article about his thoughts on config management from like 2011 or something and it's just really cool to like get the history of that because I was way just before my time and it reminds me of like when we were starting on Bolt and tasks and I was doing product manager-y stuff, whatever reading docs, and catching up. I read at least three or four docs from 2019, 2012, like early early years that read like they were true. The same problem was there, it's just a more evolved way of solving it, which I just think is really cool as we look at the history that the problems are essentially the same, it's just how we solve them have evolved.

Eric:Yep. Luke Kanies, who wrote Puppet originally and started the company obviously, and I were both CFEngine community people originally - in fact that's where I met him back in 2002 or 2003 or something like that. Funny story: When Puppet was very first getting started one of the earliest customers of Luke's consulting business was Stanford University in Palo Alto. They were really early adopters of Puppet and they in fact just paid for him as a you know Reductive Labs consultant, which was basically just him and was his company but it was just him to come out to Stanford for a couple of weeks and work on Puppet for them and to implement some of the features that they needed, some of which are still in the code base which also is sort of terrifying. But when he came out I knew from talking to him on IRC and from CFEngine and stuff that he was into bikes, so I loaned him my bike so he could get around the Stanford campus and so he rode around on a fixed gear bike at Stanford in 2006 and built some of the early Puppet features there.

Ben:That sounds very Luke.

Eric:It was a fun time and that kind of leads into the idea that like that CFEngine being the first generation - it did some stuff fairly well, but it was ,as I mentioned, hard too hard to use and a lot of that dissatisfaction with how user unfriendly parts of CFEngine were design goals that Luke wanted to fix with Puppet. And that's where we got things like the resource abstraction layer, the fact that before things got really complicated in the Puppet language initially it was more like a config file like the syntax and the way a Puppet manifest looks was modeled after the way a Nagios configuration file would look, rather than having like a lot of complicated logic that you would have to follow through and try to understand. Another related project that I think also sort of went into that second generation is Chef. Similar to how Puppet came out of the CFEngine community, Chef actually came out of the Puppet community. Adam Jacob, who was the initial author of Chef and their CTO up until very recently, was a Puppet consultant and had a consulting company where they just went around and implemented Puppet in a lot of places and similar to Luke's unhappiness with CFEngine, it was Adam's unhappiness with Puppet that led him to create Chef in the first place. And they're really pretty similar tools, but there are a couple of points of I guess philosophical difference that encapsulate like why people would pick one or the other or why they would tend towards using Chef instead of Puppet. One of them is that Chef cookbooks are written in Ruby effectively - there is a DSL over them, but you can just start writing Ruby in the center of the cookbook, which is really powerful and it's great if you are a Ruby developer but it does sort of hurt that idea of the simple configuration file style syntax - the idea that instead of having to run the Puppet manifest in order to see what happened you can reason about it you can run it in noop mode and you can check whether or not it's going to do what you expect before you actually make any changes on the system. That's really really hard to do in Chef.

Eric:Moving on to the next kind of iteration I would say that Ansible and Salt are kind of the epitome of the third generation. These are both tools that came out kind of in the early 2000s - 2014 for Ansible and about 2012 for Salt and they are pretty orchestration driven. Yeah most of the configuration that you do with them is written in yaml and they're very much like you were saying earlier go out and get stuff done make changes your system. There is idempotence in Ansible playbooks if you if you choose to make it so, but it doesn't enforce it in the same way that Puppet manifests too. And the tools are great for a certain set of problems and in fact it was you know we took a lot of inspiration I would say and ideas about how people wanted to solve problems from the success and the rise of Ansible which kind of came onto the scene and really got a lot of mindshare and got a lot of users in a very short amount of time in around 2015 which sort of leads us to this fourth generation now, which I think is kind of like the blending of all of those approaches and bringing all of those things together in a consistent way and allowing you to do both model based and state driven config management and imperative tasks and ad hoc commands and you know to just do whatever you need to do in order to get the work done that you're trying to accomplish with the tool. And that's kind of where we're at with Bolt.

Ben:One thing that I find kind of interesting about that - I think I'd probably also add another layer there is like the zero-th level was sort of like cut and pasting from a wiki or something like that. And I admit that that's kind of where I went back to after running my head into CFEngine a couple of times, but every iteration that we come through it seems like the tools are simplifying a little bit getting a little bit simpler a little bit more streamlined and making it easier to get to your desired end result.

Yasmin:I think that's one of the main goals that we like - our charter, you could say, for Bolt is if we're building something does it make someone's life simpler? Is it easier for them to do the thing they're trying to do? Because at the end of the day the software we build is meant to make your life easier. It's meant to make the thing you're doing, not only just faster but like more enjoyable so that you go to work and you're like "Oh okay, I have to do this task and I'm going to use some tools to do that task." Hopefully that is an enjoyable experience vs. like "Oh no I have to use that tool to get my job done" and kind of that was one of the main things with Bolt that we continue to try and embody is is it making someone's life simpler. And it's almost like expected from people nowadays of it used to take maybe an hour to get something done. Now it's minutes then it'll be seconds and it's kind of like there's a level of patience people have and simplicity they expect that we kind of need to operate in, and that's kind of one of the main goals with Bolt.

Ben:But at the same time it also kind of gives you those steps towards the more powerful like model language that we still employ with Puppet.

Eric:Yeah that's that's been a really awesome feature to see develop. So initially, and just to walk very quickly through the initial sort of phases of Bolt, we started actually writing code in June of 2017 if I recall correctly. And the name initially came from one of the developers Josh Cooper, who's also been around Puppet for a long time and has been one of the core code developers, he was working on this little project and we needed to call it something. We had a few different goofy code names, none of which were ever destined for the outside world. But he said, "Well, can we can we call it Bolt?" And there was no reason not to. So that's what we picked and it stuck. But initially it was a pretty simple tool and we wanted to do this progression of being able to take stuff that you already have, whether that's just a shell script like you're saying something you command that you have in your batch history or something that you ran one time and it worked for you, and be able to use the tool in order to get that done. Like Yasmin was saying quicker and simpler and maybe more enjoyably than having to run it by hand, then progressed from that into what we call a task and a task with a capital T is a shell command or a program that you've written that expects certain input and produces output in a predictable way. And ideally the task is accompanied with a little bit of metadata inside a JSON file that lives alongside of the task and is named the same thing as the task with the JSON extension that tells Bolt what that task does and lets you make human readable descriptions of it. It uses the Puppet type system which we introduced in Puppet 4 in order to describe what the parameters of the task are using a pretty rich expressive set of types. So you can say like it's an Enum, so one of a few defined values that way the tool can do input validation before we ever even try to run the command. We can say well you supplied a value that's not one of the possible acceptable values of this Enum. So we're just going to bomb out early and tell you that it needs to be one of these predefined values. We can make sure that if you need an array that you get an array and those kinds of things. So that type system is really really helpful. And because it's reused from Puppet a lot of people were already used to it from writing modules and it was great to be able to leverage that capability that we already had in the different part of that ecosystem.

Yasmin:I think this is one of my favorite features of Bolt. This came along because we were having a user research interview with one of our users Nick Maludy and he's like "I love Bolt. I can get a lot of things done with it and I like Puppet, but I use another tool to kind of do a lot of my more ad hoc automation." And he's like "I would love to take the investment I already have with Puppet and use that and apply it as more of a one off." And that's kind of where the apply functionality started. We knew that there's so much value in the Puppet language, the content that we have, the community that we've built, and we want to be able to reuse that with Bolt. Bolt is meant to help you get started with Puppet faster and expand the kind of capabilities you have today that you're using Puppet for with Bolt so you can use it as a standalone as you're getting started or you can use it to expand your current kind of automation workflows. And so because of that we wanted to make sure that we're reusing our existing investments and that's where Bolt apply came. As soon as we shipped it, it became the second most used function within a plan which was super cool and that kind of drove us to build a command for it so you can actually just do Bolt apply run give your Puppet code and then apply it to the unknown the catalog gets compiled locally and then connects out to that node and collects the facts and does all the things and you get some Puppet code applied, which is really cool. And we've seen that popping up in different demos where people are literally just copy pasting from the Forge and they've built the thing they want to do. They don't have to write code for it and they can reuse investment that they already have.

Ben:So do you mean that people are using modules from the Forge by just copy and pasting the include name of the module whatever and it just works?

Yasmin:Yep. And it literally just works. We're working with the Forge team as well to kind of have some highlights on there so you can see how you use this content with Bolt. But it literally just works.

Eric:It's pretty cool because you can take those apply blocks and sprinkle them throughout your plans and that way you can intermingle, say a build step with a Puppet code step configures the application server with another imperative step that say restarts the server once it's done, and each of those steps can take a set of targets that Bolt will know to go out and execute the stuff on a bunch of those targets and then move on to the next step. So it's really powerful and allows you to string together what used to be maybe a bunch of unconnected steps into one cohesive place that is shareable and reusable plans just like tasks themselves live inside of a module so they can be uploaded to the Forge, they can be tested. The PDK [Puppet Development Kit] knows how to create tasks. I'm not sure it has plan support yet, but it's probably close at hand and the whole ecosystem has kind of added on this capability of being able to do task-based work alongside of state-based work and that's why it kind of leads back to that fourth generation thing. You want to be able to have it both ways, to have your cake and eat it too.

Ben:And it sounds like you're talking about taking the state definition away from just one single node and saying you can define state across an infrastructure of a handful of nodes so you can look at like an application where part of it runs on one node and part of it runs on another node and kind of orchestrate application of state across those nodes.

Eric:Yeah, exactly and each one of those can be again some mixture of existing Puppet code, it can be tasks that you've got off the Forge or you've written yourself, or it can be individual commands - just whatever whatever is appropriate for the job.

Yasmin:One of the things that's kind of cool about the combination of the Puppet code with Bolt is that you can use Bolt for a very long time and not realize you're actually using Puppet. So without typing the word Puppet you're actually using you're using the RAL, you're using the content, you may have even used Bolt to install the agents, configure all the settings you want, and you've gotten so far in actually being able to accomplish your automation goals and setting up kind of your tool and what the tool itself can do. With Bolt it lets you get really far into actually accomplishing your automation goal and under the hood you have gotten so much farther with Puppet, where if you want to make that switch too I'm doing something I'm applying it as a one-off and now I want to actually enforce it, it becomes really easy because you've gotten used to the language, the content, the RAL, and you may even used the tool to install all the bits and configure them as well - without having to actually think about using Puppet.

Eric:Yeah there's an interesting side effect of adding all this stuff into Bolt, which is that for a long time Puppet supported this master list style operation where you could have Puppet code distributed out to your end nodes and they would run Puppet apply locally and not have a server setup. Bolt with apply mode is sort of like a supercharged master list setup. You don't need to have a PuppetDB necessarily, you don't need a Puppet server, but you get to make use of that content. Bolt knows about Hiera and it can pull Hiera data. So if you're used to using yaml configuration for configuring your modules, Bolt can make use of that and will collect facts and be able to interpolate variables and Hiera correctly for for each of the nodes that it's compiling a little apply block for and like you said you can get really far without really having to do a bunch of setting up a bunch of infrastructure in order to get the work done. It can all kind of go from your laptop until you're ready to move it onto an actual server.

Ben:And there's no need to setup anything on the target nodes. There's no need to like rsync out  code or install any agents or anything, all you need is SSH. And then Bolt will handle all the setting up of the tool, right.

Eric:I have one quibble with what you just said. You do need to have the Puppet agent installed on the end nodes in order for the apply to work. You just have to have the package installed. You don't need to have an agent running necessarily, but the way it works is that it will ship over a compiled catalog and feed that into the agent. It will feed that into Puppet as its input, rather than have the agent request a catalog from the master, but you do need to have the Puppet software there again, but it doesn't need to have SSL certificates, it doesn't need to be running permanently on the nodes, those kinds of things. So the setup cost is much less.

Ben:And Bolt'll automate that for you with the apply prep.

Yasmin:Yep. Apply prep will gather all the facts it needs, make sure everything that needs to be installed is there, and then allow you to apply that compiled Puppet code.

Eric:People have really taken Bolt to heart and it's really kind of heartwarming I would say to see the amount of things that people have been able to accomplish with it, without even really you know we haven't we've just sort of just started ramping up on talking about it a lot. Getting a lot of talks at DevOps Days and other kinds of events that are pretty Bolt-focused but people have really latched onto the tool and done some really amazing things with it. I mean, I at least didn't expect it to get as much popularity and get as awesome as it's gotten.

Yasmin:Yeah it's definitely so exciting to see all the usage. I track the Slack channel and every day I see new faces in there and it's just like super heartwarming and it's just exciting to see that people have it resonates, you know. When we're building software we want to make sure we're building software that actually solves problems, not that we think could be cool. And so seeing people either discuss how to use it or share how they're using it, it really it's validating kind of to know that you're solving a problem and you're not kind of making assumptions on the things you're building. One of the things we've seen with traditional Puppet users getting started with Bolt is actually using the Bolt in the Bolt apply mode to test out Puppet code. So rather than kind of commit changes, test it out, deploy them, see if it works. They can just take a chunk of Puppet code and keep iterating on it right from their laptop. And once they feel comfortable with it, it works on whatever the targets they're addressing, it's just a copy paste into your control repo and deploying it out to your traditional Puppet deployment. So that's kind of one of the things we've seen with our core Puppet users.

Yasmin:We also have a couple cool stories. The one I really like is there's a team out there I won't name them, but they had a vendor come in and apply a change to over a thousand of their nodes and it was approved or whatever, the change went out and they realized like it just took everything down and we need to make a fix real quick and with Bolt they were able to do this within minutes and deploy out a fix get their systems back into the state they need. And they had mentioned that if they didn't have Bolt this would take them days to go through and check which servers have that issue, which ones need to be updated, and actually go make the update. Bolt was really crucial in that moment to help them kind of solve that.

Ben:I have a story that I can tell about an interaction that I had on IRC earlier today. A user had certificates that were ready to expire and he was kind of worried about like how he's going to navigate that - go from old certificates, new certificates re-sign all the agents certificates and what happens if he makes a mistake along the way, and now the agents are no longer talking like authorized to talk to the master and everything. I suggested that he set up autosign and then use Bolt to just delete the certificate and rerun Puppet and that gives him like a kind of a second way in, so that even if he made a mistake he still could control the nodes with Bolt to re-request certificates. And he says that sounds like a great idea. And no kidding, like eight minutes later he came back and said that totally worked. And that went from like zero like he had just heard about Bolt.

Yasmin:That's super awesome. It's amazing just watching people install and get started with the software, how simple it is. I mean I don't believe it usually, but when we actually have it in the workshops that we're doing people install Bolt and actually try and solve a problem with it - seeing that, just seeing how easy it is - it's still surprising. I know it shouldn't be surprising, but it's still surprising how easy it is to get started and actually accomplish something before going through all the steps of setup and everything.

Eric:Pretty cool. Let's talk a little bit about what's what's coming up, what's next for Bolt, where it's headed. There's some there's some work in flight. I know around continuing to bridge that gap between agent-based management and state management. And can you talk a little bit about the network and network device features?

Yasmin:Yeah. One of the most common things people ask about are supporting network devices because we've expanded the scope of what you can automate through more agentless technologies. And because of that a commonly requested thing is "I want to automate some network devices" and we have a team at Puppet that's been focused on this area for a while now and they've been building really good content, they've been building tools to kind of help people push forward in this area. And what we've been working on with them is combining efforts and allowing Bolt to be that interface into any type of automation. So reusing again our existing investments in the network automation space, the content that we have, making sure its agentless first, using a more remote transports to target those network devices using Bolt. So whether you want to apply changes on a device right on the device, you want to apply those changes, have them run locally and remotely applied to that device, or maybe you want to send that to a proxy node where the proxy node can actually target those devices and run those changes. And so we've been doing work with our network automation team internally at Puppet to help drive that and let Bolt be the tool and help empower automation workflows using Bolt.

Eric:That way you can really bring those network devices into that same workflow. Maybe you have a task plan that involves interacting with a load balancer or a switch or something like that, you can bring that into the same place where you would do a code deployment or apply Puppet code all inside the same plan.

Yasmin:Right. Exactly. And the goal is to be seamless so you as a user don't have to think about what type of target that is. But rest assured that bolt will connect out using the right credentials, the right transports, and running the right code against them.

Eric:Are there particular device types or manufacturers that we're targeting first?

Yasmin:We're first targeting Cisco and Palo Alto firewalls, so you'll see that shortly. All the code's open anyway so you probably are seeing the changes come out either in this week or next week's release of Bolt should have support for the new remote transport. We try and ship Bolt weekly so that people can get those updates quickly. Sometimes we can't for whatever reason, but we try and stick to a weekly schedule.

Eric:It seems to like the contributor cycle for Bolt has really gotten a lot of improvements too so that if people want to get started not just using it but also working on the tool and maybe contributing improvements or even filing bugs and helping out in that regard that the team's been in my experience super responsive with everything from GitHub issues to pull requests, right.

Yasmin:Yeah, that's something super important to the team. If you're ever on the Bolt Slack channel, you'll see all of the engineers are out there answering questions, they do office hours, and merging pull requests or issues on GitHub is super - like it's something we value and we've been trying to have more of our discussions on Slack so that people can provide input and even one of the reasons why we ship weekly is so that if somebody merges a change they can actually see their change quickly and that's - I mean it's a lot more gratifying to make a fix and see that than wait maybe six months until you see that fix in the software. We definitely appreciate any pull requests or issues and if there is a way that we can do this better, please tell us on Slack. One of the things that surprised me a lot is we kind of ran a fun Twitter/X campaign. We wanted to hand out some stickers and honestly I had told someone "Oh, it's fine. We'll get like a couple responses. I'll write the envelopes. It's not a big deal." Turns out we got a lot of responses, but that's a good problem to have. And just what was so cool is that people were publicly sharing how they're using the software and there are themes, but everyone had a pretty unique way in how they were using Bolt to solve their problems whether it was for their job or for home automation - that was cool. And then it gave us ideas internally that we started thinking like "Oh, I want to automate a Nest with Bolt" or like cool fun after work things and what I really liked about that was one of our engineers was like I like Bolt because it's fun, not just because I actually have to solve a problem, but because I have fun using the tool and that that was just really cool to hear and see like all over Twitter/X.

Eric:So if people want to participate in that and get a sticker pack what should they do.

Yasmin:Just tweet how you're using bolt and add hashtag #PuppetBolt and we'll send you some stickers.

Ben:Awesome. I can definitely attest to the fun part of that. I actually just kind of for fun threw Bolt into a presentation that I did that kind of like automated a couple of different parts of demonstrations on some machines and it was really fun to do.

Ben:So we've talked about Slack a couple of times here. If you don't have an account you would want to go to slack.puppet.com and you can request an account there and then just stop by. There's people in both the #puppet channel and the #bolt channel that are always happy to help. Always happy to talk about problems, usually have some kind of answers or some advice. And every Tuesday right now we're doing office hours on Bolt and somebody will be there to help you get started if you need it. Later on this summer, if you happen to be in Europe we are running our annual Contributor Summit event - that's June 4th and 5th in Budapest and we're going to talk about a lot of Bolt things this year, so I would love it if I saw you there. We will have announcements coming out real soon about that. In the meantime I would ask for you to let us know about things that community members are doing that you would love to see them get recognized for at Contributor Summit and you would do that at pup.pt/mvp. Just let us know what what they did and if you have contact information for them, that would be great.

Yasmin:One other thing that we're doing this year and we're starting in the EMEA region but we want to expand out to the US are just Bolt workshops. They're like a half day session of starting with Bolt, using it to solve problems, learning it, and then ideally you leave the workshop ready to go tackle any automation problem that you have. So if this is interesting to you, reach out to us, let us know where you'd like to see it. If people are interested, then we can go from there.

Ben:That sounds great and they should probably just ping somebody in like the #bolt channel on Slack right.

Yasmin:Yeah, that would be great.

Ben:Thanks for showing up and hope to see you here next time.

Eric:Thanks Ben.

Yasmin:Thank you.