Managing Your Configuration Management System Shouldn't Be Hard

00:00:22 Ben Ford Hello and welcome to today's episode of Pulling the Strings podcast., as always, powered by Puppet. My name is Ben Ford. I am our developer relations director here at Puppet and I'm active in the community as @binford2k. We may have talked a few times in the past. Today we're talking with Edwin Maldonado. He's a solutions architect here at Puppet, where he helps customers improve their DevOps practices. And prior to joining Puppet, he worked as a software engineer and consulting architect in Europe. He says that he enjoys learning and being a home barista. So something that a lot of people may not know about Puppet is that we're very big on our coffee. We have had several commercial brew coffee roasters actually work at the company and they would bring their beans in and share. We have espresso machines in the office, and we even have a video on our internal confluence on, like, how to brew the best espresso.

00:01:18 Edwin Maldonado That's awesome. Yeah.

00:01:19 Ben Ford I think that this is a perfect batch. You know, it's like not only do we do dogs, but we do coffee. And then Edwin here appears to be an expert. So maybe you could tell us about your home setup for sure.

00:01:33 Edwin Maldonado Thanks, Ben. Thanks for the intro, and thanks for having me here. Hey, everyone. My name is Edwin and I'm not an expert barista, let's say, more like an aficionado. But yeah, I like coffee.

00:01:49 Ben Ford You sell yourself too short, there, come on.

00:01:52 Edwin Maldonado I'm a big coffee drinker. And, yeah, I have, like, several different methods to do my coffee, like, from filter based methods to the good old joke about the Italian tradition, let's say.

00:02:09 Ben Ford And that's the one that sits on the stove. And it just kind of percolates and boils over, right?

00:02:15 Edwin Maldonado Exactly. Yeah. It is like maybe like cowboy coffee. That's what I imagine. Like it is strong and it's like, wake up.

00:02:29 Edwin Maldonado Yeah, yeah, totally. Yeah. And well yeah, some day I actually would like to do that like on top of a volcano or something. Just like get one of these like camping stuff and just like.

00:02:42 Ben Ford Right, I'm a big backpacker. So when, when pandemic is done and we can like travel again and everything, we'll have to meet up and, and we'll take a hike somewhere. I'll bring the gear. You bring the coffee.

00:02:52 Edwin Maldonado Awesome. That's a deal.

00:02:55 Ben Ford So you started your career as a software engineer. How did you end up moving into infrastructure management? That seems like quite a trajectory.

00:03:04 Edwin Maldonado Yeah, that's right. So I started my career maybe like in 2008 or something like that, I was still in the university, but I wanted to like to have some money because I'm the kind of guy that likes to buy stuff that comes from my own effort, let's say.

00:03:27 Ben Ford You like to have the goodies.

00:03:28 Edwin Maldonado Exactly. Yeah. But I didn't want to ask my parents for that. So I wanted to make sure that I earned those goodies. So I went to like a couple of different companies. At the beginning of my career, I was just doing software development. I did some pure web development with PHP, JavaScript and mySQL back then, I help a couple of companies running like Magento sites and stuff, and then little by little I started to move into more like serious stuff. Like not only using like  people libraries could say, but starting to create something from scratch like an entire application from scratch. And I think that's where, in my opinion, things got interest.

00:04:18 Ben Ford I always loved building things. That was why I got into writing software. It was like I was always like the creative person who liked building and making and I'd like build a table or something. And then you're limited by the wood and the tools that you have and like the connectors you have. But with software, it's like if you can describe it in enough detail, you can make it. It's like you're not limited by basically anything but your own imagination. You can build what's in your head.

00:04:47 Ben Ford I mean that's a very, very fanciful description of it. And we know that it's not actual reality. But, you know, that's what I told myself when I got in software engineering.

00:04:57 Edwin Maldonado Yeah, but it takes a while, like you have to spend some time doing that, for example, like they're using other frameworks and stuff so you can get used to it and, and your mindset starts to change, I guess. So I ended up working with a couple of startups, one startup in Guatemala in particular. I'm from Guatemala, by the way.

00:05:23 Ben Ford Great hikes in Guatemala. I'm just saying.

00:05:25 Edwin Maldonado Oh, yeah, like there's huge volcanoes there. And I believe that like working at BlueKai actually like catapult me, if you can say that, to like other opportunities because like I had the chance to work with great people, like great managers, and I think that we were like doing stuff that we really believed in. So that kind of like, like empower us and it gets up from that point which was like more or less like 2013, I guess, to this point I have been like in my head like, like chasing down the rabbit hole like, like trying to learn more and more about how things work. That's after being a software engineer, I spent some time like doing consultancy. I work here in Berlin now. I live in Berlin. I work at a company as a technical architect. And I guess that that was another big step in my mindset because I started to think about how customers actually consume your product. So it's not just like you work in something and then you expose that, you make it publicly available, and then everyone starts using it. Because that's how I was thinking about like a product, let's say. So my mind started to change again, but this time more into the direction of, like, how customers really use your product. Like they use it in a different way than what it was designed for. Maybe, sometimes. My time here in Berlin helped me a lot. I love to grow into what I do at Puppet now. And I guess that I realize a couple of patterns, let's say, in working with. Like being a customer, being in a customer facing role, I notice that most of the projects were failing not because of the development per se, but because they were just deploying it into, more or less, like bamboo sticks type of thing. So yeah. Yeah.

00:07:48 Ben Ford So you just went directly into what this a solution architects team does. Like here at Puppet, it's kind of a pretty elite group. It's it's sort of like our, like our crack SEAL team or something that we send to customers to help them solve some of their biggest problems. And it sounds like you're just, like, talking about going directly there.

00:08:13 Edwin Maldonado Yeah. It's it's funny that you mention that, like, we have, like, a SEAL team. I truly like my team, like the solutions architects at Puppet. I enjoy working with them. Like these guys are some of the smartest guys that I have a chance to work with. And totally like I started looking at configuration management, more infrastructure topics because of that, because they realize that the customers were deploying stuff into fragile environments. Like it was a case for testing and development but not production rate. So I knew about Puppet Chef and all the configuration management tools that have been there since more or less ten years ago. But I never had the chance to actually be in a role where I was applying those concepts. So I decided that I wanted to learn more of those topics. And at the same time, I was thinking already into more like software architecture. Like why projects go well and what are some of the patterns of projects that fail? And that's how I ended up talking with Nigel, our CTO at Puppet and he's my manager.

00:09:48 Ben Ford It sounds awesome. So we're coming up on talking about a tool that you've built. But I kind of want to set the stage a little bit and talk about the background of why the tool is important. When I was in education, we built classrooms. We would put people in classrooms where they could sit down at a PE console and do the exercises with us. And so we did some of the earliest automation of standing up of Puppet Enterprise infrastructures, and we did a handful of different types of infrastructures. We had one server for the classroom and multiple clients connecting to it. We gave people their own machine, you know, like we did different iterations of all of these. And so we had to build all of these automations for doing this. And it was surprisingly difficult. We put a lot of blood, sweat and tears into making this work. And I don't know, I imagine that you have had a lot of engagements with customers who also have these very, very difficult challenges because, like, this is something that is inherently not easy to do, right? So what were the things that your customers were facing that sort of prompted the building of this tool?

00:11:15 Edwin Maldonado Yeah, I, I think that what you mentioned before, like we're trying to automate your Puppet infrastructure, being a structure, let's say, it is more or less like in a small scale what customers actually face every day. So. A little bit worse, let's say just because if you're a financial institution, they're like, teams of teams. So like you have not only technical challenges, but you also have communication challenges. And I think that those.

00:11:52 Ben Ford Are probably the worst ones. Right.

00:11:55 Edwin Maldonado I will say that those are, at least to me, like, very interesting problems like how teams behave, and the reason behind that, that's something that I find very interesting. But just to answer your question, the project that we're going to talk today, which is called the Puppet Enterprise Administration Module or PEADM, for sure, came out of necessity, I guess. And this is coming from direct customer feedback. And I didn't have the chance to be there in that moment when the project started because this happened way before I joined Puppet. But the customers were having issues trying to not only install Puppet, it's a thing that you will do, I will say that just a couple of times in your entire career as a Puppet administrator. And the reason behind that is because you usually upgrade that, hopefully, you never going to a point where you cannot upgrade Puppet anymore, like sometimes customers do. They're always workarounds for that. But between installations, it's something that it's annoying, time consuming, but hopefully you will only have the chance to do it once or twice in your Puppet administration career. But how do you maintain that? That takes a lot of time as well. How how you maintain your Puppet infrastructure, how you how you operate that, and how you make sure that it still serves the purpose that it was designed for, or at least it is still working towards your infrastructure strategy. Let's say. So that takes a lot of time. And that time sometimes it's like very manual and repetitive and you don't want to do that, especially if you have like a lot of support requests and you have other things to do.

00:13:52 Ben Ford In anything repetitive, it's like prone to error or at least not doing it the same way.

00:13:58 Edwin Maldonado Exactly.

00:13:59 Ben Ford Variances.

00:14:00 Edwin Maldonado Yeah, totally. So. Reed which is another solution architect from my team and Cody, I believe that they were the ones that started working on PEADM and they built the first version to install an extra large architecture. And maybe for the ones that are not familiar with the software architecture of Puppet, we have a standard architecture and we have a large architecture and we have an extra large. And an extra large architecture usually comes with compilers, maybe like a pool of compilers, and you have like a separate dedicated Postgres instance for your infrastructure.

00:14:44 Ben Ford And that's always been the hardest one to set up because things have to be set up in the proper order and you have to connect to them in the right order and everything.

00:14:51 Edwin Maldonado Exactly. Yeah. So this can easily take you, I don't know, like a couple of days I will say like depending on the level of permissions that you have, maybe you need to request something else. But this takes time. If you have something that can easily do it for you, and it works every time, I guess that would be awesome to have, like just thinking out loud. So that's the first purpose of PEADM like taking the lifecycle actions related to maintaining and running Puppet interface, like provisioning or installing and running updates.

00:15:33 Ben Ford Well, that's perfect because you didn't design it just to like install. It wasn't just a script to install the thing. It's like administration tasks for maintaining it over its entire lifetime.

00:15:42 Edwin Maldonado Right, exactly. I believe that maybe at the beginning it was just like that, maybe like an MVP, let's just create this that installs like an extra large architecture. But then it has been evolving and now we support upgrading to different architectures. And if the project was not created with PEADM, you can convert your whole infrastructure to something that works with PEADM and then you can easily upgrade to the latest version of Puppet.

00:16:20 Ben Ford And then you said something about Platform I think a moment ago. It also helps you when you're upgrading the platform that it's built on, right? And the reason I ask is that we just did a podcast with Paul, who was talking about is automation tools for helping you upgrade your systems.

00:16:42 Edwin Maldonado Yep, I would think so. I think that the main goal and I guess that this is going to be like maybe like a pattern with the people that work at Puppet, we all have automation in our minds. And that's kinda cool because like even though we work in different teams, we have the same goal. Like, it is like a shared goal, right? Like we are working on different tools in different levels to help the customer at the end, like doing better at managing their infrastructure.

00:17:17 Ben Ford So how did you end up getting involved with the project?

00:17:21 Edwin Maldonado That's a that's a good question. And, sadly, it's not going to be like a wow type of thing. So when I joined Puppet, I knew what Puppet was. I believe that I had a chance to play with Puppet in 2013, and it was different. It was like maybe version three. I guess it was.

00:17:48 Ben Ford Yeah. We've grown and changed.

00:17:49 Edwin Maldonado Yeah. Now we're better. And so I already knew how Puppet works in principle, let's say. But it was like super new to the idea of Bolt. I didn't know what Bolt was. So when I joined Puppet, one of the first things that I remember that I was involved with was like collaborating with this thing called PEADM, which is a module that we have been discussing today. And the first thing that I did, the first thing I remember was to make sure that the download process was more reliable. Inside PEADM there is one task in the specific that is in charge of downloading Puppet.

00:18:39 Ben Ford Like getting that agent installed or whatnot.

00:18:42 Edwin Maldonado Basically just like downloading the project to start with, I guess so you have the link to download PE.

00:18:54 Ben Ford Oh, so you're saying that this tool not only helps you install and manage and maintain Puppet, but you don't even have to download PE to begin with. It'll do that for you.

00:19:03 Edwin Maldonado Oh yeah. Yeah. It is an option actually. Like PEADM works based on parameters. And as you can pass a different list of parameters, maybe we can like share the link later with the audience. But one of those parameters is the download option. We are actually working, we have a ticket for that to allow the customer to specify the download link for Puppet Enterprise. But if you want to just like run it as it is, it will just download Puppet from the source and it will do it for you. You don't have to do nothing other than just basic parameters that you need and you will do.

00:19:48 Ben Ford That's really cool.

00:19:49 Edwin Maldonado I believe that. That's awesome. And if, for example, for a standard architecture, if you will do it manually, you might think you need one day, let's say more or less, but if you do it, if you do it with PEADM, it will take 25 minutes, maybe. That kind of gives you the idea of the power of PEADM if installing it's like 25 minutes and then I end up with a full Puppet interface system running. Wow. Like I can have more time now for other important stuff.

00:20:25 Ben Ford Yeah, that's really cool. I think I got us derailed talking about another really interesting topic although that's really awesome to hear. I didn't know that PEADM managed all the downloading for you as well, but it sounds like you're saying that maybe working on PEADM and doing some of this was like your onboarding to the Solutions Architect team.

00:20:48 Edwin Maldonado I will say that it was not officially my onboarding, but I had a chance to actually learn what Bolt was because of PEADM. So long story short, I had this task which was to get on JIRA in our team to modify that task inside PEADM. And the first thing that I realized was like, I'm not sure how Bolt works. I had the idea that you can use Bolt just to run commands on remote targets. And I say like, okay, so that's kind of cool, but I didn't know that you can write tasks and plans and what's the difference between those things and so that was all new to me. I would say I finally learned what Bolt is and how it works. And it's pretty cool. Not only because I work at Puppet, obviously, but I believe that Bolt actually has a lot of potential.

00:21:54 Ben Ford I do, too. And you say that thing about plans? Sort of. The way I see plans is, is like an infrastructure wide application that does orchestration tasks for you. And you just write the different nodes that you want to do different things and then they just coordinate together and all result in the action that you are trying to go for. I think Bolt is really cool. Could you tell us what it is about Bolt that makes it so suitable for the tool that you're working on?

00:22:25 Edwin Maldonado Sure. Yeah. I would say that Bolt is powerful for maybe two things. The first one is it allows you basically just to execute stuff. Like it could be a scrape or just like a command in a list of targets. So you can just pass those parameters to Bolt and and Bolt will run it for you. That's like the basic use case. The second one, and I believe that's one of the coolest, is that if you're already a PE user, you don't have to like do crazy stuff just to deal with access and security. You can run Bolt from the Puppet server and access the whole node list that you manage without dealing with like, like security credentials and stuff. Yeah. I think that's the best thing.

00:23:19 Ben Ford It's really cool. It uses the PCP transport layer that Puppet uses on the back end. So once you have PE installed, you can just tell Bolt to use that and you can do it from the console or you can do it from the command line. It's real easy, straightforward.

00:23:35 Edwin Maldonado Totally. And you mentioned something a couple of minutes ago. Plans and tasks. I think that when I first was working with PEADM, I was confused a little bit about like, what's the difference between tasks and plans? I thought it was the same. And from talking to customers, I believe that some of our customers have the same questions that I have. So for example, just to clarify that my software engineering view, let's say I can see a task as a method or like a function. You write a task to execute one single thing. So it is more or less like a single responsibility type of view. And that's what a task should do. A task should only do one thing. Like one single responsibility for that task. And that's it.

00:24:41 Ben Ford And that's why we have plans. Plans that pull them together.

00:24:44 Edwin Maldonado Exactly. And then plans. Plans are like a class. If you're coming me from some software engineering, a plan is just like a class. You have a bunch of like business logic if you want to. And from a plan you can call different tasks. That's the main difference to me.

00:25:03 Ben Ford The plan is the thing that wraps it all, pulls it all together and kind of orchestrates everything. I'm working on some Relay workflows right now and building some integrations and it's exactly the same kind of thing, is like how much fits into this particular step? It's one single thing. It's like one composed task that has to get done. And then the workflow itself is what wraps all that together. And this is same thing with tasks and plans. It's like thinking about these things in composable way because once, if you build your tasks so that it does all the different things, like you said, like things out to Slack and it does database administration and everything else, then it becomes very single purpose and you can only use it for that one thing ever. But if you make it to be a single purpose thing, then it's composable. You can stack them together and you can make other things that use that same function.

00:25:59 Edwin Maldonado Mm hmm. That's exactly how how I see things. And I guess that, as you mentioned, if you want a task to be reusable, having that single responsibility in mind helps a lot. And it's better if you do it from the beginning.

00:26:13 Ben Ford And to me, I think that automation itself isn't always, it's not just about like working more efficiently and saving time and doing stuff faster and whatever. It's about making things repeatable and provable. And so you can expect the output and there's less, less variance along the way so that you don't end up tracking down and troubleshooting later. It just lets you take your process and codify it so that once you start it you know exactly what's coming out on the other end and it seems like PEADM is like the perfect example of that is like not only do you have all of these, these things that are saving you time, but you've got these tasks that are going to give you the same result at the end.

00:26:59 Edwin Maldonado Yes, totally. And I'm glad that you mentioned that, because if someone in the audience wants to I don't know, like take a look at PEADM, go to the public repo. Everything is public. And you will just like read the plans, like read the basic one, which is that install plan. And you can see all the different calls to the, to the tasks that we have. And you will see that single responsibility idea behind it. And it automates repetitive tasks, as you mentioned. And I think that that's very important for our customers.

00:27:37 Ben Ford That's cool. Is this module open source? Are you inviting people to contribute or poke at it or anything?

00:27:47 Edwin Maldonado Yes. I think that this was like, yes, since the very beginning, we still believe that, we welcome contributions. There's one thing to keep in mind, though, is if you, as a customer, if you're a customer in the first place, and you want to use PEADM, it will be better if you talk to your TAM or maybe you talk to your SE first and we can give you an intro. So we work a lot internally to make a PEADM module that is supported at Puppet, but it's important that we introduce you to the project if you want to, let's say.

00:28:29 Ben Ford Like training around how to use it.

00:28:34 Edwin Maldonado I will say that it's not really training, but sometimes customers think the PEADM is for something else. And it's important to set expectations. That PEADM helps you to install upgrades and basically maintain your Puppet infrastructure. So I think that it is always helpful, I guess, that a TAM or like a solution architect introduce you to the project, but if you want to contribute, feel free to do that, we accept pull requests and we are happy to review that.

00:29:08 Ben Ford So if people are taking the same kind of approach that you've been talking about of building this and trying to automate a new process or whatnot, do you have suggestions for how to get started?

00:29:21 Edwin Maldonado Yes. I think that this is always tricky, let's say, because sometimes you don't even know what are your sources of toil, let's say. And I remember that toil is like that repetitive task that is not helping you or worse, your strategy or goal. But it just takes time like it is just like maintaining work that it will not help towards the goal of the company. So I would say that if you want to start automating your processes, it will be a good idea  if this is like a team effort, you will go further if you take that approach. Like talk to your team, maybe you can start having like a brainstorming session listing all the topics that you do every day. And maybe from there, you can discover that, huh? You know what? Maybe we can just ignore this step or, like, write a script for that, and maybe that's how you can start. So map your processes, even if you think that's something that cannot be automated, let's say. But maybe someone else will have more information or more context and you will end up automating that thing.

00:30:45 Ben Ford And by map your process, basically, you mean just kind of write down all the steps that you're doing and kind of collate them with other people on the other teams and all the things that they're doing.

00:30:55 Edwin Maldonado I don't want to say simple because yeah, nothing is simple. Right. And that sound was, was me just like hitting my hand like, nope. So I will say that that's one way to do that. And this, this might sound weird, but like some of the stuff that we do in the Solutions Architect team is to help customers map their processes. So what we do is talk to the Puppet infrastructure team on the customer side. We try to invite everyone that has knowledge about their infrastructure and their set up. But at the same time, we try to invite managers and someone that has like business logic, and we take that, like almost like a domain driven design approach, we set everyone in a in a Slack nowadays because of COVID, but usually it should be like in one room. We have that room with a lot of stickies and we start mapping what they do. For example, they can start with mapping their current module development and deployment. That's one process, right, where they actually do processes like developing and then deploying. If you map that, you will find things that you can fix or automate or like improve if you want to. But you have to start somewhere. And that's one of the ways that we usually recommend customers do.

00:32:32 Ben Ford Oh, well, let's go ahead and wrap this up, I suppose, close up now. I think that the lessons that that we can take away that you've been talking about are really straightforward. And it's like when you've got a manual task eating up your team's time, it's worth looking at automation and and the key is capturing all of that institutional knowledge that you have into some kind of processes that are documented or mapped out and just kind of make sure they're in alignment with all the teams that are involved. And then once you have all of that, like concrete and solidified, then it becomes a whole lot easier to look at that and just automate the layout, pull out the different parts that you can automate that now and put them together later on. Do you have anything that you'd like to add to that.

00:33:24 Edwin Maldonado I think that was a great wrap up wrap up.

00:33:31 Ben Ford Yeah, it's really just about like taking abstract ideas and just kind of like making them concrete. And once they're on a piece of paper or on stickies or on a whiteboard or whatnot, it becomes a lot easier to, like, look and see pathways through and turn those into tasks and plans and or whatever it is that you're using to automate. And those become very, very easy to write.

00:33:55 Edwin Maldonado Yep. And it's way better, as you mention, if the whole team can see that document or that artifact or like the sticky notes on that whiteboard, because it's easier to have conversations around that than just like talking abstract things in a Zoom video call, that helps a lot.

00:34:17 Ben Ford Cool. So if people have feedback or ideas, you said that they can try out the module and like file tickets and whatnot and we'll put links to it in the show notes. But it's on the Forge as just   Puppet Labs PEADM. Do you or other team members hang out in Slack or any of the places that people could talk to you or maybe Twitter or other social media?

00:34:45 Edwin Maldonado I believe that the whole team is in that is in the Puppet community in Slack. You can find me there, it's very easy to remember my nickname there, it's just my name, Edwin. You can also send us an email which is SA@puppet.com and you can find me on Twitter. I'm usually like active, let's say, on Twitter just sharing ideas and my like Twitter handle is @MCInside.

00:35:21 Ben Ford And we'll put links to all of that in the show notes. So you don't have to be like frantically scribbling things down right now. And that is a wrap for today. And once again, thank you for being here, Edwin, and thanks, everybody, for listening and thanks for being here on Pulling the strings,  powered by Puppet.

00:35:43 Edwin Maldonado Thanks, man.