Introducing Puppet at Scale: A Panel Discussion on IT Automation and Model Trains

00:00:00AndrewHey everyone. Welcome to today's podcast. It's a special one. We're going to talk to you today about the Puppet at scale project because sometimes big enterprise class challenges need to be solved at a smaller scale. In this case we are throwing Bolt one of Puppet's open source tools at an HO scale model train set. We'll be covering who is involved, we'll get to know some of the crew, we'll talk through some of the interesting discoveries, challenges, get a sense of where this project is heading and let you know how you can keep tabs on it. So first and foremost let's meet the crew.

00:00:49CasHello everyone. My name is Cas Donahue. I am an engineer on the Bolt team here at Puppet.

00:00:55KenHey my name's Ken Johnson. I'm an engineer on the infrastructure core team here at puppet.

00:01:00JacobJacob Helwig Coremunity team.

00:01:02AndrewAnd I'm Andrew Nhem, managing editor of puppet.com. We are an interesting crew, folks from different teams and whatnot coming together to work on this awesome project. We definitely want to give a shout out to folks who aren't at this table who are also giving a lot of time and energy to making Puppet at scale happen. Those are folks on our business technology operations team, the workplace team, our greater community, more engineering folks and the marketing team. So what is puppet at scale for those of you who have been paying attention, one of our open source projects is called Bolt and, Cas, Bolt is, you probably have the best definition of the four of us, go for it. Give your plug.

00:01:38CasI think for this audience when I describe Bolt I'd like to frame it in sort of how it compares and extends Puppet. So with Puppet we have an amazing way to declare state and make sure that it's continuously enforced. Sometimes you need to do an imperative action, one off actions just to do a sanity check on something or something needs to happen right away. So what Bolt does is it allows you to make those imperative actions happen in a way that integrates well with what's happening with Puppet. We can affect change or make queries about your infrastructure using Bolt and we can use, we can talk to those pieces of infrastructure with different transports so instead of just the Puppet transport we can use SSH or WinRM or others. This allows us to have a bit more freedom and to play around and actually figure out how to solve things. The other great thing is that sometimes Bolt will allow you to orchestrate things by doing things in defined order and also allows you to declarative changes as well and keep your infrastructure in a good state.

00:03:05AndrewThanks Cas, that wass spot on, definition of Bolt. Throughout the year we've been running workshops, spreading the good word about Bolt all around the world and randomly around the March timeframe in which every technology company is wondering what funny clever things should I do for April Fools, A group of us got together and you know those of you in the community, you know Nigel and Ben Ford they randomly kept talking about solving Puppet problems at a smaller scale. They kept having this idea around model trains and we'd be like yeah okay whatever. After about the fifth time that was brought up someone in a brainstorm session made the connection with Hey why don't we, could we possibly tell a train what to do with Bolt tasks? Obviously at that time the answer was I have no idea. And those are often the fun things to software: we have the time and at the time in March had no idea this would work but there is a lot of energy in forever around it and we kind of knew in our gut that we had to try to figure it out.

00:04:07KenYeah that's kind of a an idea that's been kicking around Puppet for a long time. I remember several years back when we moved from when we moved into the Lovejoy office I want to say, someone used their part of their departmental redecorating budget to buy some stuff to set up a model train set but it never really got off the ground then. But people have been saying we should do it for a while and it's really neat that we got the opportunity to do that and integrate it with one of our products.

00:04:34AndrewYeah that's going to be neat. Whatever happened to that train V1?

00:04:40KenYou know I think it kind of just disappeared after a while of never being assembled. There were a couple of projects like that that sat around for a while collecting dust and then didn't make it through the next move.

00:04:54AndrewSo hopefully we can make train V2 stick around.

00:04:57KenThis one already looks like it has a lot more traction to it.

00:05:02AndrewWell I wasn't really upon the traction track close enough. Whatever. For those of you listening I will throw in a couple of images and whatnot into the posts that holds this episode so you can get a sense of where we're at. Definitely work in progress. But a fun one at that. What's been pretty neat is that you know you probably have seen Puppet's State of DevOps Report. You've seen all the work that we do to empower DevOps managers or teams to do more to improve their agile practices. This was a pretty neat, different way to live a bit of that. We're bringing different teams together. We're trying to solve a big challenge together. And like Ken said it helps us integrate our into some of our new products and solve business call. And we want people to try Bolt and learn how to use it, and what better way than irreverently through a model train. Specifically HO scale. For the folks listening who think that all model trains are Fisher Price size that's definitely not the case. Can you weigh in a bit on how broad this hobby really is?

00:06:04JacobA massive hobby. I mean there's, so you mentioned we're using an HO scale train. It's a good size and a typical layout on those is like maybe 15 by four feet. So they take they take up a fair amount of room to do like a small town or a couple of small connected towns. But I mean model railroading itself there's ones where a typical locomotive is like the size of your index finger as far as how long it is all the way up to ones you can ride around in people's backyard gardens. So there's a huge variety of scale and complexity in Model Railroad layouts. We're doing a pretty simple one to start out with because we wanted to have as few variables that we had to work out the kinks in as possible.

00:07:03AndrewI remember when we were first doing discovery on this project like not even the technical requirements we wanted to get some advice from Portland's local model train club, and this is a serious hobby. I mean they were actually concerned that we were going to start it incorrectly. We approached this career I think a handful, of Ben and I believe A.J. Johnson one of our Engineers, went over to give some advice and I mean they're also just very enthusiastic. I think they wanted to get involved as well but it is interesting to hear the reporting of hey you know make sure you kick this off properly and four months into it, I kind of get the sense of why they gave us that warning. Things are going smoothly but we've been learning a lot of things. The most surprising thing for me is obviously we've been testing the train and we've been experiencing some derailment. And obviously we're fixing that you know yay iderative improvement. Whoo. But I was surprised to see was a site that we found on HO scale train requirements. And at first glance it actually was well styled and written, I thought it was a like a real dot gov like Metro site.

00:08:07JacobOh you're talking about the NMRI the uh...

00:08:10AndrewYes. Like I thought I was reading a site for civil engineers or like train engineers to like actually like architect real deal trains and layouts throughout like a metro area. But it's a well done model train site. This is how serious this is.

00:08:24JacobYeah, oh sorry. Not NMRI an NMRA, National Model Railroad Association. They have a lot of really good and really in-depth recommendations for how to do things at scale so that they look realistic. And it's kind of mind boggling the level of detail they go into in their standards and specifications.

00:08:46KenYeah I kind of like that we're going a slightly more whimsical with ours just the wildly out of scale bear stalking the countryside around the layout, kina. I just really appreciate that.

00:08:58AndrewYeah, there's like a district of the train track that's like populated by Happy Meal toys I believe from the last 20 years. Someone donated all those

00:09:04KenOh man, I'll gonna have to go on eBay and get some good ones.

00:09:07AndrewYeah. I think my mom might still have like all of ours in a hutch somewhere in her house.

00:09:14KenI remember those like weird Transformers ones, so you would have your burger backs that turns into a pterodactyl and stuff.

00:09:19AndrewThere you go.

00:09:19JacobLike a box of fries.

00:09:21KenYeah that one.

00:09:22JacobBut yeah I definitely agree. I like that we're not doing like this is a hardcore super realistic model railroad layout. So yeah we're ignoring large swaths of what it NMRA has to say. But at the same time they have some really useful things like here's recommended wire gauges for doing the track wiring and here's recommended radiuses for curves to be able to do this type of railcar and have it be somewhat realistic. And probably not derail.

00:10:01KenMm hmm. Yeah. We've got the rail model railroading equivalent of best practices document for us setting up some new systems. It's very handy. Make sure we don't trip over our own feet. Trying to make this thing happen.

00:10:15AndrewYeah we're doing it the right way but in a very welcoming inclusive way which I absolutely love seeing for those of you who've been paying attention at Puppet this year, you know we've got refreshed leadership. There's definitely been a resurgence of like wanting to work together towards toward big company goals. One of those goals is to get folks around the world to know about Bolt. Yeah sure we can come up with more workshops and webinars and different ways to promote ad campaigns and whatnot. We wanted to try something different. And it's been pretty magical to watch different teams come together to A. try and complete this train set for those of us who don't have the engineering chops then also B. To see folks come together to try and make this happen. So I'd love to hear around the table you know where are we at now what have you learned and what's keeping you excited about this project?

00:11:10JacobMostly I've been concentrating on getting the lay out talking to a computer so that we have some API to interact with from the Bolt side. And it's been really interesting learning what what software is already out there for doing things like that. So we're we're using a system called DCC, digital command control, for doing model railroad control and it lets us have multiple locomotives on the layout and control them independently instead of the really old classic straight up DC system where you just apply DC voltage to the rails and whatever voltage you apply is however fast whatever motors are hooked up to those rails is going to run. So DC lets us do things like have multiple locomotives on there running at different speeds in different directions and all of that, control switches, control lights and all kinds of other things from a central point. So from there we go into a computer which is running JMRI, which is one of the software packages that lets you do software control of a layout. And it's been interesting seeing exactly what the mindset they expect a user to have when interacting with it. They very much expect you to be running a scale railroad, so everything is kind of pre laid out where you have from point A to Point B is something that's predetermined because you have two industries that are connecting with each other and you want trains going between those two places. So there's not really a whole lot of on the fly dynamic creation of things like that. So one of the challenges we have ahead of us is figuring out how to get something that can do a little bit more of the dynamic route generation and scheduling of things. Talking with JMRI so that we don't have to build out a table of every possible point A to Point B combination and basically so we don't have to have people who are going to be interacting through this with Bolt have absolutely intimate knowledge of the layout itself so that they have something a little nicer to work with. So yeah that's that's one of the big challenges we've got ahead of us and what we've got going on.

00:13:37CasYeah in that sort of vein from from understanding how we would put an interface to this using Bolt, something that has become pretty clear to us is the system that controls the train is a very good use case for the concept of a remote transport that Bolt has. So when you have things like a network device or an HDGP endpoint, those things don't necessarily have a full shell environment like a server or something like that. So what the remote transport allows you to do is to run the logic bits on a proxy device. So if you had a fleet of network switches that you needed some kind of configuration on you could use the bolt remote transport to manage those devices from a proxy with just by using the endpoints that they have defined. So something that we're investigating is where do we put in the declarative, this needs to be this way, perhaps the layout of the switches on the track or where the train should be. Versus things that we just want to happen quickly like sounding the horn or turning on a light. So these problems are a perfect example of how DevOps people would be needing to control things that are both have a declared state on them versus stuff that they'd need to just do one off quickly and also to use something that is not traditionally a node in that it would have its full shell environment and you could interact with it with like package managers or service managers or something like that.

00:15:36AndrewIt was cool to see the whole company get together, what we did was you know we didn't promise any kind of operational train but really like hey come hang out this come add some crazy art. Put some model train stuff together, paint the train. It wass really cool to see folks of all walks of life and all these stories pop up too. Apparently our CFO is huge into model trains, walked around upstairs, the sixth floor at Puppet HQ. Well that's misleading we only have two floors. I made Puppet HQ sound like massive.

00:16:10KenOur downtown Portland Campus.

00:16:10JacobSeven story Builiding.

00:16:11AndrewYeah. So it's been really cool also is to have the blessing of the CEO. Yvonne was at the train lay out a couple of times taking pictures of the train. She really wants to learn how to use it apparently so she can impress her kids which I thought was pretty cool.

00:16:25KenYeah that getting the blessing of the CEO part is always a little bit important for these kind of little side project things. I was one of the instigators behind having arcade machines in the office and I was always a little like lukewarm on how I felt that company leadership thought about that until one day I was playing Samurai Shodown 11 on the Neo Geo and Luke Kanies walked up and tapped in on player 2 and just like spent 20 minutes kicking my butt in that game because apparently he had grown up playing that in arcades.

00:16:59AndrewA new challenger appears.

00:17:00KenRight? It's true and I didn't expect it. It came out of nowhere but after that I figured he was probably pretty warm to the idea of having that stuff around. It was kind of fun.

00:17:09AndrewWe'll throw a picture of the Puppet arcade did that thing is going strong. And I see it's pretty cool to watch people of just all walks of Puppet life come in and hang out and play. It's funny to see all the guests ask if they need quarters.

00:17:25KenI found a few in a few of those machines. Every so often you'll get 50 cents in it and it's like oh someone's kid must have come in.

00:17:31AndrewDoes that go to your maintenance fund?

00:17:32KenYeah. The arcade slush fund.

00:17:36AndrewBut it's been great to have people ask questions and what not. We've had a couple of guests and whenever they walk by the train you know they always asked What is this what do we do with it and by the time we tell them the whole premise of what we've got going on what the big vision is they get really excited. Especially when those guests are Bolt users. We had a customer in last week and he was looking giddy and I'm like Hey this isn't ready yet. We'd love your help when it's done and like you know you can feel free to kick it around and test it but wee're not there yet. But he definitely wants to keep tabs. But yeah, International Train Day or the day before. It's just great to see different folks from different teams you wouldn't normally see. There was lot of painting going on. Someone had some like 70s jams going.

00:18:18JacobIt was actually a good time along the lines of the painting and the art. I really like a lot of really like a lot of the graffiti style art that people have been doing on the box cars and things like that. It looks it looks authentic without being like overly pedantic about what goes on there. It just it fits really well. I just think it looks really nice.

00:18:44KenI like it. We get to see firsthand the flexibility of the DAG placements and layout designs. I love that we have a whole design document on that like with the rules of how an extension of the DAG logo can flow. It's kind of kind of interesting but it's clearly something people kind of intuitively grok and understand how to extend it in a way that makes sense. I don't know haven't have any of you read that thing when it launched?

00:19:14JacobI saw the presentation when they were talking about it but I didn't dig into it. My art skills are anemic at best.

00:19:23KenMine are too, but it was fun to read this like clearly super thought out doc from people who like you know think about this stuff professionally and like you know for for being something that you know to someone who's not initiated into you kind of thinking about design and that way they like might kind of mock it like Oh you paid them how much to design that thing but then you read it and read the like thought process that went into it and kind of the way it's been that this simple image has been articulated in words and the instructions to expand it. And it's like Oh actually that yes that is that is worthy of the stature it has as it's own like special discipline.

00:20:02AndrewAnd the dag that we're talking about is the logo of Puppet. Dag. I always forget with the D is.

00:20:09KenDirected acyclist graph. Yeah yeah.

00:20:12AndrewDirected acyclic graph. Check it out. Or just go to puppet.com you'll see a dag. You'll see many dags. I'm totally looking up train trivia right now. Did you all know that the term horsepower originated as a marketing tool.

00:20:25KenThat doesn't surprise me.

00:20:27AndrewThen back in the day when America was first testing locomotives. Guess who won the locomotive or the horse. It was racing against.

00:20:34JocobI'm guessing the horse because the train broke down.

00:20:38AndrewDefinitely the horse won.

00:20:39JacobMaybe if it was a role race.

00:20:41AndrewAnd apparently the travel agency industry was kickstarted by a train trip.

00:20:47JacobYeah we have railroads to blame for time zones.

00:20:50AndrewYikes yeah. The railroads also gave us standardized time zones.

00:20:57JacobOne thing that I actually found really amusing doing a lot of the train research is that a lot of model railroad information out there refers to prototype railroads. Prototype railroads are not the HO scale or the end or the Z or any of that. They are the full size ones that you get stuck behind at a railroad crossing. They refer to those as prototype railroads. They run the real railroads at their smaller scale. So I found that to be greatly amusing when looking through all of the documentation and everything else. Like I kept seeing this like prototype railroad prototype railroad. Like what are they talking about. And then I finally found something that actually put two and two together for me. It's like oh they mean the the real ones.

00:21:43AndrewBecause it's easier to scale down right.

00:21:46JacobYeah I mean in this case, but they look at it as like the prototype railroads, the real railroads, went out there and they figured out all of the things that don't work so that when they go run their railroads on their layout they don't have to go relearn everything that the prototype railroads figured out. But also like with the expectations that JMRI has, like they expect like the typical thing that people do with Model Railroad layouts is they run a railroad. So they might have other people over and operate on that railroad as well. So they've got a dispatcher and people running individual trains and things like that. So they're running a railroad on the layout. So that's why all of JMRI expects that you've already you already know what the industries are you already know everything else you already know all the interconnects that are gonna be happening. So that's why they just kind of like well you just predefined those and then you'd say that well train number whatever is going to operate on that route or that transit right now. So you don't have to come up with one on the fly.

00:22:54KenIt's kind of taking this thing that in the real world is already pretty regulated and regimented and then like boiling it down and getting this even more perfect expression of the relationship between these things in the physical world and what's connecting these industries across this super complex Infrastructure we built for it.

00:23:13JacobYeah pretty pretty much exactly. And because that's not what we're trying to do. We're running into all kinds of edge cases where it's like well but we don't know where a person is going to want to send a train.

00:23:26KenRight? They have no way to express the fact that someone might just decide to say Oh actually I wanted to go the other direction now midway through the run.

00:23:34JacobYeah. That's that's just not a thing that you would do on a railroad.

00:23:38KenYeah.

00:23:39AndrewBut we're running into edge cases together.

00:23:44JacobYeah. A lot of it is like a lot of the challenge going forward is just figuring out how to smooth over those things so that people don't have to have intimate knowledge of our layout, intimate knowledge of how railroads work, intimate knowledge of all of the little sections of track and how they're labeled in the system. We don't want someone having to download a several hundred page PDF with different combinations of labels on it just to figure out like all right, so how do I get the train to go from this corner of the layout to that corner of the layout.

00:24:14CasYeah. What comes to mind for me relating it back to Puppet is something we haven't talked about yet which is Puppet as the sort of enforcement tool is one thing but then there's the module content available on forge which takes complex systems like a load balancer or a web server and we write an abstraction over it that anyone can use. And with everyone sort of like combined knowledge about the thing we make something that anybody can come and just set a couple parameters and boom they have it stood up. So it's really a nice sort of parallel here. We're taking a very complex piece of software and writing an interface to it, allows you know, just someone who's never operated a model train even with a controller let alone a computer, be able to issue some simple commands and see how that works.

00:25:09AndrewThat's a great segway to a question that I want to ask you, Cas. I'd love to hear your thoughts on you know let's say and this is we're a bit a ways from this this greater vision near end state. But tell the listeners a bit about you know where you see this going, how you want folks to interact with this, with our model train.

00:25:31CasSure. So I would love for existing Bolt power users to find it interesting enough to write beautifully complex code that does amazing things. And I'd like it to be as easy enough that someone who's never even touched Bolt could install it and at least make the train go around the track. And so I think that one of our sort of design goals is to make this a a good reason to understand sort of how, why Bolt is so powerful. We'd like to use this as an opportunity to illustrate for both technical and non-technical users and a really powerful important part of bolt which is to effect change on systems that aren't classically a full functioning computer with a shell environment or a server. This could be like an HDPN point or a network device. And by relating that to a train that someone can see and anyone can understand. If I do X, Y happens. That will allow them to see, sort of make that comparison in their mind that while this tool can help me control things that aren't typically controlable by your traditional enforcement models or full, I guess computing environments. So yeah I think that the design goals are to get people familiar with Bolt and show them something that they can really start to appreciate what the tool can do for them.

00:27:23AndrewIt's awesome. On the topic of learning what's been pretty exciting to me to watch our different teams. You know originally this this case was hey we want folks to learn about Bolt in a new way, a different way, a fun way. That was our only ask and we came up with all kinds of other use cases in which folks wanted to solve their very Puppety way. For instance you know in order to fulfill that greater vision we've got a couple network security hurdles to overcome. And it was awesome to loop in our business technology org, BTO, as it's known internally and they're all about it. They wanted to know what we're up to, what's going on. They're on their way to solving that problem.

00:28:03KenI'm mostly just excited to see, as Jacob is and has been working on what the interface between Bolt and between this digital controller ends up looking like just because that kind of integration stuff is sort of the bread and butter of what I do in my own role and also generally the interesting part of the problem to me. So that'll be neat. And also just like once the thing is operating and you can throw a bolt plan at it and have it move around and do neat stuff like what's the next steps from there. Add on some additional things besides the training that you can you know execute these plans against and kind of make the landscape change around it. And also just like what is the artistic expression of the crew here at Puppet going to look like as we continue to build up the layout and apply everyone's you know talent and vision to creating the set around it.

00:28:52JacobMight be a spoiler alert but the answer to one of your questions is yes we are actually planning on having like activate-able scenery and all kinds of other things like that.

00:29:00KenCool. Yeah I thought I thought I had heard of that and probably talked about it with a couple people.

00:29:05AndrewYeah I heard the rumor of that sea monster or something like the Kraken or Cthulhu depending on which giant sea monster you subscribe to.

00:29:15JacobI hadn't heard that one but that'd be cool to see.

00:29:20AndrewPopping up and saying hello. On the note of expression I'm pretty stoked from a communications standpoint to find different ways to package, it is probably a longer term goal but, a new way to teach folks about Puppet. In this case Bolt. Yeah you can sign up for training. Yes you can read our many blog posts. We're very education oriented here. You can check out the docs and get started in the plethora of ways. But this is just a different unique way to do it. It would be so fun to have classes predicated upon Hey can you send a task and our plan to tell that train what to do. Or perhaps tell the kraken to appear.

00:29:52KenYeah I think that kind of thing is super helpful to have, you know kind of a physical expression of what you've done in this digital format. Especially just with technologies like like ours and within the space we operate in. You have these super super complicated things you know hidden from kind of direct human understanding behind many layers of abstraction and you know being represented in this way that is you know somewhat unnatural. You know typing things on the screen. It's not the most intuitive. So being able to have that translate back into output that something that is very much in the physical world and you can you know see and touch it and observe what it is doing in reaction to what you've typed into the computer, that's a super powerful learning aid and way to you know make it clear hey actually you know this is still real stuff even though it's happening in a data center somewhere.

00:30:48JacobActually one thing that we haven't mentioned yet but it's kind of important is how people would know that what they've said to bolt or through to the layout through bolt it's actually doing what they wanted to do. And I mean we're planning on having a couple of cameras pointed at the layout with a livestream set up for it. And I mean there's a lot of details to work out but the idea is that there will be a livestream of the layout with some sort of identifier so that you can know that it is in fact your commands affecting the layout right now instead of whoever was playing with it just before just after you. The goal is eventually to have a way for you to see definitively that what you're doing through bolt is having an effect on the real world train layout.

00:31:37AndrewPerfect segway into us plugging where you can go to. For now just stay tuned and get some updates on what we've got going on with this Puppet at scale project. You can always check out the latest footage that we have from builds, pictures, commentary, what not, just go to puppet.com/puppet-at-scale. And with that I think we've covered all we need to today. There's so much more exploration to go. Cas, Ken, Jacob thank you so much for taking the time to hang out today on the podcast. Folks internally, externally, we're all really excited to see where this goes. I'm already proud of the crew. I mean just this is the first company I've been out where like different departments are all coming together. You know sans project management. There's no Jira involved. It's just straight up. You know let's hang out together, let's be accountable. Let's do some cool stuff together. It's been pretty awesome to watch and I'm looking forward to how the rest of the year plays out. We'll have another episode I think when we hit a new milestone and you can kind of keep folks updated on what's going on. But yeah check it out. Puppet.com/puppet-at-scale. You wanna learn more about Bolt, head over to puppet.com/bolt. And with that everyone thanks for listening. Have a great day.

00:33:00AndrewI keep calling the podcast "The podcast" but it actually has a name now. It is called pulling the strings and you can find it at puppet.com/podcast. You'll see all of our more recent 2019 episodes, some older ones and you'll start seeing that name throughout different ways to find our podcast iTunes etc..