The Best Part of the PowerShell Gallery? It's On the Puppet Forge

00:00:15 Ben Ford Hello, everyone, and welcome to today's episode of the Pulling the Strings podcast, here, as always, powered by Puppet. My name is Ben Ford. I'm the ecosystem product manager here at Puppet and active in the community as @binford2k. Today, we're talking with Michael Lombardi. He's our resident PowerShell expert. You may have heard about him in the Slack or seen some of the things that he's been working on. And being an expert, he's probably tired of me slacking him all the time with weird and obscure PowerShell or DSC questions, but no surprise since he's a wizard at that. He's also known for Picaroons, and you may have to correct my pronunciation on that. It's a tabletop role playing game for scoundrels that he's building in his free time. So Mike, could you tell us about the game a little bit if that's okay?

00:01:05 Michael Lombardi Yeah, sure. Thanks for the shout-out, by the way, I didn't see that coming. Yeah, so I've been doing indie tabletop role playing game development for a while now. Actually, a very smart, well read person in the community, Marcia, had written a blog post about different ways to use dice, and that captured my attention and what I thought was going to start as just sort of like a little bit of a blog post, maybe like a little joke, very quickly ballooned into a whole system - had sort of one of those ADHD moments where it's like, "Well, I'll just do this thing for like maybe five minutes" and then five hours later, you're done. You're like, "Oh well, guess that's the whole thing now."

00:01:44 Michael Lombardi So real simple, lightweight. There's no stats. It's just hit dice domains and tricks, and it all works off of rolling a couple of dice. Very lightweight little framework that is sort of intended to be very kit-bashable, very modable. Push it into whatever sort of setting you have in mind. And I've got a couple of supplements in the works for that right now. First of which is the Scrivener's kit, which is basically tips for running a game, includes a bunch of addenda, you know, optional rules to kind of flesh out that very basic system. I think that the original system Pickeroom's itself is like 20 pages if I stretch it with art and a lot of whitespace. So there's a lot of fun to work on. Got a bunch of stuff in the pipeline for it continually working on it. I'm funding it right now, which is sort of an alternative to Kickstarter because Kickstarter is a nightmare. Anybody who's ever run one hates it, but, y'know, it's a way for me to get the money for layout and editing, to clean it up and put it in the right state to do a physical release. Because it turns out you can just update the digital whenever you want as often as you like, and it's totally fine. It's a little bit harder to reach into people's houses and snatch up a copy and then fix a mistake there. So trying to get everything right before we hit publish.

00:03:02 Ben Ford Did I just hear you say that you're applying the CI principles to your game development?

00:03:07 Michael Lombardi I actually do. So, all joking aside, the digital iteration is all written in markdown, and I publish to another side project I have, which is called Platten, which is sort of a digital toolkit for interactive books in a digital format. I won't go too deep into it, but sort of standing alone on the hill PDFs are terrible, and we shouldn't use them for digital releases and being yelled at for that. But so doing it all through Platten means that I have CI that takes my stuff from draft. I use net lo-fi, so I get an automatic preview and everything is just kind of nice. But that also includes like linting, and I'm getting ready to add some accessibility linting to it as well. So like right now, just checks that my markdown is correct, but in the future, to also check language and remove things like there's little stuff that's pretty common, such as like instead of saying guys, say folks, that sort of thing. So trying to get all of those pieces up together and then that goes hand in hand with actual human editors who pass over the work and approve it or suggest changes in that sort of thing.

00:04:07 Ben Ford That sounds really cool. We should get community members involved in your game. I remember way, way, way back in the day, I used to play GURPS every now and then because it was it so lightweight. You could just pick it up for an evening or a weekend or something and not have to put a whole ton of time into like building a character and growing it, evolving it. I bet you haven't heard the name GURPS in a very long time.

00:04:32 Michael Lombardi Well, my first role playing game was actually Rifts, so my idea of what is lightweight/heavyweight is just entirely up ended, and I have no idea how you're supposed to balance things or anything like that, but GURPS still pops up occasionally in the community. I think they're still putting out supplements, believe it or not. Despite it being, what, 30-ish years old now?

00:04:50 Ben Ford Something like that. It's kind of fascinating how these little like sparks of an idea just kind of grow and take on a life of their own. One of my own projects, like I think maybe my most popular project, actually started from a little friendly argument at Puppet cup years ago. And just like by the end of that Puppet cup, I was like, "See, it totally works! Look at that." And it just kind of kept going.

00:05:15 Michael Lombardi I'm familiar with those conference driven development and discussion driven development as well.

00:05:19 Ben Ford So here we're going to talk about DSC. We've got a brand new DSC framework that Mike has been working on. Maybe we'll start with like what is Puppet's history with the DSC and why did we build DSC support into Puppet proper?

00:05:37 Michael Lombardi Yeah. So with the caveat that I wasn't around for the very first iteration, I can speak to it a little bit. So Puppet and Microsoft actually partnered very, very closely when DSC was first coming out. There were talks before DSC had been publicly announced between Puppet and Microsoft, if I understand correctly. And so the goal really was to make sure that when DSC launched, DSC being Microsoft's desired state configuration platform. So when that launched, the idea was on day zero, you would already be able to kind of like start using those DSC resources. And so DSC is a way for you to define in PowerShell sort of the same things that you would define in a Puppet resource, like what is this thing and how do I configure it? And DSC had some slightly different concerns from Puppet's concerns.

00:06:26 Michael Lombardi DSC was really aiming in a lot of ways to be a hierarchical agnostic tool, right? So you would layer a CM tool on top of DSC. DSC did have an implementation that would carry you from like, I have the resources now, right? I can configure a thing. I want to tell a machine what to look like with these resources. You can do that in just DSC. But you very quickly run into problems that Puppet people are familiar with, problems like, OK, these VMS need to look almost exactly the same, except this is for App A and this is for App B, and they get slightly different config settings. How do I do that? So you can kind of get around those things in DSC, but the more complex that the use case becomes and of course, it immediately becomes astronomically complex as you begin to look at applying it to legacy stuff. The harder it is to just use DSC. Right?

00:07:21 Ben Ford So it sounds like you're saying that DSC is more like primitives and it expects you or maybe doesn't expect it. It provides those primitives to higher level tools, configuration management tools like Puppet, to actually orchestrate and manage nodes and think about like an infrastructure level rather than the primitives to make individual settings.

00:07:44 Michael Lombardi Yeah, exactly. And that's really the biggest advantage of partnering Puppet with DSC was anybody who uses infrastructure that can be configured or changed with PowerShell could then reuse those primitives, regardless of what CM tool they were using. So the advantage there from a community perspective, is if I write a DSC resource, it'll work with Puppet, but it also worked with Chef and it'll work with Ansible. What that means is instead of, you only have this relatively smaller community, you have all of the community of people who want to manage that resource available, who can write and maintain for it. And so that was really powerful, and a lot of it has first party like Microsoft engineers who had implemented it originally or still maintain some of it. So that was really, really useful, too, because having been on the the modules team at Puppet at one point, a perennial problem, right, is it is functionally impossible to be an expert in all software that can be configured. It turns out people just keep making more of it, like the longer you sit around. And so having other people write that for you is a strong advantage.

00:08:52 Ben Ford Yeah, to be honest, I have wanted for a very long time for something like that to exist in the Linux or Unix world. And we've done a ton, like over the years even, I remember back in 2001 there was some tool that RedHat made that I use that I can't remember the name of it now, but it attempted to be that thing that the knew how the comp file worked and also the password file work and also all the other files. So you didn't have to do any of that. And we now have other things, but there's nothing just universal. There's nothing in the Linux world that's universal the way that it sounds like DSC has become.

00:09:29 Michael Lombardi Yeah. And the advantage of that is that it sort of fulfills a long standing promise of PowerShell, right? Which is if you learn PowerShell, it will make it worth your time. That was something that the team had originally been very clear about. And so what happens is you learn PowerShell and you're probably using scripts to maintain a lot of the things that you're doing here. Are you or you bounce into a box over WINRM and you say, OK, let me go ahead and change the setting and then you bounce back out. And the Windows world is a little bit slower to pick up on just how much easier and better your quality of life gets if you kind of shift those problems left into treating that infrastructure as code, which has always been where Puppet shines.

00:10:10 Ben Ford Well, that would mean that we would all have to agree on something. And you know, that's something we're really good at.

00:10:16 Michael Lombardi Yes, humans famously good at agreeing across groups. Yeah, for sure.

00:10:21 Ben Ford So we've done a couple of iterations of different DSC support. I think this is now our third iteration. Could you tell us about the other two and like how this is different? And maybe a little bit about what this system that you built does that the other ones don't?

00:10:38 Michael Lombardi Yeah, sure. So the very first iteration, because Puppet was working hand-in-hand with Microsoft, it was very tightly coupled to how DSC first rolled out. So it was able to parse the MOF files, which is machine something file machine oriented or orchestrator file anyway, an esoteric XML ish format that is absolutely awful to work with. But we were able to write a parser for that so we could look at a DSC resource and say, OK, this is what Puppet needs to know about it, and then we could turn that into Puppet resource calls. So Puppet just sort of would natively understand how to, you would you would write a definition in a manifest and then Puppet could just take that and then invoke DSC. Now this is really, really good because it meant that we had strong mapping of everything that was happening in DSC, which meant that we could do things like, say, you enter the wrong data type, right? This is supposed to be a boolean. You put down something else, we could tell you at author time if you goofed. It also meant that we could do a little bit of work around validating and calling. But because it came out at the same time that DSC first launched, it was very tightly coupled to how DSC was then conceived of and implemented and released.

00:11:55 Michael Lombardi And as with all software, it turns out it doesn't stay static. Things change. You learn new idioms and you learn new best practices and then you adopt those new best practices. And so as those adoptions kept coming, it turned out that the original sort of layer between Puppet and DSC got more and more fragile and became much harder to maintain. And the way that it worked is we would turn hundreds and hundreds of DSC resources into Puppet resources in a single module. But because that was slow and kind of fragile and it didn't always work, and there was a lot of going back and fixing and handholding that had to happen on our end. That meant that our releases were slow, so they would be out of date. So if there was a bug fixing a DSC resource and you wanted it right now, you might not get it for a month, maybe two. Right? We were doing it on a sort of as needed cadence by the end of that.

00:12:45 Michael Lombardi And so we realized that that was not great, particularly for customers who were like, No, I need the updated functionality right now. I contributed it to the DSC resource, right? And a lot of cases that was happening. Puppet customers and Puppet users would find a problem in DSC via using it with Puppet, file a bug report, get it fixed and then have to wait for us to update. And that's not a good experience. So we wanted to do something to bridge the gap where like we need, we need to reapproach this. And so the bridge the gap solution, DSC light, we just sort of said all those nice things that we were doing for you that put up the guardrails. That's what makes it fragile. We're going to knock all the guardrails off and we are just going to enable you to do the things you need to do, but we want to hold your hand along the way. Right. We have to paint a giant dragon on the map now because you're definitely in dragon territory.

00:13:29 Ben Ford Yeah, I was in education at the time, and I remember the big difference for us was that massive, massive plugin sync time, the very first time that the module got sync out and how much better that got with the DSC light. But that was only because we were sitting there looking at it in real world that wasn't really that big of as much of a deal.

00:13:52 Michael Lombardi Yeah, and one that was the one of the other problems, right? Was so our initial DSC implementation vendor to all of the things that you needed. DSC light doesn't. DSC light has no way of knowing as a Puppet module whether or not the the PowerShell module containing the DSC resources you want is on the system.

00:14:10 Ben Ford So how do you distribute them? That means that you as a user, have to get the DSC resource everywhere you need it, right?

00:14:16 Michael Lombardi Exactly. And so there's two options, right? There's one which is go outside of Puppet and just do it yourself. And then there's a couple of Puppet options. You could do files you could just like, copy the folder from a shared drive somewhere and throw it down in the boxes as needed. Henry Buckle HBuckle on the Forge has a forge module that handles some of that, and we've been in talks with Henry about whether or not we should adopt it and kind of pull that in. A lot of that has been obviated by the new DSC stuff. When James and I sat down to do a reapproach, we're like, OK, we're no longer constrained by anything that we did before. What if we started fresh? And so that's what we did. We sat down. We thought that through and then came up with what we have now.

00:14:56 Ben Ford Well, that's really good because honestly, as you were saying that I was kind of having flashbacks as to how miserable that was trying to figure out how to get the DSC resources shipped around where we needed them to be.

00:15:07 Michael Lombardi Yeah, it was not a fun experience for sure.

00:15:10 Ben Ford Yeah, and it always felt like I had to know too much about the tools that when all I wanted to do was just make config changes. And I was like, I thought Puppet just made it easy for me to manage configuration. Why do I have to know all of these weird details?

00:15:22 Michael Lombardi Absolutely. And that was one of the things that we wanted to address. Our focus with the new implementation was to take all of those problems and shift them left again, put them back on our plate and pull them off of the user's plate. And so that was really front of mind for me and James when we started on this project. And then I kind of kept that front and center as I became the lead designer and maintainer for it.

00:15:44 Ben Ford Right on. I really like that approach is like, if we can solve the problem for you, then you don't have to worry about it yourself. So could you tell us about like, how the new DSC modules work and like, what's so revolutionary about them?

00:15:55 Michael Lombardi Yeah, absolutely. So the original builder that did all the nice guardrails was written in Ruby, which also meant that you had to know Ruby to know how to use it. So we were like, OK, we're starting from scratch. What should we use to look at a DSC module? Well, why don't we use the language it's written? So we decided from the very beginning to do this entire implementation in PowerShell. So there is a PowerShell module up on the PowerShell gallery called Puppet.DSC that that does this work. And so what this does is this introspects on a PowerShell module with DSC resources. It looks through all the resources that that module has, and then it maps out their API surface, everything that you should be able to call and use those resources for. And it turns that into a Puppet resource. So this was really a thing that we couldn't do originally because the resource API did not yet exist. But now with the resource API, when you define a type. You're very much just sort of writing some key value pairs into a hash. It's very, very straightforward, comparatively. And so we were able to use PowerShell to map the DSC resources and then tell Puppet very neatly this is what a DSC resource looks like, and the project involved a whole lot of meta programing, which is writing code that writes code, which is a little bit of a brain pretzel for you. But because of that, what we ended up with is a system whereby the actual Ruby and Puppet code that ships with these generated DSC modules and I'll get to that in a second, is very, very minimal. It actually just inherits some base types base providers from our Ruby PowerShell module which meant that if you need to get an update that will affect all of the DSC resources, you don't have to wait for all of the DSC resources to be rebuilt. Instead, you just update this other dependency and then everything can take advantage of it.

00:17:55 Ben Ford So what you're saying is that we've got basically kind of a translation library that takes a DSC resource and takes the code of a Puppet module that knows about the DSC resource and actually interfaces with the DSC itself. If we need to update something about that library, it will improve the behavior of all the DSC resources that we have on the Forge.

00:18:18 Michael Lombardi Yep, and sometimes there's a very minimal update that has to happen to the existing DSC resources, like if we add a whole new feature that has to be exposed and the type, that will require a rebuild. But those rebuilds are very fast and they can be done sort of on an as needed basis, which was great because again, one of the largest problems we had with the original DSC was that you didn't have a way to kind of speed that process up. So if original DSC gave you all these guardrails that it knows what all the possible things that you can pass out and it knows what the DSC resource looks like and DSC Light has no idea, you just handed an arbitrary hash and you say, I hope these are the right things. I have no way of validating this before I run it.

00:18:59 Ben Ford And the only way you know if something goes wrong is if it blows up when you run Puppet.

00:19:03 Michael Lombardi Absolutely. So the new method, because we're able to introspect on those DSC resources and find out what their API surfaces, we can again tell you at author time. And since the original DSC module launched, there's also been the launch of VS code extension. So what does that mean? That means you have IntelliSense if you're using our language server extension, which is not just pinned to VS code, it's just one that I talk about all the time because it's the one I use, but if you're using our language extension, then you can get author time feedback about what it is you're writing and what the correct values are and you get linting. And so, you know, while you're writing a manifest now, if what you're doing is correct for that resource, so we can tell you before you ever try to run code, you goofed or no, this is correct.

00:19:45 Ben Ford That is incredibly helpful. That's something that I use all the time.

00:19:50 Michael Lombardi Yeah. And one of the other advantages is sort of when we were doing this, this redesigned from the ground up, right, we got something new and very, very exciting from my perspective as a terminal user of DSC, so DSC, by default, does not really care about state change in the same way that Puppet does. So DSC resources, there's three I used to think of them as like API points, right, that you can hit for a DSC resource. You can get the resource, you can test the resource and you can set the resource. But by default, DSC doesn't really track change of state over time. And so when you run set resource, it's up to the DSC resource author if they're going to track changing from current to new state and it is almost never implemented. I can't think of any that actually implement this, although it is technically feasible.

00:20:43 Michael Lombardi So what's good is that our new implementation, unlike DC Light, which has no way of knowing what the state ever was until you run it. And unlike our original DC, which was sort of hampered by that API, we're now able to say, OK, I'm changing property A from one to two, and we can just see that across all of them, which is great because one of the biggest advantages of using Puppet has always been I want to see how my configuration has drifted over time and how I have enforced the state that I know it's supposed to be in. And so now you can just do that with DSC. And as far as I know, our implementation is the only one built on top of DSC that'll do that for you. If you're using DSC with any other thing, it's limited to the same get test set, right? And what that has always historically meant is it's out of date. We're going to make it be in sync. Why was it out of date? Uh-Huh. I don't know.

00:21:52 Michael Lombardi Yep, exactly.

00:21:53 Ben Ford It's surprising to me. I'm kind of stumbling with words because I don't know why you would not want to build that into a system to begin with, because that's such valuable insights into how your infrastructure is working, you know?

00:22:07 Michael Lombardi Yeah, I think a lot of that comes from it was built to be resource first and then abstraction collar second, right? And so all of that functionality in Puppet does not live in the resource definition. All of that functionality, even in Puppet, lives in Puppet. So Puppet says, give me the state of the things. And then which one are you iterating over? OK, well, that was its current state. This is the state that you say it should have. Here's the diff. This is what I'm changing. And DSC doesn't really have that. DSC also, unlike Puppet, does not have a way you can't say I want to change just this property of these seven properties on this resource. DSC, as designed, is all or nothing. So when you call set, it's on the resource author to make sure that it's not trying to unnecessarily perform changes that don't need to be there.

00:22:57 Ben Ford Yeah, I didn't know that that was the case. That's a thing.

00:23:00 Michael Lombardi Yeah, it means that they're a little rough to implement, right? Like having written some DSC resources. There's a lot of best practices, and one of those best practices is you only change exactly what you need to when you need to. But then there's also some, some things that DSC resources were like, so using files, as an example, it kind of can't change a file without changing the whole file. So if I'm going to rename the file well under the hood, that's actually moving the file, right? And so for that implementation perspective, it's like, well, if I'm doing that and changing the content, I don't want to do two things. I'll just create the new file and delete the old one, right? And so all of that is up to the resource implementation under the covers. And so one of the nice things about our new approach is we're staying absolutely all the way out of that. We have no interest in getting into how the resource is written or designed. We're just making sure that you can call it, and we're making sure that another advantage over original DSC is now it's on a module by module PowerShell module basis.

00:24:10 Michael Lombardi So if you grab a module that has these five resources, you don't need to grab all of the modules that we've Puppetized, just that module with those resources, which is a significant advantage. And again, better than DSC light. We go back to vendoring bring those resources for you. So you know exactly what module with what resources at what version has been deployed, and you can control all of that.

00:24:33 Ben Ford So the tool that you built is actually, I mean, I suppose you could call it a Puppet module builder of a DSC resource module, which is a lot of different words. And you talked a little bit about how it works, but how did the modules get published up to the Forge?

00:24:49 Michael Lombardi Yeah. So, that's a great question. So initially, the way that we did it was I would have the PowerShell module available locally. I have the API keys to push to the Forge, so I would do the build and push from my workstation. That is obviously not scalable. I like to take vacations. People sometimes release software while I'm on vacation. So what we did is we actually have a GitHub action now that runs every hour and it just looks at the gallery for any DSC resources that have not yet been Puppetized. And then I think it does the top five or 20. And so this is brand new modules that have never been that been Puppetized before, and this is also modules that have a new version that has not yet been Puppetized. And so what that means is we're pretty sure unless there's like a flurry of releases that's, you know, hundreds and hundreds, we can get anything that lands on the PowerShell gallery Puppetized and pushed up to the Forge within about a day.

00:25:43 Ben Ford Well, that's like eventual consistency, right? It'll just keep running until they're all updated. But if I heard right, if I, as a DSC author, published something, it just automatically migrates to the Forge so I don't have to do anything to tell Puppet about it. I just put it up on the PowerShell gallery and within X number of hours, the Forge also knows about it?

00:26:03 Michael Lombardi Yeah, and the release cadence is very slow for DSC modules. They're not launching like several an hour, so it's usually within an hour or two. Whatever you just publish is going to land up on the Forge.

00:26:13 Ben Ford That's really cool. And so if somebody is wanting to, like, learn how to use, if they go to the Forge and they find a module and they want to use that resource, how do they figure out what parameters it has or how they can use the module there?

00:26:29 Michael Lombardi Yeah. Another great question. So one of the advantages that this is just the DSC or this is just a Puppet module, right? Is that from the perspective of the Forge, it's treated just like a Puppet module. So there's a reference.md file that gets generated prior to these things being published. So the same way that you look at all the other Puppet resources that you might want to use, all of that same help information is available right there on the Forge. And then again, if you're using VS code, all of that help information is surfaced by the extension for you. So if I start typing DSC underscore website right by the time I'm getting to the "s" in website, it's already pulled up to help and is like, Did you want to write this one? Here's the help for it, and then you can auto fill out the parameters from there as well.

00:27:12 Ben Ford So you can use like Puppet resource on the command line to list out the parameters and everything too. So just like any other Puppet module?

00:27:19 Michael Lombardi Yep. The biggest difference is again, because these resources don't have property by property setting, even if you do, Puppet Resource DSC underscore foo property equals block that will run that whole DSC resource against it. So that's something to kind of keep in mind, but that does work. Getting the resources is a little bit different. So Puppet resources by default assume that like as an example, if I was listing the Apache sites on a node, I could just do Puppet resource and then that resource name and that would list back all of those, right? If I was doing Firewall and Rspec all this, DSC works on a single resource at a time, so you can't run Puppet resource and expect it to tell you what that current state looks like. It just doesn't work, unfortunately.

00:28:08 Ben Ford Oh, that's too bad. But I mean, not all Puppet resources work that way and the ones that are innumerable like you, you can't say Puppet resource file and have a list out all the files on a system.

00:28:19 Michael Lombardi I would not want it to, and that would be very, very slow.

00:28:22 Ben Ford But you could, because you could with a user type, because it is only a finite number of users that can would come out of the password file. So imagine that having these different translation layers, because that's really what we're talking about. It's like there's a translation from Puppet to the PowerShell library and from the PowerShell library to DSC and from DSC to the system itself. I mean, Puppet itself is one big translation layer from Puppet to the system. That's how we work. But I imagine all those extra layers makes troubleshooting a little bit difficult. And when we were first starting to deploy these, I remember hearing about some of the field engineers, Bill Wilcox, was one I was thinking of, kind of helped you troubleshoot some problems in the field. Is there a story there? Could you tell me about like how that went or like how you go about troubleshooting things with so many layers of abstraction?

00:29:15 Michael Lombardi Yeah. So one of the great things about that actually is that Bill and I together worked on an actual troubleshooting guide for the DSC Resources via Puppet, right? Which makes it a whole lot easier when you can just follow step by step. So there are some things that we have a pretty good handle on. We kind of know what it is when we see the error message in one of those is couldn't find the dependency gem Ruby PowerShell. What does that mean? Well, that means that you didn't install the dependency library, which is Puppet Labs PowerShell. So when that pops up, we know exactly what that is, and that's a very quick fix. There are other things that get a little bit stickier. And so one of the, just to kind of take a step back, when you write a manifest that's going to use one of these resources here's what happens. So you've written a manifest in Puppet's language, so Puppet turns whatever you declared in your manifest into a ruby hash. So this is the shape of the thing, right? Because Puppet is written in Ruby and it needs to understand what that looks like. So now we have a ruby hash, we have a Puppet manifest declaration, then we have a ruby hash. That ruby hash then has to be sort of munged inside of our library code to be able to create the right PowerShell hash to be handed over to DSC, which then says, OK, now I know what I'm doing and I'm going to go ahead and do some stuff and send some data back. So when DSC sends data back, we automatically turn that into JSON because I was certainly not about to write a PowerShell hash munger from Ruby. So instead, that returns JSON. So now we know we're getting JSON back, we can convert that into a ruby hash and then munge that back into the shape that Puppet expects. Because what PowerShell hands back in data structure, types, key names, etc. is not always what Puppet is expecting, so we have to kind of munge that back. And then now we have the ruby representation of what you put in your manifest, and we have the representation of what is on system and we can compare those and that's how you actually get that property by property reporting. So now we can say, Oh, I'm going to change current state to desired state. That's what allows that.

00:31:18 Michael Lombardi So there are a couple of things that we can kind of do to troubleshoot. The very first thing that we decided to do was if you do your Puppet run in debug mode, if you add the debug flag dash dash debug to the end, you're going to get a whole ton of debug statements that print out because those are really, really useful when we're looking at the DSC stuff. So one of the things that that'll print out is exactly what is being sent in PowerShell to DSC. So what that means is you can just copy that code out of your debug log and then take that over to a PowerShell terminal on the system that you're running from. And then just run that there. And so what that does is that says, OK...

00:31:59 Ben Ford That's actually not all that different from how we debug a lot of things with Puppet. Like I said, Puppet itself is a translation layer. When I write code, I'm not literally writing the command line that Puppet passes out, it figures all that out for me, but it'll tell me what it's running. And so if something goes weird and I want to troubleshoot, I can do exactly what you just said and copy that command line out and run it and see what errors I get on the terminal. So it doesn't sound like it's that much more complex to do it with PowerShell.

00:32:33 Michael Lombardi It's not. It tends to be a little bit longer because instead of a single line, there's some helper functions that we had to throw in to kind of get everything to work. But you can still take that entire block and then run it. And so what that's really, really good for is finding out is this bug that I just found with Puppet calling DSC or is this bug in DSC? And because we don't write any custom code in these modules, we can't really, and frankly don't want to, be on the hook for fixing bugs in somebody else's code. That's a lot of work. I've done it before, and we still do occasionally, right, if I write up a good enough repro of a problem that I'm debugging from calling DSC from Puppet, if I write up the repro and it looks like the fix is three or four lines, I might just go ahead and throw a PR on the upstream project for that. But that really helps us narrow down. Is this problem Puppet? Is this problem PowerShell DSC? Where does that kind of lay?

00:33:28 Michael Lombardi So the normal problems that we see in Puppet's side, because the DSC side could be any number of things, the normal problems we see on the Puppet side is there's a mistranslation. So something about the way that I designed that translation layer is not working correctly or, we also see this a fair amount, the API surface that the DSC resource declared does not match its implementation, which looks at first glance like a Puppet problem, but is actually an upstream problem because an unfortunate truth about DSC resources is they're written in such a way that you can actually write API spec that does not match your implementation. And DSC has no way of knowing. DSC when it runs doesn't look at its own API spec, it just looks at its implementation. And so you can end up in a case where, common example of this is if you call get on a resource, the resource does not require that it returns all of the key value pairs that it can possibly have. So you could just be missing things. And so then when Puppet gets it, it says, Well, I don't know what the name of this resources, that's nil. Well, that doesn't make any sense.

00:34:29 Michael Lombardi The other thing that we do that that Bill and I have done quite a bit is we can do pry debugging. We have a blog post on this as well. And so pry is a ruby tool for stepping through code as it executes, which means that we can sort of, if things are getting really sticky, we can see exactly what's going on, right? Where does that problem lie? And we can kind of go and figure those things out. These problems have been largely smoothed out over the last several iterations. We still get a couple of bug requests that come in, but a lot of them are obviated by sort of a, we actually had to add to the Puppet API surface for how you tell if a resource is in a desired state or not. And so for these DSC resources, you now have two options you can do the default, which is I want property by property reporting, which is how all of the rest of Puppet works. And you can also say, OK, I give up on property by property reporting something's weird in the DSC resource. I'm just going to do, I'm going to trust DSC and I'm going to run that test interface. And if the test comes back true, I won't change any state. And if it comes back false, I don't know what's wrong, but I know I need to run set, which is sort of the default DSC behavior. And so we're able to do both of those within the same module with very little extra code and that I'll just sort of works.

00:35:53 Ben Ford Cool. Maybe, maybe we can shift gears a little bit. I'm curious about something that I've been hearing kind of rumblings about with big changes coming to DSC with DSC 3 and how exactly is that going to affect your modules and the Puppet infrastructure?

00:36:14 Michael Lombardi Yeah. So good news, bad news, right? So good news. The primary way that Puppet interacts with DSC, skips the DSC agent and uses Invoke DSC resource, which is PowerShell command. And so that's always been single resource and you give it the properties that you want to change and then it goes off and does its thing. So the great news is that's staying the same. From that perspective, we don't need to change anything. So the bad news is the new version of DSC is going to require class based resources which make up, I think, less than 10 percent of all DSC resources, which means if you want to flip to using that, you're going to have to wait for those resources to be reimplemented as classes instead of functions. And there's also going to be a little bit of work that we'll have to do around. Right now, we just flat out say we're only supporting Windows PowerShell 5.1 and above. And the reason for that is that the DSC engine is only released stable under that version of PowerShell. So new DSC is going to be PowerShell 7 forward and it'll be actually cross-platform, which is pretty neat. But the downside, of course, is that it's only in 7 and it's going to be all class based. So there's a little bit of work that we have to do. Like right now, we just dump out and say, like well we can't do anything in PowerShell 7, so we're not even going to try. We're just going to fall back on Windows PowerShell. We may have to implement a migration strategy there that sort of allows you to use both side by side for your time until the new way is the only way that anybody still supporting. But for right now, that's still out on the horizon. DSC v3, I think, is slated for release sometime in 2022. And I do not expect a release to have, you know, more than maybe 15 to 20 percent of all of the resources converted over to be even compatible with V3. So people will probably still be using V2 for quite a while.

00:38:16 Ben Ford So maybe kind of a step back. When you talk about function versus class, maybe like a super super super, like a high level, you could say that this is maybe analogous to like a script versus an object oriented programing language or what exactly does that mean?

00:38:36 Michael Lombardi Yes. So they're both written in your PowerShell. But so the function based approach has a MOF file, which defines the interface, and it has a function file that really it's a module file that contains some functions, test resource, get resource and set resource. I think target resource get, test, and target resource. And so the migration from that to class based is now you can use PowerShell classes and define the entire thing right there. So the class then becomes both the API declaration and the implementation, which is all around quite a bit neater. There's some drawbacks around test ability and those sorts of things, but you know, by and large, they're much more easy to reason about because they're a single class in a single file, and that's just a lot less, no jumping back and forth between two different windows, trying to figure out what is what and why things work the way that they do, which I'm hoping means that the gap between the declared API and the implemented API will shrink considerably.

00:39:40 Ben Ford So in a way that, like the class itself, is sort of self documenting the way that if you look at a Puppet class, you can see what all the parameters are and what it accepts?

00:40:05 Ben Ford When you say PowerShell team, you're talking about Microsoft's PowerShell.

00:40:07 Michael Lombardi Yeah, yeah. Correct.

00:40:09 Michael Lombardi OK. No, no, I'm not doing it. Wow. There'd be a lot of work, but they'll face a lot of the same problems that we've seen before at Puppet. When you're changing between Puppet 3, Puppet 4, when you do major changes like that, any automated tooling you can give people is really only going to go 80 percent of the way tops. And then thereafter, there will still be a lot of sort of like smudging things into place. And so I expect that process won't be fast. And it's one of those things that's going to be a slow sort of process. And for the first six, 12 months, the DSC V3 is out, it's going to be bug hunting season because there's going to be, any major change like this always means that there are new edge cases that they'll have to discover and work through. And so while we're going to be there through that process and we want to make sure that we are adding support as early as possible, I don't expect adoption of that support to be necessarily very high.

00:41:15 Ben Ford So this is kind of a two part question here is like, what do you have to do in order to move along with this change? And based on what you said earlier about the PowerShell library, I suspect that it's mostly going to be in that interface library. And then the second part is what is the impact to the end users of our modules?

00:41:36 Michael Lombardi So the impact to the end users should be they can use newer resources and that's about it.

00:41:42 Ben Ford So the code would just continue to work.

00:41:44 Michael Lombardi Exactly. They shouldn't really notice any changes. There will be a little bit of work in the translation layer so that we can determine in that dependency library between whether or not we're running for Windows PowerShell or running for PowerShell 7 and above. So there'll be a little bit of work there, but the largest portion of the work will land in our PowerShell module that does that conversion, where right now it just says, Look, I'm not going to get you anywhere if we're using PowerShell 7, it's just not going to function. So we'll have to actually implement that and we'll have to have some logic for what are you trying to parse? Because the old DSC resources can't be parsed in 7 and only the new resources will be able to be parsed that way.

00:42:25 Michael Lombardi The old method involves CIM, which is computer interface management, I think, something like that anyway. So it involved CIM instances and mapping these sort of esoteric spells cast by the system and figuring out when all that stuff looks like.The PowerShell 6 and above does not have CIM instances, so they cannot be mapped that way. So how we map the resources will have to change entirely. Now, my hope is with everything moving to class-based, I can actually do what I wanted to do in the first place, which is parse the abstract syntax tree, the AST, which is sort of how a program interprets a script file or a code file. So it tells you when you pass the AST, it knows what all the code blocks are, and about all the variables and all those kinds of things. So ideally, we can go much, much faster because I don't have to load anything into memory. I can just do a static analysis of a file and then tell you what the shape of the thing is supposed to be. And then we can get all the exact same stuff that we're getting now just out of that instead.

00:43:28 Ben Ford So effectively, you've got a bunch of work to do in order to translate between the different systems. But our end users, they can just keep running Puppet code.

00:43:36 Michael Lombardi Yep, that was that was, you know, sort of circling back to that original goal for this project was as much as possible we want to move the maintenance and thinking pain off of the end user, which is where we put it with DSC light and back onto our plate. If there's, you know, 40 to 60 hours of engineering time and effort to spend on our side to enable hundreds of thousands of skipped hours of the same work for other people, that seems like a win for us. Because that's our job is to make sure that we can take these really hard problems that are tractable and pull those problems onto our plate so that you just get to have the nice dinner. Instead of having to go find the ingredients, prep the ingredients, make the dinner, plate the dinner and then eat the dinner. You should just be able to eat, right. So sort of moving from like, go to the garden and find the right herbs to go to dinner, that that's kind of the way that I see our work.

00:44:31 Ben Ford I like the analogy. So, so for end users wanting to upgrade all this, all they have to do is start moving to PowerShell 7 and upgrading their end nodes, and your tool will just kind of take it in stride.

00:44:47 Michael Lombardi Yep. So there'll be two sides to this one, obviously. Then they'll need to have PowerShell 7 on the end node with the DSC, we may vendor it. I'm not sure yet if we do knew vendor, we'll vendor it into our dependency module, kind of up in the air on whether and how we're going to do that because again, we want to kind of take out the pain of like, you shouldn't have to know you need to deploy this module. A new DSC will be released in a module format instead of as part of the core language, then moving it out of the core, which means that it can be updated more frequently, but also means that I have to figure out how we're going to get that there. And then the other thing that they'll need to do is just update their version pin for the Puppet module that has the DSC resources to the one that they want to use and write the new V3 class based ones. And then that should all just work. It can, theoretically, assuming that there's no language features used in the DSC resource that are PowerShell 7 specific should still work with the Windows PowerShell 5.1 stuff that should all continue to function. There should be no difference. They're not breaking the API contract, as far as I know, with the class based resources, it's just that all the function based resources will stop working.

00:46:03 Ben Ford So does PowerShell allow you to have multiple instances installed? And if so, does your PowerShell library know how to talk to those things? Or does this mean that when users upgrade to PowerShell 7, they'll lose out on 80 percent of the resources and until they start being upgraded?

00:46:24 Michael Lombardi It can all be side by side, and that will all just work. So one of the other neat things that we sort of touched on and then didn't go into is I mentioned that we vendor the modules. The other thing is, we don't just vendor the PowerShell module, the DSC resources you want. We also vendor all of its declared dependencies. So presumably any DSC resource that's going to depend on this, this the DCS v3 module is going to declare it as a dependency, which means that you just get vendored. So maybe that'll solve my problem for me. That does mean that you might have, you know, 11 versions of the same thing sitting on your system. But since I think the current size of that module is like a meg, maybe you won't notice.

00:47:07 Ben Ford So just to make sure that I'm understanding, right, this means that the runtime itself will be vendored, so you'll be able to have as many runtimes as you need vendored into the PowerShell library?

00:47:18 Michael Lombardi Yep. So the module and all of its dependencies will be vendored. And what that has meant is there used to be some, some conflicts with the original DSC. So if two modules share a dependency and they have different version requirements for that dependency, we didn't have a way to resolve that. So we just kind of went "oopsie-woopsie, good luck". So now we can and they don't cross over each other, there's no contamination. It's exactly what that module needs, and only what that module needs is what gets seen during a run. So we handle all of that for you.

00:47:48 Ben Ford And so what I'm hearing about here is that the end user really just doesn't have to care about any of this. They can just keep writing Puppet code, and maybe they start using new modules when they come out. Maybe they don't, but it doesn't really matter. They don't even need to care about it.

00:48:03 Michael Lombardi Yep, that is my goal. For the end user, it should just be transparent. They shouldn't need to think too hard about it. If they have the things that are meant to work, maybe they toggle a flag and they say, I want to use PowerShell 7 instead of 5 for some reason. And if the resource supports it, that will just work. And maybe they, you know, it just all kind of keeps functioning.

00:48:29 Ben Ford We do that now by letting people pick different providers if they absolutely have to. Yep, but that works. I mean, hearing a transparent, like painless, no effort involved upgrade process that just makes me happy. That's something that we should all be working for.

00:48:47 Michael Lombardi Having lived through several very painful upgrade processes, it's always front of mind for me when we work on these things. That was one of the reasons why we wanted this to just as much as possible, the new approach to DSC is it's just Puppet modules. From your perspective, as a user, it's treated just like a Puppet module, just like a Puppet resource. You don't have to know or think about any of the other stuff, which is the ideal for me is that you've already spent so much time learning how Puppet works and how to be effective with Puppet. Why would I make you learn another thing? So I just kind of want to enable that.

00:49:19 Ben Ford All right. Well, it looks like we are running up on time here, so we're going to have to bring this to a close. It sounds like we've got a lot more to talk about, but maybe that'll be for another session. But are there ways that people could get in touch with you or anybody on the DevX team if they were curious or wanted to maybe get involved with any of this? Are there ways you can get maybe in the Slack or social media or anything?

00:49:43 Michael Lombardi Yep, we can be reached in Slack. I hang out in the Windows channel primarily, but I'm also kind of just around generally. You can find me there, Michael T Lombardi. You can find me on Twitter/X at TrebuchetOps. We also have discussion pages up for the DSC project in particular on GitHub, though we can probably drop a show link to those kinds of things, and I'm pretty responsive to issues being filed, I check my GitHub notifications fairly regularly to prune out a lot of the noise, but that is a great way to get in touch with us. I think you can also email us DevX@ Puppet.com. I think comes to us and I think DSC@Puppet.com will come particularly to me. I'll be paying more attention to that one. That is a email address specifically for these problems and projects.

00:50:35 Ben Ford Right on. Your modules, honestly, they really seem like the easiest way around to actually use the DSC resources and to keep up with developments in the ecosystem. So I'm really glad to see this. And then seeing where we can go and pickerooms honestly, sounds like a really fun game, I have a few people I'm already thinking of that I want to introduce it to, so I'll make sure to put a link to that in the show notes for sure.

00:51:00 Michael Lombardi Thank you so much.

00:51:01 Ben Ford Of course, well, that's a wrap for now. And once again, thank you for being here and thank you for listening to the Pulling the Strings podcast, powered by Puppet.