Season 3 — Episode 5

Acceptance testing is fun, I know! In this episode of Pulling the Strings, Lucy Wyman, Gene Liverman, and Trevor Vaughan explain multi-node testing with Beaker, how Beaker is different from Litmus, and future plans for Beaker.

 

GET A DEMO OF PUPPET IN ACTION

Transcript

Demetrius MalbroughHey, everyone, thanks for joining this episode of Pulling the Strings podcast powered by Puppet, and I'm delighted to be your host. My name is Demetrius Malbrough and I'm on the product marketing team here at Puppet. And I'm really excited today to talk with Trevor Vaughan, Lucy Wyman and Gene Liverman. Trevor is a co-founder of Onyx Point and also the co-creator of the System Integrity Management Platform, the largest compliance focused, Puppet-based open source platform. And he is also a contributor to multiple Puppet communities and has been using Puppet for over a decade and Beaker since it was released. Now, Lucy Wyman is a senior software engineer on the Bolt team here at Puppet and also co-leader of the Open Source Stewards, which we've talked about on the pod before. Now, Gene is a newcomer, Gene Liverman, senior SRE and also the lead for the Customer Zero program here at Puppet, working on all the behind the scenes things engineering needs to do its job. So welcome to Pulling the Strings, everyone. And let's start off with a short history lesson on what is Beaker and why was it created. Lucy, would you like to take that?

Lucy WymanSure. Beaker serves a couple of different purposes. I think that the kind of short description would be people call it a test harness, but really it's meant to provision any remote or even local machines that you need to use for testing various Puppet infrastructure. Admittedly, by most common use for it was actually at the Puppet console, but obviously used throughout the Puppet ecosystem. And it also, so it will provision your targets for you. It'll also then execute your tests on those targets and then can handle some of the cleanup afterwards. I guess in terms of history, it was created before my time, which was five years ago. So I'm going to say, you know, Trevor would probably know better than me, but maybe eight or nine years ago, back when we had a quality assurance team and it was created by that team in order to make it easier for developers to write their own tests. And so you driven forward and maintained by that team until our quality assurance organization in Puppet was kind of disbanded, maybe disinvested from, maybe three or four years ago. And then, I mean, ever since then, it's been just kind of maintenance mode, you know, not really getting a lot of attention from Puppet as an organization, despite, like I said, being used throughout the community by wonderful module authors like Trevor. Trevor. I don't know if there's anything you want to add.

Demetrius MalbroughYeah, I was going to see if Trevor wanted to fill in the blanks for us and add some more color to the history of Beaker.

Trevor VaughanOh, sure thing. So I actually at the last Puppetize in 2019, I did a presentation on Beaker and PDK and in that I have a slide, if anybody wants to go look it up. But basically rspec Puppet came around at about 2012. It was a great testing framework for Puppet modules. It really let us get the deep and kind of have a more focused DevOps approach to getting things done. In 2014, by my notes anyway, Beaker came out and was added as kind of an adjunct to our rspec Puppet. So you use rspec Puppet for unit tests and then you pulled in Beaker for your acceptance testing. It took my project about a year to kind of integrate it into our testing framework. But once we pulled it in, honestly, there really hadn't been anything better for us to date, because we focused a lot on multimode testing and actually testing small infrastructures. And as one of my colleagues likes to say and his opinion, Beaker is basically a variant of vagrants. It essentially makes it easy to stand up very, very small infrastructures, do your tests and pull them back down again. But all within the rspec, the niceties of rspec.

Demetrius MalbroughOK, and Gene, did you want to also add something to that as well?

Gene LivermanSure. It was about three to three and a half years ago when our QA team got disbanded as a team and that function got rolled into other parts of the organization. And at that time, the quality engineering team and the Infrator team, which have since merged to become DEO, and that's the team that I own, took over maintenance of Beaker, but we didn't really have the time or inclination to do anything more than what was absolutely needed to keep the lights on with it. From our perspective, it worked. And the things that we put effort into was when PE tests started failing because except for that, it worked fine. And from our perspective, its purpose was to be able to test Puppet Enterprise. We knew the community had it too, but we just simply didn't have the bandwidth to do anything beyond that with it.

Demetrius MalbroughOK, awesome. So you have the history and I hope everyone learned some things that they didn't know before about Beaker and why it was created. Now, I guess one of the most important questions right now would be how it's different from some of the other testing tools that are out there, like Litmus, in particular.

Trevor VaughanSo we actually looked at a lot of different tools over time, as both Lucy and Gene were saying, you know, Puppet, again, didn't abandon it, but it wasn't the primary focus, which makes sense. And so, of course, we kind of farmed around. The big thing for us was both, well, the big two things for us were both the integration into the rspec testing framework, the Ruby Rspec testing framework, because it allows us to parse the different positions, kind of tweak things, play with things on the fly and get a really good systematic output of all of our tests from the top down. Some other platforms have had similar capabilities in the past. And I'll be honest, we went and started looking at Test Kitchen. A lot of the community and different communities use Test Kitchen for their testing. But we found very quickly that it could not do the second part of what we needed, which was multinode testing. So from our point of view, we do a lot of just widespread systems automation, and we want to make sure that when we automate a system it, actually all the components can still actually communicate properly. And so we had a particular set of issues where we need to be able to sit, stand up four or five nodes at a time, test them quickly and then tear everything back down. And basically when we started talking about, OK, what are we going to do in terms of trying to find something that's not Beaker, if Puppet wasn't interested in playing along, we basically didn't find anything better. And we looked at the other Beaker, which is actually a RedHat project, Test Kitchen and we looked at Litmus and Litmus is making great progress, but right now it still doesn't have the kind of seamless multinode capabilities that we really need. And when we started looking further into things, basically, it turned out that what we started talking about was building something that looks suspiciously like Beaker. So at that point, basically, we got in the talks with the folks at Puppet and Vox Pupuli and the decision was made to start kind of porting bigger itself over into Vox Pupuli so that the community can start basically the care and feeding of this project that we currently don't have a replacement for.

Lucy WymanI think there is a couple of other things that I want to mention. Not that Trevor left anything out for just, I guess, things that are top of mind for me. I think Beaker has a pretty rich ecosystem of like helper modules and ways of interacting with common technologies. There's like a Beaker AWS, Beaker Docker Helper, you know, at varying levels of kind of doneness, as it were. But I think that that is a really important part of talking about how Beaker is useful and how it's different from Litimus. I think a lot of that we are maybe going to need to recreate for Litmus and a big part of us moving all of the Beaker repos to Vox is moving all of those helper's over as well so that those can still be maintained. And I think the other thing worth mentioning besides multinode is like under the hood, Litmus uses Bolt as its like remote execution driver, whereas Beaker is just using like kind of raw SSH and WinRM, which is I mean, ultimately Bolt is also just using a SSH and WinRM, but I think that eventually what we are looking at is kind of converging those two. And, you know, it's too early to tell, really like is Litmus going to develop a kind of better user interface to match more of what Beaker has and become the one tool to rule them all? Or is Beaker going to get refactored so that it uses Bolt to drive execution and become the one tool to rule them all, or are they both going to kind of coexist and fill their separate niches as they do now? So I think that's kind of a big part of what we're thinking about within Puppet right now.

Gene LivermanOne other thing of note is that several of those Beaker plugins had not been touched by Puppet in so long that when we went through doing some cleanup a while back, we actually archived the GitHub Repos. And about a year after those repos were archived, we would get a ping periodically in the community Slack from somebody who had been continually using that Beaker plugin without error until some new operating system came out or something like that. And all of a sudden they needed like a two line patch and they were reaching out to us because with repository archived, they couldn't even put up a pull request. And that kind of opened my eyes a little bit about not just how widely it's used, but to a lot of extent how stable a lot of the stuff in Beaker is. It literally can go years without being touched and serve people's needs just fine.

Trevor VaughanYeah, honestly, I don't know if you guys interacted much with Alice Nodleman, but the care and feeding that went into the creating of Beaker was just amazing. And I kind of think of it as like the best secret out there. It's basically it's extremely well engineered, extremely well crafted. When the folks took it over and actually refactored it and split it up into different components and modules, they did a great job in terms of making really solid interfaces. But the documentation just kind of never followed along, which basically means it's kind of like the best kept secret now.

Gene LivermanYeah, that's always been the hardest part for me is that there was virtually no valid documentation for it. It was trial and error and magic voodoo to be able to put it to use.

Demetrius MalbroughMagic Voodoo, hold on, I think that might be a new title. I like it. We're going to keep it. But something jumped out at me when you mentioned that it's not maintained often, but I guess that means it's a solid implementation. But I want to dig more into why it moved over, why Beaker moved over to the Vox Pupuli or however you pronounce it, I still mess that name up. Now, the Vox team, who can answer that for me? Explain that to me.

Gene LivermanI'll start with that one. So Trevor and I had talked on the community Slack and at Puppetize a few times about the future of Beaker, and he had interest in helping with it as I learned that he had interest in helping with that as a byproduct of our conversations. And we did a bunch of talking internally. And what we decided was that Beaker should live with somebody who actually wants it. So it's not that we have any intention to stop using it, but we don't really need to do anything with it for it to keep doing what we need. While people in the community were actually interested in doing the care and feeding of it so that it could do what the community needs it to do beyond just what the company needed to do. And our main community group is Vox. And so we started up the conversations to see if Vox was interested in taking it. In no way did we want to dump it on Vox. There has been a misconception or possibly an accurate analysis at times that Puppet has tended to dump things they don't want on Vox and we actively wanted to avoid that.

Demetrius MalbroughHold on. Who would say that? No one should say anything like that.

Gene LivermanAnd I'll take your word for that. Oh, I wish that wasn't the case, but that was at least a perception that a chunk of the community had. And so with Beaker, we wanted to actively avoid feeding that conception. We didn't want that to be the appearance of what was happening. And so we had been forward and myself and Lucy and I think some others had conversations with various people at Vox and Trevor, and found that there was interest on that side. And so we went through the process of getting all of our ducks in a row on the internal side to actually start transferring the Beaker repositories over to Vox, which for us meant that we needed to make a few structural changes internally, such  as pinning to known versions within our testing pipelines so that we didn't have to worry about changes that were coming in from the community, inadvertently keeping Puppet Enterprise from being able to be shipped one day. We just need to be a little more cognizant of that. So from my perspective, anyway, that's kind of how that happened. Trevor or Lucy might be able to provide some additional context to that.

Trevor VaughanYeah, actually. I mean, so I participate in the Vox community in various ways, as well as many of my team members. And as I said, we didn't want to rewrite Beaker. That seemed to be a complete waste of time, so we're interested in keeping Beaker alive in the Vox community and really helping to make sure the PR is reviewed in a reasonably timely manner. No guarantee that it's a open source community, after all. But we use it daily right. We use it every day. We use the deep parts of it and we want it to keep working. So for those parts we use, we're going to keep them moving along. And to Lucy's point, it will be a heavy lift, but I would absolutely love to see the SSH and WinRM guts of Beaker completely removed and replaced with calls directly to Bolt, I think that would be honestly the best thing ever.

Gene LivermanAnd that does sound pretty awesome, but it also seems like that would help close the gap between the use case of Beaker and the use case of Litmus significantly.

Demetrius MalbroughGreat. Well, it seems like Vox is doing a great job maintaining Beaker thus far, and we really appreciate the entire team to continue on that journey. And I guess what our future plans for Beaker. So if we had a crystal ball and we wanted to take a take a look into the future, what what would the future look like for Beaker, especially around like supported operating systems, etc.?

Trevor VaughanI certainly have opinions. I think this is based on what we need, right. So in terms of operating system support, it's pretty solid. And adding new operating systems has become easier and easier over time. One of the things that actually I completed recently and pushed up was the ability to use podman instead of Docker using the Beaker Docker module and actually required fixing the upstream Docker API rubygem. But it looks like in the very near future we should actually be able to do multinode testing using containers and podman, which I'm pretty excited about. It fully works, I was amazed. Like it spun up multinode, no swarm, no nothing. Just, just boom. It works.

Trevor VaughanAnd I also tested rootless support, so you can actually rootless podman, you can't do multi rootless that has to be on the main system podman because of the IP address name spacing --I'm sorry, the network name spacing-- but it totally works in rootless mode if you don't actually need to do anything multinode, which is really exciting. So I'm excited about that. And of course future looking. I'm hoping that somebody from the Bolt team will actually be happy to help us integrate Bolt and replace the nasty SSH. But that would be actually wonderful. And then, you know, kind of looking at documentation I when I updated the Docker pull again, I went ahead and actually added some of the missing documentation to it. So if we all kind of chip away as a community, then I think we can really have something really solid moving forward.

Gene LivermanAs far as new operating system support, that's kind of a combination effort between Vox and Puppet. As I've mentioned, you know, we use this for testing Puppet Enterprise. So we need to make sure that the OSs that are represented is supported from Puppet Enterprise's point of view are also supported by Beaker. But sometimes Vox is ready for those OSs before we are. And so one of the things this is enabled is those new operating systems to be added at whoever needs it first pace, instead of when Puppet is ready to get around to it pace.

Demetrius MalbroughNow you do know Lucy is on the Volt team, right, Trevor? I'm just kidding.

Trevor VaughanOh yeah. Lucy and I have had many conversations where she's wanted to throw things at me from across the country.

Gene LivermanThat would never happen!

Trevor VaughanLucy's too nice for that.

Demetrius MalbroughYeah. And I didn't hear anyone mention anything about like Litmus and, you know, the wrapper etc. around Litmus. Are there any plans to make that, I guess to maybe make that more friendly?

Lucy WymanActually, yeah. The ecosystem, kind of teams within Puppet, have been meeting just this week to kind of plan out what the next, you know, six, 12, 18 months look like and talk about prioritizing. You know what does the Litmus team want to work on? What is PDK want to work on? And so they actually have a prototype up from Bolt, I know, testing at the moment. And we also know that UX is a big concern. And so I think the results of the many, many, many ecosystem meetings we had this week will become evident in the next week or two. So it's hard to say right now like how much investment the Litmus team is going to get. But I think at the very least, multinode testing is going to be coming up pretty soon. And, yeah, hopefully kind of improved. Like I said, I am not sure how much Litmus is working on those kind of plugins or that kind of like, you know, how does Litmus work with Google Cloud and AWS and all those other kind of wherever you would have your machines you want to be provisioning? So I can't speak to that.

Gene LivermanOne other thing worth mentioning on the Litmus wrapper and those bits is the text output that's generated from running Litmus versus running Beaker is notably different in readability. And that's something that I believe Trevor has brought up more than once. Trevor, do you want to speak to that aspect of it a little bit?

Trevor VaughanYeah. I mean, that's what I was talking about earlier with the rspec based output. So basically, when I get this output right now from Beaker, I get an extremely understandable chain of events. I can loop through nodes, I can pause in between nodes. I can pass communication very easily because it's really just using the procedural power of rspec to wrap everything. And that's one of those items where when we're trying to use Litmus, you know we love Bolt, and when we do things with Bolt, it's awesome that we can do things in parallel and it's really nice, but I don't want them in parallel. And that's been part of the issue for us moving forward right now.

Demetrius MalbroughOK, great. Well, look, I know I've enjoyed the conversation and once again have learned a ton about Beaker and also some of the things that are happening and the importance of multinode testing. And I even learn something about rspec, which is a little scary. But you definitely don't want me at the keyboard trying to do anything with that. But, yeah, it was awesome to have each and every one of you on. Is there any way that the audience can reach out to you on social media? Let's start with with you, Lucy. You want to share maybe LinkedIn profile, Twitter handle or any other social communication?

Lucy WymanI mean, my call to action is going to be like, put up some PRs, people, PRs welcome to Bolt and Beaker now. Um, yeah, the the best way to reach me. I'm pretty and active on Twitter, but I'm @thelucywyman and that's really my only social I want to share.

Demetrius MalbroughOkay. All right. That's OK. You don't have to share your IG or the new clubhouse or you guys have heard a clubhouse, right? .

Gene LivermanNope. Don't know what you're talking about on that one.

Demetrius MalbroughSee you got to get with it. But that's for another conversation. Trevor, what about you?

Trevor VaughanYeah, actually I'm on a totally back Lucy up with PRs. You can find me on Slack, I'll be Trevor Vaughan or tvaughan. I'm pretty much everything. And you can also find me, of course, the OnyxPoint.com.

Demetrius MalbroughAll right. Bring us home, Gene.

Gene LivermanThe easiest way to get a hold of me is through the Puppet community Slack. I am genebean there. So kind of a little green alien looking dude. You can also find me with just my name, Gene Liverman on LinkedIn or other places. On GitHub, I'm also Gene Bean. But the easiest way to find me is through the Puppet community Slack. And in particular, if there's anything that hasn't been transferred to Vox that someone has interest in, be sure and reach out to us. Tell us we're happy to transfer more of the bits of Beaker. We just don't want to transfer things over unless there's an interested party that's interested in helping run with it, because again, we don't want to just be dumping stuff on people.

Demetrius MalbroughAnd Gene, you said Gene Bean?

Gene Livermang-e-n-e-b-e-a-n.

Demetrius MalbroughWhere did you get that from? I am so curious.

Gene LivermanJust a long time nickname.

Demetrius MalbroughOh is it because it rhymes or is the story behind it?

Gene LivermanNo, just cause it rhymes.

Demetrius MalbroughAll right. Simple enough then. I had to mute myself to, to keep myself from cracking up. And I knew you had a story behind that though, Gene.

Gene LivermanI just like most little kids, you know, there's rhyming things that are said about him and some things just stick.

Demetrius MalbroughAll right. Well, we're going to stick with that. I would like to thank you for sharing with us on Pulling the Strings podcast powered by Puppet.

Automate Testing + A Bunch More with Puppet

DEMO PUPPET HERE 👈