What is Onceover testing you may ask? A tool to automatically run basic tests on an entire Puppet controlrepo, it was written mostly while sipping inflight cocktails on airplanes. Dylan Ratcliffe explains Onceover testing and addresses some of the reasons it exists in the first place. Tune in to gain the clarity you need for your automation woes.
- Get started with Onceover.
- Learn how to use Onceover to start testing your Puppet Control repository.
00:00:15Demetrius MalbroughHey, everyone, thanks for joining this episode of Pulling the Strings podcast powered by Puppet, and I'm delighted to be your host. My name is Demetrius Malbrough and I'm on the product marketing team here at Puppet. I'm really excited today to talk with Dylan Ratcliffe. Dylan has been working at Puppet for five years. He loves automating his job, his home, riding motorbikes and playing board games. Dylan, how are you today?
00:00:44Dylan RatcliffeDoing really well. Demetrius, how are you?
00:00:46Demetrius MalbroughI am doing great. So let's go ahead and start off nice and easy here and jump right into the question. So we're going to talk a little bit about, you know, onceover testing. First off, what is onceover testing and some of the challenges or struggles that was born out of it.
00:01:07Dylan RatcliffeYeah. So onceover is a testing framework for testing roles and profiles as a sort of a higher level, sort of like integration testing for your Puppet code. It was born out of a problem where it's been very common to test Puppet code using rspec-puppet or the Puppet development kit or some combination of lots and lots of different open source tools for a long time. And most of the modules that you download from the Puppet Forge are really well tested and people have put a lot of effort into write the tests and stuff like that. But when you're just consuming those modules, you know, you're grabbing stuff off the Forge, you might have 30 different modules and combine them together to produce a machine that does what you want. But it's pretty rare for people to test all of that stuff together in concert because it's all tested individually. You know, each module works on its own and writing all of the tests sort of all over again to test that it works in the specific way that you've combined it together is a lot of work. And I wanted to write something that removed all of the work in writing that.
00:02:20Demetrius MalbroughAre you the primary author of onceover?
00:02:26Dylan RatcliffeYeah, I do a fair bit of it. I certainly started the project and we've got a whole bunch of contributors now from inside Puppet and from the community who are doing a really good job of helping me out. Yeah, I started it.
00:02:42Demetrius MalbroughAnd I also heard some from somewhere that some of the writing was done on airplanes as you were traveling around the globe, is that correct?
00:02:51Dylan RatcliffeYes. So I used to be a professional services engineer, which is we went to customer side and helped with installing Puppet and designing workflows and scaling it and then doing lots and lots of stuff. And so I was on the road pretty much every week for a while there, there was a while where I was home four or five days out of 80, I think. And so I did a hell of a lot of time on planes and in hotel rooms and waiting for public transport and stuff like that. And that's where a lot of this got written. The planes one was a bit hard, because you can't Google things when you're on a plane. Well, at least not when you're flying over the Atlantic, you can't. And so I had to do a lot of learning of Ruby things by rote because I can't just Google: "How do you do this in Ruby?" That was a bit of an experience that I'm not used to.
00:03:47Demetrius MalbroughOK, well sometimes that's the best learning, right, is when we don't have Google or we didn't have Google. Like what did we do before we had Google? Like I guess we, now I don't know I might I might show my age here, but I have encyclopedias back then. So there were only these big, thick books that you had to rely on to get information for the most part. But we won't go there, so we'll go with the thing that, you know, Google is awesome and you can just type in whatever. And before you finish typing it, it just pops it up for you. So back to onceover testing. Is onceover testing is this something that's only done for like a certain size type of deployment, like for small or large or medium? Could you give more information around the type of deployment that once over testing would actually fit for, or is there a fit?
00:04:38Dylan RatcliffeThere's definitely a fit, but it doesn't have anything to do with the size. The onceover testing is specifically designed around the concept of roles and profile. So you familiar with that?
00:04:50Demetrius MalbroughEnlighten me. And I'm sure someone else may not know as well, so go ahead.
00:04:55Dylan RatcliffeOkay, cool. So roles and profiles is a standard that came out of the community, I possibly might have been Craig Dunn's blog post is one of the early ones. The Shit Gary Says blog also has an excellent post on it, and it's continued and grown out of the community for a long time. It's a way of structuring a Puppet code that sort of is a bit standardized and logical and splits the work you do into layers of abstraction that make it easier to maintain. At a real simple level, a profile is how your company does a certain thing. So where you might download a module of the Forge that installs a Microsoft SQL database, that module will do that database however you want. It doesn't care, it's like here are a million parameters and you put in whatever you want. I don't mind, but your company will have standards for how it's done and you'll personally have ways that you want to set it up. And so that's what a profile is. It's like it's probably just something straight off the Forge, but with opinionated stuff passed in that you've done.
00:06:04Demetrius MalbroughSo it's customized to your liking.
00:06:07Dylan RatcliffeExactly. OK, and then our role is just many profiles put together to make up a useful server that the business actually wants. So, for example, the business might want a public facing Internet banking server that runs the bank's Internet banking website, and they'll be a lot of components of that. And each of those components will be how the bank does the baseline, how the bank does monitoring, how the bank does security. And these are the profiles and they all roll up together into a role. And a server gets one role. And that concept meant that testing everything becomes quite easy because every server has one role, which means that as long as you are testing every single role, you're testing every possible combination of all the stuff that you combine together in all of your Puppet code, because the roles represent a specific combination that you use. So now if you're using roles and profiles, which a lot of our customers are, and if you're not, I thoroughly recommend reading the blog posts and looking at refactoring to use roles and profiles. But if you're using it, it means that you have a set list of 'these are the types of things we build' and once you have that list, you can take that and go, all right, here are the things we need to test because there's a set list of stuff. So onceover takes that list of 'here are the things we build in your rolls' and tests them all together as a role. It doesn't test like all the individual components separately. It just puts them all together in the way that you've already done with your role and makes sure that in that specific instance, they all work together. It doesn't care about like, well, if you did something different, it wouldn't work, doesn't care. Says with this specific instance that you've done in this role, everything works and it's a pass or no, there's some kind of a conflict and it's failed or whatever it is.
00:08:09Demetrius MalbroughAll right. So this sounds almost like a gateway drug to automated infrastructure testing. Is that is that a fair statement?
00:08:16Dylan RatcliffeSo that's what I see this project as well. Actually, that's what I saw this project as in the beginning, I thought that if I gave people an easy way to test just their roles and it's not even like a particularly good test, to be perfectly frank, it tests, it compiles, but it doesn't test anything else. It doesn't like apply loads and loads of different, you know, different situations to it and stuff like that. It just says if you were to run this, would it compile into a catalog, yes or no? And I totally thought when I developed this that that wouldn't be enough for people and people would use this to begin with, but then they would go on to write their own much more complex and much more complete tests to, you know, to flesh out all of the little idiosyncrasies that they might care about. But to be honest, it hasn't happened. I really thought that it would, but it hasn't for most people that use one. So there certainly are some users that really extend it and do test quite a lot of their own stuff. But for a lot of people, that level of just making sure that everything compiles after you've made a change is sort of the it's like the 80/20 rule. It's like 20 percent of the effort will catch 80 percent of the problems. And that's what this is. It's the 20 percent of the effort to set onceover up or catch 80 percent of the problems. And I thought that people would then go and put in the other 80 percent of the effort to catch the last 20 percent of the problems, but to be honest, a lot of people just deploy their code to a test environment and catch the problems there or use some existing mechanism that they already had. They generally sort of don't take testing to that nth degree, in my experience, because people already have testing environments. They have to CD4PE. They can roll changes through a pipeline and if it breaks, roll it back. And that's kind of, a lot of that stuff already there, which I didn't think about when I developed it. But I'm kind of glad because it means that it's been a more complete solution for a lot of people than I really ever thought it was going to be.
00:10:40Demetrius MalbroughYeah. Dylan, where did that name onceover come from? Because I'm sitting here thinking, you know, once over. So is it something like, you know, once you're done, you can test it and it's over or, you know, where that name come from?
00:10:56Dylan RatcliffeHave you ever had you ever heard the phrase, like, 'to give it a once over'?
00:11:00Demetrius MalbroughYeah, I have. I have.
00:11:02Dylan RatcliffeIt's like kind of peruse, like have a look at it, make sure it's kind of okay, kick the tires a little bit and then send it out. And that is essentially what it does. It's really not taking it apart and making sure everything's great. It's just sort of walking around like, you know, like when you hire a hire car [rental], if you've done a lot of hiring a hire car, when you hire the first one, you really, really check it. But then you get to the point where you're like, walk around, it's got four wheels. It's fine. It really just kind of walks around, has a little bit of a look, kicks the tires a little bit and then it's like, all right, cool, it's, it's good to go. And that's super helpful for people, which as I was saying, I definitely thought that people would see that and be like, I really want to do more testing and you can. But the value of just doing that, but doing it every single time is super important, especially because onceover tests every role, not just the one you're working on. So you might be working on one specific profile and for what you're testing, it works, but it turns out that somebody else is using that same profile, but they're passing some different stuff to it. And that's going to cause it to break with the changes that you just made. But because you were testing against your stuff, you haven't thought about that. So even though it basically just walks around and kicks the tires, it does that to every single role, every possible sort of use case, which means that while yours might be fun, it goes to kick the tires of this other person's role and it all falls apart and then, you know, oh, God, I've broken it for somebody else. And so it does provide a lot of value there.
00:12:37Demetrius MalbroughAnd I guess, you know, back to the name, you know, once again. So where did the inspiration come from and were there any other products similar that were out there? And you say, you know what, I need to create something that's a little different, a little better that can do what I want it to do. Is there anything out there like that? I guess, one, when you were thinking of, you know, writing this onceover testing?
00:13:00Dylan RatcliffeWhen it comes to the name there was, I'm trying to remember what possible names were, naming stuff like this is so hard. I really hate doing it. It's so difficult to pick and I struggle. And then you pick a really good one and then you search on it or someone's already using it for something. And nobody was using this and it was the first sort of reasonably good idea I had that nobody was using. And so I just jumped on it. But in terms of the the actual like what it does, is there anything that does the same thing? There was wasn't. Yeah, no, not at all. There are the tools that onceover uses in the background are staples of automated testing with Puppet and that sort of written by the, you know, the Giants on which on whose shoulders all this stuff stands. And like the rspec-puppet tool by Rogich and all of the Puppet Labs spec help plumbing and stuff like that, there's all this open source tooling, which is awesome. But everybody sort of had to roll it together themselves, especially when I wrote this, because the PDK didn't exist either. And so there was a lot of tools that if you combine them together in the perfect way, would do this. But there was no one tool that did it. And since with roles and profiles, a lot of it is standardized. Like, for example, every node only has one role and a role represents like the what the server does for the business. That part of it was standardized, but the toolchain wasn't. And so there really this was the thing to just pull all of those tools together in a standardized way so that both sides of it could be standardized. The PDK has done much the same thing for modules, and the PDK doesn't yet support control reapers. So PDK sort of has done the same thing for modules that onceover has full control reapers, hopefully at some point we will just integrate the two so the PDK will do all of it. And when that happens, it'll probably go out to onceover to do it. But at the moment, it's sort of absolutely stands on shoulders of giants. There's not a hell of a lot that onceover does that isn't reliant on some other thing, but it does it in a standardized way and abstracts away a lot of the complexity so that you don't have to deal with it.
00:15:27Demetrius MalbroughDoes the term 'octocatalog-diff' mean anything to you?
00:15:32Dylan RatcliffeYes, it does.
00:15:34Demetrius MalbroughOK, tell me a little bit about that, because, you know, as I was preparing for the episode, that's something I saw out there. And I guess I want to better understand what that does as well. In addition to, you know, onceover testing.
00:15:49Dylan RatcliffeYes. Octocatalog-diff is an awesome tool that was written by the people over GitHub to differentiate between two versions of the same catalog. So basically, if I made a change with Puppet, I wanted to update the version of some package. I want to make sure that when I do that, that's actually what happens. Like, it's not that it updates the package and makes ten other changes. I want to make sure it just updates the package and octocatalog-diff does that. It compiles a catalog with the old code and it compiles a catalog with the new code. And then it compares them to say, all right, here's what you changed in the code and here's what the actual result was in the catalog. And it's super valuable for verifying that what you've done makes sense, it's a logical thing to do. And it really takes what onceover does substantially further because onceover just compiles a catalog and tells you whether it worked, octocatolog-diff compiles two catalogs, tells you whether it worked, then compares them, tells you the differences. And so it does a lot more, but it's also very flexible, like all of the tools that onceover is built on, like rspec-puppet and stuff like that, which means there's a reasonable amount of setup required to get octocatalog-diff working based on your repository, you've got to make sure your modules are installed, that you give it all the right directories and stuff like that. And that was one of the things that I wasn't willing to compromise on onceover, I was like, you shouldn't have to know that sort of thing. And so onceover, while it totally does not have anywhere near the functionality of octocatalog-diff, it's much simpler in what it does. But it's also much more opinionated. And it just as set it up like this than just run onceover and it'll just work. So maybe a little bit less flexible, but also a little bit easier if you're sort of just getting started and you don't really care how you set it up, you just want to work. And also, though, I wrote a plugin for octocatalog-diff for onceover. If you have your controller already configured using onceover and you want this sort of catalog diffing functionality, you can use my module. It's called onceover-octocatalog-diff, which integrates the two. So basically onceover handles the complexity and passes all that complexity to octocatalog-diff, then octocatalog-diff does the rest. And if you find any bugs in it, let me know because there might be some. I haven't checked it for a while, but it definitely worked when I wrote it.
00:18:49Demetrius MalbroughI guess with octocatalog-diff, you don't really need a Puppet server to actually use it, is that correct?
00:19:51Demetrius MalbroughAll right. Well, we have run out of time here. I think that was the last question that I had. Unless there's something else that you feel is like super relevant to kind of give details to the Pulling the Strings listeners here, but other than that, I feel that I'm walking away with some information that I didn't know before. Also, before I let you go, you seem to favor dogs, right? Like, your dog. You don't own a dog, but you like pugs. Is that correct?
00:20:28Dylan RatcliffeI like all dogs, but especially Pugs.
00:20:32Demetrius MalbroughI guess we won't ask where that came from, unless you want to answer it.
00:20:39Dylan RatcliffeSo my grandma had pugs and so she, she loves pugs and the rest of my family think about pugs the same way that a lot of people do, is that they're weird and gross and they make gross breathing sounds. My grandma and I love them and she sends me a newspaper clippings. I live in London, I used to live in Australia, sends me newspaper clippings of pug stories. Any time there's a pug in the newspaper, she cuts it out and sends it to London. And then I have them filed away.
00:21:12Demetrius MalbroughSentimental value there. It isn't like your grandmother's like she didn't bake, but she raised pugs. Maybe she baked too, though.
00:21:22Dylan RatcliffeShe bakes, she makes a mean jam tart.
00:21:41Demetrius MalbroughIs there any way you would like for anyone to reach out to you on social media?
00:21:46Dylan RatcliffeI don't actually have a Twitter or anything, but if you want to reach out on onceover specific stuff, submit a ticket on GitHub or a Pull request or whatever it is, you have questions, open it up there. And we have like a fairly active community. People are pretty active in helping people. Or if you have all the onceover questions, the testing group in the Puppet Community Slack has loads of onceover users in there as well. So people can if you go in that people can help you. In terms of reaching out to me, I do not have Twitter. You could I guess you could message me on LinkedIn if you wanted.
00:22:23Demetrius MalbroughAll right. You keeping it strictly strictly professional, yeah, go to GitHub and submit a Pull requests. Dylan, it's definitely been a pleasure having you on Pulling the Strings and to have a little talk about onceover testing and octocatalog-diff and the gateway drug to automated infrastructure. So thanks for coming on the show.
00:22:46Dylan RatcliffeThanks, Demetrius. Thanks for having me.