homeblogdeveloping puppet server interactively

Developing Puppet Server — Interactively!

With the release of Puppet Server 1.0 just around the corner, we’ve been working on some finishing touches that really begin to showcase the capabilities of the new architecture. You’ll see examples of this in the product itself (now available in Puppet Enterprise 3.7), but there are also huge advantages surfacing in the daily workflow of developers who are working on the project. In this post I’d like to highlight a few of those; if you’re a developer who wants to hack on Puppet Server or are interested in Clojure, hopefully you’ll find this informative. If you’re a Puppet user who doesn’t dive into the source code too often, this post may be a little towards the extreme of “how the sausage is made,” but it’ll highlight some of the power and dynamism of our new architecture nonetheless.

So, let’s get to it!

The Clojure REPL

One feature that makes Clojure a joy to program in is the Clojure REPL. REPL stands for “Read-Evaluate-Print Loop”, which is basically just a fancy way of describing an interactive shell into which you can load or type code, and it will be evaluated and the results printed to your console. If you’re familiar with Ruby, you’ve probably used “irb”, which is a similar tool.

REPLs have been around for a very long time — for example, the Smalltalk language and many Lisp dialects prominently feature a REPL — and Clojure seems to be re-popularizing some of the development patterns that those languages established. For most Clojure developers, the REPL is a much more integrated part of the programming workflow than what you see in other modern languages.

There is excellent support for editor/IDE integration with a Clojure REPL session, so it’s really easy to load, re-load, run, modify, and test your code directly in the REPL. Once you get used to working this way, you may find that you rarely do any development work without having a REPL open, and you may wonder how you ever survived without it :)

(For more info on Clojure REPL development workflows, check out Jay Fields’ blog post on REPL-Driven Development, or just google and you’ll find lots of information.)

Let’s walk through a really simple example of something you might do in a REPL session. (If you’d like to play along at home, clone the Puppet Server git repo and change directory to its root.) First, we’re going to fire up the REPL:

Now, let’s run a few Clojure functions:

OK, so far so good. The REPL can interpret Clojure code. (Big whoop!) Let’s do something more interesting. Let’s load some actual code from our project and see what we can do with that.

For this example, we’re going to play around with a function that is defined in the Puppet Labs “kitchensink” Clojure library. The function is called true-str?, and we can look at its source code and play around with the function from directly within the REPL.

OK, great, it works. But now let’s do something more interesting…let’s interactively change the definition of that function, live, in the running REPL session:

If you’re not familiar with Clojure, what I did there was to switch to the namespace where the true-str? function was defined, redefine it to always return false, and then switch back to our user namespace. Then, I called the function a few more times — with the same inputs that we’d given it before — and we can see that it is indeed returning false all the time now.

(Note that in practice, it’s not all that common to modify a function definition at the REPL shell like this. Most editors/IDEs that are Clojure-aware have nice REPL integration, so you can actually modify the original source file and then have the editor send the changes over to the running REPL. The end result is functionally similar to what I’ve done here, but has the benefit of preserving your changes in a real source file after the REPL exits.)

This is interesting, but it’s a very simple example, as I said earlier. What’s really noteworthy about this is that we’ve changed the definition of the existing function puppetlabs.kitchensink.core/true-str? globally, so now any other code that calls that function will use our new implementation for the duration of this REPL session.

You can imagine a more complex scenario in which we’re loading up much larger chunks of our application code in the REPL session, surgically redefining individual functions, and then running application code to see how the changes affect it. Then it’s clear his approach really starts to give the developer a much shorter feedback loop… no editing files, saving them, re-compiling/launching the app to see how a change affects things. Just modify the code at run-time, in a live system!

Those are some basics of the Clojure REPL. Now let’s get into some more powerful stuff related to the life cycle of our application.

Reloaded Workflow

A pattern that I’ve written about briefly several times in the past, and which has seen a good amount of uptake in the Clojure community, is the “Workflow, Reloaded” pattern based on this blog post by Stuart Sierra. The general idea is that you avoid having any global state for your application, and write your code so that the application can be instantiated as a self-contained object. Then you can call functions to start or stop the application, and if you make changes to the code, you can call a reset function that will destroy your application object, reload any code that was changed on disk, and construct a clean new instance of your application. You can do all of this interactively in the REPL without having to shut down and restart the JVM process.

We designed Trapperkeeper, our open source Clojure framework that Puppet Server, PuppetDB, and other apps are built on, to be compatible with this workflow. In the Trapperkeeper documentation, we provide some example code that illustrates how to set up some REPL utility functions that allow you to manage the life cycle of your Trapperkeeper app in the REPL. Most of our downstream projects contain an example REPL utilities namespace that’s already all set up to run the application. Let’s check out one of these example namespaces that is included in the Puppet Server source code.

Running the Server

The Puppet Server source tree includes extensive documentation for how to run the server from source, as well as a Clojure namespace that contains some pre-built utilities for starting and stopping the server. Getting started is as simple as this:

And just like that, I have an instance of Puppet Server up and running right there in the REPL! I can now do agent runs against it, or just poke at it with curl, or whatever I like.

From here, I could call (stop) to shut the server down, or if I change some of the code (either Clojure or Ruby) on disk and want to restart the server with the latest changes, I can just call (reset):

With this approach, it takes only a few seconds to get a brand-new Puppet Server instance running in the REPL, and we can iterate really quickly when working on changes to the server. But really, this only scratches the surface of how we can interact with the server in the REPL. It’s time to take it up a notch!

Interacting with the Server

Clojure in general, and the “Workflow, Reloaded” pattern in particular, both advocate trying to minimize the amount of state that you have to manage in your application. This makes it much easier to maintain and reason about your code, but no matter how pure your motives, you’re probably still not going to end up with a useful application that has absolutely zero state to maintain.

In the case of Puppet Server — and really, all Trapperkeeper applications — the required state information is managed as a single nested map of data. Each individual sub-component of the application may own a given subtree of that map of data. So, for example, there is a section of the map that maintains some state information about the embedded Jetty web server, and another section of the map that holds data about the pool of JRuby interpreters that Puppet Server is using to run the Puppet Ruby code.

One of the great things about doing development in the REPL is that we can easily inspect (and even modify) all of this state data at run time. In the Puppet Server REPL utilities namespace, there is a utility function called print-context that we can use to examine the state data. Let’s use that function to see what the state of our Jetty web server looks like:

If you’re not a Clojure developer, there are probably a few spots in that output that aren’t entirely clear; don’t sweat it, they’re not really that relevant in terms of the broader point we’re trying to convey here. But you can see that there are some paths to the pem files that we used to configure the server’s SSL, and there’s also a reference to the main Jetty Server object that represents the running web server. So we can actually grab that object and do things like assign it to a local variable or call methods on it (using a utility function called context that is defined in our user-repl namespace):

In the example above, I’m just calling getURI, to have the server object tell me which URI it’s available at. But you can imagine that having a reference to the live server object and being able to call any of the methods it provides is a very powerful feature. In the traditional Apache/Passenger Puppet master stack, it’d be analogous to running Apache in an interactive shell and having access to inspect and control all the Apache internals dynamically while the server was running. Whoa!

This is really cool, but it gets even better when we start playing around with our own domain objects (e.g., parts of the system that actually come from the Puppet Server source code). As an example, we recently started working on a feature that would allow a user to dynamically request a flush of the directory environments cache. (If you’re not familiar with Puppet’s new directory environments, or the related timeout/cache settings, don’t fret. This article is more about illustrating the possibilities of interacting with the running system than with any of the particular details about the caching.) Puppet Server now keeps track of some of the state relating to the cached environments, and you can examine it from within a REPL session (we created a new REPL utility function that makes it easier to do so):

So here we can see that we have four JRuby instances, and they are each aware of only one environment, :production. Further, the cache is not expired for any of the instances ({:expired false}). Now, outside of the REPL, if I trigger a Puppet agent run against an environment other than :production, and then print the environment states again, here’s what I get:

We can see that one of the JRuby instances is now aware of the new environment called :foo_environment. So we’re able to inspect the state of the running system, even while things are changing based on external influences. But it doesn’t stop there; we have another utility function that lets us mark all of the environments as expired:

The function mark-all-environments-expired! modified the state of the running server, and then when we call our print function again, we see that all of the environments are flagged as {:expired true}. This means that if we do another agent run, we’ll be guaranteed to get the latest version of all of the changes in the environment — the cache will be flushed.

Again, caching implementation details aside, the interesting point here is that as we are working on new features for Puppet Server, we can continue to add domain-specific REPL utility functions to our utility namespace. That allows us to achieve really rich and powerful interactions with the running server from right inside of the REPL. We can inspect the state of the system and even make changes to it without shutting it down or restarting it. This allows us to do development and testing on features, such as the cache flushing illustrated here, without even having exposed the functionality to outside callers yet!

Tell us what you think

We’re really excited that the new architecture underlying Puppet Server is going to open a lot of doors to new features we can add to Puppet. It’s also exciting to start to see some of the ways we can leverage the technology to provide a dramatically improved development experience. I hope you enjoyed this brief glimpse into what some of those workflows look like for us, and hopefully this will inspire a few of you to play around with running Puppet Server from source yourselves.

Let us know what you think! Here are some great ways for you to get in touch with us:

  • Sign up to be a Puppet Test Pilot. You'll get free goodies as a reward for your participation, and it takes just a few minutes of your time to tell us what you think about prototypes of upcoming features.
  • Ping us on IRC: we're usually online in #puppet, #puppet-dev, and #trapperkeeper on freenode.
  • Send an email to the puppet-users or puppet-dev mailing lists.

Chris Price is a principal engineer at Puppet Labs.

Learn more

Resources for learning about and getting started with Puppet Server and Trapperkeeper: