homebloga new era of application services at puppet labs

A New Era of Application Services at Puppet Labs

Recently at Puppet Labs, we’ve been putting a lot of thought into how to make our server-side applications faster, smarter, and more modular. Today, we’re excited to give you a sneak peek into the future of some of this work.


Trapperkeeper is a new Clojure framework we’ve written for hosting long-running applications and services. It takes some of the lessons we’ve learned about high-performance server-side architecture over the last few years and combines them with patterns for maximizing modular, reusable code. It’s entirely open source, and we’re looking forward to seeing how the community helps it grow and evolve.

We’ve actually been thinking about this for a few years; the fruits of our original efforts became available in the form of PuppetDB, originally released in May of 2012. PuppetDB has been an overwhelming success for us, so since that time, in addition to improving PuppetDB itself, we’ve been thinking about how to take some of the technology choices from the project, and take advantage of them elsewhere in the Puppet ecosystem. Let’s take a quick dive into some of those technology choices.

The Java Virtual Machine

The JVM is an ideal platform to use for long-running server side applications. It's been battle-tested over the last 20 years, and it’s an extremely fast, stable, and reliable platform on which to run applications like web servers, message queues, and other services. Additionally, it has a wealth of existing tools, libraries, and frameworks that serve as tremendously valuable building blocks to enable rapid development and allow developers to focus their efforts on domain-specific problems.

Several great profiling tools and debuggers available in the JVM ecosystem give developers and users unparalleled insight into the behavior and performance of their applications. Sophisticated instrumentation and metrics APIs allow us to track the most important characteristics of a running application, and expose them in a highly visible, user-friendly fashion. And because JVM-based applications are portable across most of our supported platforms, we don’t need to spend as much time building and packaging things for each individual platform.


Given that Puppet and many of our other projects are written in Ruby, Puppet Labs has been mainly a Ruby shop until recently. The programming language we chose for PuppetDB is Clojure, and we’ve been moving towards adopting Clojure for most of our new back-end projects.

Clojure is a relatively new (and, if you ask us, super-cool) language. Like Ruby, Clojure is a dynamically-typed language, but beyond that it differs from Ruby in many ways, and has a few important features that make it an ideal choice for back-end services development:

  • Clojure is a functional programming language from top to bottom. This means that code written in Clojure is very modular, composable, reusable and easy to reason about. The vast majority of your code is written in a way that is free of side effects (it’s idempotent, like Puppet!), so you don’t have to worry about hard-to-reproduce bugs caused by state changing out from underneath you.

  • Clojure has simple, powerful concurrency primitives. This means that you can write multi-threaded code that can take full advantage of all of the processing power available in a server, without all the complexity of traditional concurrency primitives like locks.

  • Clojure provides high-performance immutable data structures. This is another huge boon for writing concurrent code; you don’t have to worry about putting locks around your objects because it’s impossible for them to be modified by another thread.

  • Clojure has excellent interoperability with the Java language, which means we have access to all of the powerful, well-tested libraries available in the Java ecosystem.

  • Clojure also has a thriving and very active community — a recent study found Clojure’s popular has increased drastically over the last 6 months.

  • Clojure is really fun to write!

Broadening Our Horizons

OK, so, Clojure is awesome, and the JVM is a great platform for building our server-side applications, and PuppetDB has been enormously successful in the field. Next, we needed to decide how to take the best parts of the PuppetDB architecture and mold them into a state that would serve as a foundation for other new products and services at Puppet Labs.

We found that many of our applications needed a lot of the same functionality that has become commonplace in today's application frameworks:

  • the ability to configure and control which parts of the system are loaded at run-time

  • the ability to compose modular bits of functionality

  • a way to cohesively manage the lifecycle of the various components making up an application

  • a common logging mechanism that wouldn't need to be set up and configured by each component

  • a web server flexible enough to load multiple web apps, and to support a rich set of options for configuring SSL

Of course, our first inclination was to use an existing solution for this; after all, we’re not in the business of re-inventing wheels when we can avoid it, and we’re certainly not the first shop who has had a need for something in this space. So we allocated some time to survey the landscape and see if any existing solutions were a good fit for our needs. Here are some of the options we explored.

Alternatives to "Build it Here"


OSGi has some features that were very attractive to us; in particular, it provides a Service Registry that can be used to define services, load them together into the JVM, and manage dependencies between them. Services are defined via Java interfaces, so it’s possible to write different implementations of a given service and swap them out without affecting other services they depend on, or provide functionality to.

However, OSGi can be a fairly heavyweight solution: It’s easy to drag in a lot of dependencies that result in a very large file size for your application’s redistributables. Also, OSGi provides a lot of very powerful — but very complex — functionality to allow you to hot-swap service implementations in a running JVM. It relies heavily on very advanced usage of Java’s classloaders to manage this, which can often lead to the phenomenon known as classloader hell. The flexibility that this usage of classloaders provides comes with a cost in terms of complexity, and one of the main tenets of Clojure is that avoiding complexity is key to writing maintainable software.

Furthermore, we don’t really need a lot of the hot-swapping features provided by frameworks like OSGi. These kinds of features are critical for SaaS applications, but not always useful for on-premise applications like those we build at Puppet Labs.

JBoss / Immutant

JBoss is a popular Java enterprise application server. There is a related project called Immutant that provides Clojure bindings for most of the JBoss functionality. Immutant brings a lot to the table, and we gave it strong consideration… but in the end, we decided that there were a few details that prevented it from being an ideal fit for us.

  • The Immutant/JBoss runtime is a pretty large, full-featured deployment artifact. The "slim" version is in the ballpark of 50MB, and the “full” version is in the ballpark of 125MB. Those sizes are perfectly reasonable for SaaS applications, but felt a bit unwieldy for Puppet Labs’ on-premise use cases, partially because we have to be careful about bloating the size of our installers too much, but also because in most of our applications / services, we’d only be using a small subset of the functionality.

  • Immutant ships with some predefined technology choices: JBoss web server, HornetMQ message queue, Infinispan key/value cache, etc. While these choices are all perfectly reasonable, we were looking for something that would allow us to choose the specific technologies for individual subsystems. We were also hoping we could find a solution that would allow us to swap out other technologies for the subsystems if the need should arise — for example, a solution that would allow us to decide to switch from one message queue to another without necessarily changing much of the consuming code.

  • Since we package all our own software at Puppet Labs, we need something that is lightweight, modular, and easy to fit into our packaging paradigm. We also need to be able to easily package our application code along with the framework as a single distributable. Immutant and JBoss are more amenable to the model where you install them first and separately, and then load your app into them afterwards.

Clojure Frameworks and Libraries

We also evaluated a few other Clojure-specific frameworks, including Jig and Component. These both look fantastic and useful, but neither seemed like quite the right fit. That said, the authors have done great work, and their projects provided useful inspiration for us, so we highly encourage you to check them out! They may be a perfect fit for your project, even though they didn’t seem like quite the right fit for ours.

At the time we evaluated them, Jig felt more targeted to developers (as opposed to administrators / end users), and not necessarily intended to be something you would ship with an on-premise application. Component required you to define your application structure in code, and we needed something more dynamic and configuration-driven. Neither had built-in solutions for things like logging and service configuration, and neither seemed to provide a solution for specifying an "interface" for a service that would allow you to swap out alternate implementations.

Enter Trapperkeeper

After careful consideration, we decided it would be worth our while to build our own Clojure services framework. The result: Trapperkeeper, a new open source project from Puppet Labs. You can think of Trapperkeeper as a sort of "binder" for Clojure Ring applications and other modular bits of Clojure, Java, or JRuby code. It is a Clojure framework for hosting long-running applications and services. Of course, the word "service" means very different things in different contexts. In Trapperkeeper, we define a service as:

  • a set of functions

  • a set of dependencies on other services

  • a simple lifecycle

To use these services, Trapperkeeper also provides a basic application framework. At startup, the framework decides which services to load based on a configuration file, and it also provides a common configuration mechanism to each service. This enables the user to configure the list of services that should be loaded into the framework, as well as providing a way to configure each individual service.

So, what does all this mean to you? That depends on whether you’re wearing your Clojure developer hat or your Puppet user hat. For those of you in the former camp, check out the github repo, and stay tuned for another blog post in the near future where we’ll explore Trapperkeeper in depth, in all of its Clojure-y glory. For those of you in the latter group, here are some of the reasons we’re really excited about the project:

  • It will help us accelerate our move to a more robust and scalable platform for all our new server-side projects.

  • It will make it much easier for us to re-use the various architectural components of our server-side stack across a broad range of Puppet Labs applications and services, so that we can focus our energies on the core features that matter to you.

  • It includes some of the core technologies that have been successful in the field with PuppetDB, so we have confidence we’re building our new products on a solid, battle-tested foundation.

  • It will mean that we have consistent, simple configuration formats (including logging configuration) across all our new applications.

In short, we’re excited about the new applications and features we have in store for you in the near future. We’re excited about building on a robust and scalable platform, and we wanted to give you a sneak preview of some of the related technology. Stay tuned for more good things to come!

If you are interested in more technical detail about Trapperkeeper and how we're using Clojure, check out our follow-up blog post.

Tell Us What You Think!

We love hearing from our users and our community. At the end of the day, you are what keeps the lights on over here at Puppet Labs — so, your opinions are very important to us. If you have thoughts on any of our current features or suggestions to help us shape and prioritize features for upcoming releases, we really want to hear from you! Here are some great ways to get in touch with us:

  • Sign up to be a Puppet Test Pilot! You'll get free goodies as a reward for your participation if you are able to volunteer a few minutes of your time to tell us what you think about prototypes of upcoming features.

  • Ping us on IRC: we're usually online in #puppet on freenode.

  • Send an email to the puppet-users or puppet-dev mailing lists.

Chris Price is a software engineer at Puppet Labs.

Learn More

Resources for learning about and getting started with Trapperkeeper and PuppetDB: