Published on 23 September 2014 by

We’re talking about a lot of important updates to nearly all of our software at PuppetConf this week, and I am excited to be able to share more details here.

I started Puppet Labs about nine and half years ago, and Puppet’s age has been on my mind a lot recently. There’s so much value you get from the maturity of a project that’s been around that long, and there are also so many areas where you’d like the opportunity to rethink some core aspect of what you’ve built. When software is as widely deployed as Puppet is, it becomes tough to make significant changes, so it’s easier to let it ossify rather than risk the disruption that significant change can bring.

We aren’t willing to let that happen with Puppet. We know that constant evolution is a necessity for survival, and the changes we’re talking about this week at PuppetConf, in our open source projects and in Puppet Enterprise, are big steps toward enabling the next 10 years of evolution.

We’re working on rebuilding our entire client-side technology stack, so it takes fewer resources, runs faster, and is more maintainable. The first step toward that is cfacter, which will become Facter 3 in early 2015. We’ll have packages for it soon, but you can try it now if you’re comfortable getting your hands dirty: Compile it from source, and use the --cfacter flag to Puppet. In the long term, our whole client-side stack will also be rebuilt in smaller packages and in compiled languages. We’re using C++ right now, because our stuff has to run absolutely everywhere.

On the server side, our services are moving to the JVM, generally written in Clojure. You’ve already seen how great this has worked with PuppetDB, and we’re using it as our example. We’re announcing a new Puppet Server, which replaces your Apache and Passenger setups with a more resilient, powerful, and scalable front end. Combined with client-side improvements, we’re seeing roughly 3x faster request response times and 2x faster catalog compile times. So far we’ve rewritten the certificate authority (CA) service in Clojure, and over time, we’ll get through the rest.

We’re also making significant changes to the language. Many of you are already familiar with the work done in the future parser, which will become the default parser in the next major release of Puppet in the next few months. We’ve improved error handling and added data typing to make it easier to write high-quality code and to improve the mean time to recovery when there is a problem, and we’ve added iteration and inline templates to make the language a bit more powerful. It’s always a challenge to find that balance between power and simplicity, and we think these strike the balance well.

As you can see, we’re not letting maturity get in the way of major changes to our core projects. We’re working hard to make moving to these new versions as painless as possible — for example, by supporting your existing Ruby Facter plugins in cfacter, and continuing to follow semantic versioning, so you can be confident about how much change to really expect.

Speaking of maturity and evolution, we think it’s critical that we allow the core platform to evolve separately from the applications and functionality built above it in Puppet Enterprise. Toward that end, we are beginning to build our new functionality as separate Puppet Apps. These will be installed, updated, enabled, and managed separately from the core infrastructure and from each other. This decoupling will enable them each to evolve at their own rate, and at a faster rate than the core platform.

The first Puppet App we’re releasing is the Puppet Node Manager, which is an application built to cover the entire lifecycle of node management. The core functionality of this new application is a new rules-based node classifier, which will help you treat your nodes more as cattle and less as pet. The new classifier allows you to group your nodes based on their facts, so you should rarely have to classify individual hosts.

The new Puppet Node Manager

We’ve also built a new activity service that plugs into the new Puppet Server. This service generates statistics about everything happening in the Puppet Server, such as active connections, functions being run, and what kinds of requests it is seeing. We’re initially providing a configuration for Grafana and Graphite to build a dashboard that will track your infrastructure, making it easy to get a consolidated view of how all of your masters are performing, and where bottlenecks might be.

Puppet Server Reporting

There’s a lot more going on in the new release, and I’d encourage you to sign up for the live webinar we’ll be holding to talk about what’s new in Puppet Enterprise. In the meantime, we’d love to know what you think.

Luke Kanies is the founder and CEO of Puppet Labs.

Learn More

  • Haven't tried Puppet Enterprise yet? You can download and use it for free on up to 10 nodes.
  • Already using Puppet Enterprise? Upgrade to version 3.3, and get its additional capabilities.
  • Check out the press release about the latest improvements to Puppet.
  • PuppetConf videos are now available for 87 talks and workshops. Watch now.
Share via:
Posted in:
Tagged:

" We’re using C++ right now, because our stuff has to run absolutely everywhere." When you're using "right now" in this sentence, does that mean that you are looking of moving to some other language in the future for the client side and if so to which? I was under the impression that Puppet Labs would move it's client stack to Golang however given that this won't run on RHEL5 and this is still present in a lot of companies, did you decide to use C++?

Matthias Viehweger

This seems all fine and sensible for installations in the scale of netflix and google.
We have 11 nodes and if I would put each puppet service onto its own machine, we would have 14. I am really concerned that this is adding a lot of overhead for small installations.

I like the graphite-backend and the performance gains in theory. I just feel that puppet is no longer a solution suitable for small installations.

Hi, Matthias,

It's true, a lot of development now underway benefits larger and growing environments. But we're working to not add overhead, both because we believe Puppet is for nearly everyone, including people running small environments, and because unnecessary overhead isn't great for anyone. For example, you absolutely can continue to run all the Puppet services on the same machine.

A lot of what we're doing (described in the post) should benefit customers of all sizes. Reporting, by the way, is additive and not required, and its resource requirements are pretty low. If reporting to your stats collector did end up being expensive for any reason, you could choose to schedule less frequent collection so it won’t impact performance in your environment.

Best regards,
Susannah

Pierluigi Vernetto

I absolutely love the concept of "lingua franca" to communicate infrastructure configuration, but I strongly doubt that Puppet - a non OO language/DSL who at most can express data as messy, brittle, non-refactorable code and yaml hashes, can ever become this lingua franca.
If I may give an advice to Luke Kanies, it's to hire some real good mathematician and language design expert OR just build his DSL on top of some strong OO language like Java, Scala or Python, which support all best Software Factories best practices and tools.
Infrastructure IS code , but at least give us state-of-the-art code.

I have to disagree. Who's writing this infrastructure code? The full-stack developers who have the skills, experience, and talent to quickly pick up a monster of a language like Java while having limited experience, if any, in tuning, configuring, securing, monitoring and managing servers? Or system admins who have very limited experience with writing code, if any at all. I most certainly think (and hope) it's the latter of the two. Managing systems is most certainly the job of a sysadmin or a sysadmin turned DevOps. I, personally, was a dev, who went to ops, and is now somewhere in the middle. The language used in Puppet MUST be easy to read, learn, and use. The point of the DSL is to abstract all the overhead of those languages. Ruby is a pretty easy language to learn and can do pretty much everything you'd need it to already, then Puppet is one more layer on top which you can extend with plain Ruby if needed. Do you lose runtime speed? Sure, but it's worth it. Often the case is 1-2 in the ops area have solid coding skills, the rest of the team has limited or no coding skills. If Puppet wasn't relatively easy to learn there would only be 1 person able to do the work. I think Puppet needs to be even easier to learn than it is now to be honest.

This is my point, If you have the skills to write up a huge infrastructure in Java, had to or wanted Puppet DSL to be Java-based just write your own Puppet derivative. An infrastructure written in Java would be tens of thousands of lines long and likely impossible to read. If Luke took the advice, I think MOST wouldn't waste their time and simply go back to writing bash/powershell scripts to do our jobs. Puppet is definitely on the right path. The goal is to make it easy to adopt and use.

As Mike Lehner points out, the typical users of the Puppet language are more comfortable doing system administration and scripting than programming in a "computer science" type of language. In fact, the Puppet language is also used by less technical managers as documentation of infrastructure, and it is used to review changes. The Puppet language is a domain specific language - and it will continue to be a language that specializes in making configuration and management of systems easy. The language will evolve and over time include more and more features supporting what users of Puppet find time consuming or complicated to achieve. We are also evolving the language to be more strict / safe and we are borrowing concepts from "computer science" languages to do so. One example in Puppet 4.0 is the new type system, while not being equivalent to a functional language like Haskel, it is much more advanced than what you find in a typical OO language. The Puppet language is more declarative/functional in its style than imperative/object-oriented, and is more likely to evolve in the functional direction instead of becoming more imperative and operating on mutable objects.

I would also like to point out that Puppet really consists of two "languages" - the Puppet DSL for creating catalogs, and the "language" (or model if you like) of the catalog, which is what agents use to synchronize a machine to the desired state expressed in the catalog. IMO It is the catalog that is the true lingua franca of Puppet.

We have seen members of the community develop other ways of creating catalogs. There is even a Haskel implementation. While it makes it possible to describe the catalog in "state-of-the-art code" as you put it, it is also difficult to read even for most programmers (unless you are fortunate to be a Haskel programmer). Will there be libraries for Java, Scala, Clojure and other languages to construct catalogs in the future? Yes, not unlikely - although not a goal we are striving for per se, there are those that want to use Haskel, Scala, Java etc. as the language in which infrastructure is expressed in - or used as the language to transform descriptions in other systems into catalogs. As we continue to improve in the area of having published specifications, and well defined APis, providing interoperability with other languages will become much easier. While this is technically cool and everything, it also makes the infrastructure expressed as code less accessible to those that are non programmers - something worth remembering when fostering a DevOps culture.

With Puppet 4.0 we are taking a big step away from "messy/brittle/non-refactorable" code as we have a new more strict platform to innovate from. We are doing so in a largely backwards compatible way as there are thousands of available modules that we do not want to break. As modules gradually move over to 4.0 we will continue to make the language more robust.

The fact that there is no tool that currently can refactor Puppet code does not mean that it is not possible. We have the foundation for refactoring in the Puppet IDE Geppetto - as an example you can navigate references (show all references to an element / what is this a reference to), but functionality that lets you change things and automatically update all references has yet to be implemented.

Being a programmer, and a lower of computer languages, I find the community's general addiction to yaml hashes fascinating to watch - like moths to a flame.

A few observations from every puppet shop I worked at:

(1) You sink considerable hours into understanding the broken language, parser, get ridiculed by the community for lack of knowledge,
(2) the diagnostics, debugging, troubleshooting tools are junk; there is a diagnostic message that literally says find graph cycles by rendering a graphviz dot file and examining the visual representation; that is beyond absurd,
(3) the language is far more complicated and less documented than any high-level language.

You keep saying the Puppet DSL is easy for the non-programmer sysadmin types, but it is one of the most complicated and most unusable languages in CM today.

Hi
During one of the Devops meetings in London I was told that Puppet was going to be re-written in 'Python', In that it will match the other tools like Ansible and Salt. This was highlighted as a point to make it faster and benefit from the flexibility and availability of Python frameworks and libraries. Could you please confirm or correct this assumption.

Best Regards
Molix

A long complex evolution that seems like a lot of architecture astronauts are involved. I'm personally biased against Java/JVM so Clojure/JVM evokes an instinctual revulsion. Using Ansible myself, the simplicity of it has always had huge appeal and continues to do so.

Add new comment

The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.