homebloghow heartbleed fixed my puppet master

How Heartbleed Fixed My Puppet Master

Temboo is a cloud-based programming platform for hardware and software developers. We provide a single consistent interface to connect with databases, code utilities, and over 100 APIs for all kinds of popular internet services, such as Facebook, Google and Dropbox. Similarly to how Puppet can manage resources in different ways as required by different operating systems, developers using Temboo can access different internet-based resources through a single unified programming interface. As a service provider at the hub of the Internet of Things, Temboo needs to respond rapidly and continuously to the myriad miniscule changes demanded by morphing APIs, all while maintaining compatibility with the memory-constrained microcontrollers, connected devices, and various applications that rely on Temboo.

We have been using Puppet Enterprise at Temboo for about a year now, and it has significantly improved our operational architecture, providing a stellar mechanism for pushing those changes out to our customers with alacrity. Prior to Puppet, we were managing our infrastructure with RightScale, and in the RightScale model, configuration is process. Effecting a change in that model requires defining a process to both update running nodes and act as part of a template for newly launched nodes. This practice was time-consuming and error-prone. Because Puppet defines configuration as state rather than process, we can more easily and confidently push out changes incrementally, and with greater precision — an important capability when dealing with the rapidly moving target that is the Internet of Things.

We found adoption and usage of Puppet Enterprise smooth and relatively painless, though we did not get every detail quite right on the first try. One issue in particular caused frequent aggravation. When we originally set up our Puppet master (yes, we use a master — exported resources are miraculous!) the certificate naming scheme we would employ for our agents was not yet apparent. Also not yet apparent to us as new Puppeteers was the fact that the Puppet master would eventually act as an agent of itself. So, with no immediate reason to depart from documented example, we named our Puppet master `puppet.temboo.com`.

As it turns out, we eventually converged on a different naming scheme for our agent certificates: The certificates would use the local hostname of the node only, rather than the fully-qualified domain name. Unfortunately, this left us with a Puppet master unable to share a Puppet codebase with all other agents, because that code included plenty of references to the value of $clientcert, with the assumption that the certificate name did not include “.temboo.com” on the end.

The problem became apparent to us fairly quickly, as we found ourselves writing special-case code for the Puppet master node all too often. Meanwhile, the time constraints of our rapid development schedule prevented us from setting aside the time required to fix the certificate naming issue in a more permanent and complete fashion.

Heartbleed changed all that.

I Learn to Stop Worrying and Love Heartbleed

Puppet uses SSL certificates to manage connections to and from the Puppet master, and Heartbleed exposed the possibility that those certificates were compromised. We received an email from Puppet Labs shortly after Heartbleed became public knowledge, recommending that we cycle the certificates — a nice touch from Puppet Labs. Because Puppet Enterprise creates and manages its own certificates and Certificate Authority so silently and seamlessly, Puppet interactions would not necessarily have been on our radar as potential victims of Heartbleed. That sort of proactive communication is greatly appreciated.

All of our Puppet agent processes are firewalled so that only our own Puppet master can reach them, so we knew it was unlikely that the Heartbleed vulnerability had affected our Puppet installation. Nevertheless, we thought it prudent to immediately rekey and re-issue all Puppet certificates, to avoid the slightest risk of corruption.

But wait: As long as we had to re-issue the master certificate, might we not also rename it in the process, dispensing with that thorny exception that had been nagging at us for months? In short, could we leverage the catastrophe of Heartbleed to walk away with a Puppet Enterprise installation that is better than ever?

I submitted a request to Puppet Labs support for instructions on renaming the master certificate during the certificate re-issue process. In short order, I heard back from Jay Wallace with those instructions. I ran into some bumps along the way, and after a few more emails back and forth, Jay reached out to me by telephone, though that level of support is not included in our contract. Soon after that, support lead Celia Cottle joined the call as well. To make a long story short, we wound up spending a couple of hours working through the problem together, in typical, grueling fashion. Jay and Celia were clearly interested in understanding the issues presented by my case, so that others could ultimately benefit from our experience at Temboo. Meanwhile, we wound up with a fully recertified and renamed Puppet master. Now our Puppet master can finally use all the same code that every other node uses. The benefit: We save both time and precious cognitive space, because we don’t have to write every manifest twice.

External Node Classifiers

After renaming our Puppet master, we noticed that it appeared to be lacking a knowledge of its own role as master, MCollective hub and PuppetDB host. Repairing that situation gave me a more solid understanding of the Puppet Enterprise console’s role as an external node classifier (ENC).

When I first read about ENCs, they were described simply as “a program that takes a certificate name and returns a YAML data structure representing parameters for that node.” That sounded easy to me, so we whipped up our own in 15 minutes, written in Python. We automate almost everything, so we rarely even looked at the console, and it never occurred to us that the console as ENC had, at installation time, already applied its own node configuration to the Puppet master. All it took was switching puppet.conf to point at the console as ENC for a single Puppet agent run to fully bring the Puppet master back in line with all its duties. A powerful hack, and one we may well use again for any number of future needs.

The process of renaming the Puppet master also demonstrated how Puppet Enterprise uses classes to establish the Puppet master as a hub for MCollective orchestration, and thus how orchestration capabilities could be extended to other nodes with relative ease. Such a mechanism suggests interesting possibilities — for example, a hierarchal or even web-shaped orchestration architecture.

Puppet Labs Support Is Over the Top

Puppet Labs support went far and above expectations in assisting me to rectify and improve our Puppet infrastructure. Jay and Celia spent hours with me, saving Temboo days of work piecing together the solution ourselves, using available documentation. Because Puppet Labs uses its support team not only to help customers but also as a feedback mechanism for improvement of its own product, the level of support I receive is more than worth the cost. That, combined with the extensive automation and integration provided by Puppet Enterprise, has made Puppet Labs’ product a great choice for Temboo.

There is a lot going on under the hood with Puppet Enterprise, and for us, that translates into the ability to provide a better product to more of our customers, without straining our internal resources.

Jeffrey Froman has been doing DevOps since before anyone thought to call it that. He is currently Director of DevOps at Temboo, Inc.

Learn More