A few weeks ago, the Puppet Labs Security team received emails from two community members informing us the our website was vulnerable. Two of the attacks mentioned were CRIME and BEAST attacks. Up until now, most of the mails that have been addressed to this team have been to report security issues with one of our products, but rarely about our infrastructure.
When we received the message, it was an immediate reminder of how little attention had been paid to the parts of infrastructure that didn't immediately require our attention. Sheepish facepalms abounded.
Puppet Labs strives for transparency as much as we can, and we like to keep our users and community in the loop when a bug is reported. For this particular case, it was left to the Operations team to validate and respond to the issue. The Operations team began immediately trying to first understand what the heck CRIME and BEAST were and then validate that there was indeed a problem.
Within a few minutes the Internet had told us that these attacks could quickly be mitigated on our Debian infrastructure by a config change to turn off compression on TLS sessions, and to upgrade Apache to the most recent version available in Debian. Now, a configuration change seemed simple enough. We only had 30 divergent Apache vHost ERB templates lying around for Puppet to compile and deploy. It was going to be a minute to fix, but we needed to get back to the user. The claims were quickly validated by a free SSL scanner put out by Quallys. Yes, Portland, we have a problem.
Now for the mitigation. We began to dive into some of the oldest Puppet code in our infrastructure -- Apache deployments -- the oldest of which was written nearly four years ago and by some guy named 'root', whoever that is.
As our users know, module development has been something of a craft that has evolved in recent years, since our Apache module has carried some of the legacy methods for module authorship. (Yes, even at Puppet Labs, we have to update how we write modules. We haven't always known the best way.) Our Apache module used a template for each vhost. Even though we had the 'default' vhost, almost no site we had deployed even used it. This meant that in order to correct the configuration issue and disable compression for TLS in Apache, we were going to have to hunt down every place in our templates that we had configured SSL and modify the behavior. Luckily, we were able to get the list of Apache servers that were not up to date with one simple MCollective query. And this exercise gave us the opportunity to dig into the way we deployed our Apache configurations! Instead of fixing every instance of our SSL configuration, we fixed the central SSL configuration and simply included the template fragment in all of the other templates.
<%= scope.function_template(['apache/vhost/_ssl.conf.erb']) %>
This turned out to be awesome, since now we can tune all of our Apache SSL configurations at the same time. We also found a reason to finally get rid of that last remaining Ubuntu box since it did not support the version of Apache without doing some backporting. We chose to move the application to a Debian system to be in line with the rest of our infrastructure. We then combined this with an upgrade to our affected Apache instances and merged the new code for our templates. We’re more than a bit chuffed that we not only resolved the issues but resolved them using Puppet. It’s the best thing in the world when “eating your own dog food” actually means something tangible about managing our business.
But more importantly we’re thrilled that when a security issue arose two members of the community chose to let us know about it. We’d like to thank: Adam Ziaja and Chiragh Dewan for their responsible disclosure of the vulnerabilities. We’d also would like to thank our community for its history, and the continued practice, of disclosing security issues responsibly.
How You Can Help
We take these issue seriously and are committed to the security of our products and its surrounding infrastructure. If you find a security issue with any of our products or services, please let our security team know by sending an email to [email protected].