Published on 15 October 2014 by

Yesterday, an OpenSSL vulnerability was announced. It is nicknamed POODLE (Padding Oracle On Downgraded Legacy Encryption, CVE-2014-3566) and was discovered by researchers at Google. Of course, SSL is a heavily utilized transport mechanism in many of our software components. This post provides our impact assessment and recommended actions.

The recommended fix from the OpenSSL.org team is to disable SSLv3 wherever possible — forcing components to communicate over newer versions of the TLS protocol. Our next series of releases (both commercial or open source) will disable SSLv3, but for immediate manual remediation, please see the steps below.

If you can’t change your settings to disable SSLv3, your exposure is still quite low. One of our in-house SSL experts, Adrien Thebo, dug into this and found exploitation of this vulnerability to be extremely difficult in a Puppet setup. Below is a quote from his findings.

"I just crunched the numbers for Puppet, and assuming that the agent checks in every 20 minutes, request payloads are evenly distributed in length, and the attacker performs the attack whenever possible, it'll take about 56 days to decrypt one byte of information at the end of a block.

"We've also been assuming that there's interesting content to steal from Puppet, and specifically the agent. Since we don't use basic auth or session cookies, there's not a lot of interesting things to steal. If someone implemented a custom fact that had sensitive data, that might be something that could be stolen, but again we're looking at 56 days on average to steal one byte of data, and we can only steal the last byte in a block."

MCollective shares the same protections and hurdles for this attack. The main difference with MCollective is that the timing can be modified by the client since it uses an exponential backoff for connection retries with a short initial wait, which means that it could be attacked more rapidly than Puppet. To make the attack viable you still have to have correct alignment, a full block of padding, and that's pure chance. If the attack succeeds you might be able to get one byte of sensitive data, but MCollective uses randomly generated passwords around 20 bytes. If you were trying to attack the password, you would be much better off running a brute force. Lastly, even if the password could be fully compromised — a big if — MCollective requires that clients have a certificate as well, so you still couldn’t connect. You’d need to be in control of a certificate to authenticate.

Manual remediation

The risk of a successful attack is very low. However, if you’d like to modify your settings on your Puppet installation, we’ll walk you through disabling SSLv3 on the components in our ecosystem.

Puppet master and Puppet Enterprise console

To disable SSLv3 in a Puppet Enterprise installation, add -SSLv3 to the SSLProtocol section of your httpd settings. If your console and Puppet master are on different systems, you’ll need to do this on each of them. After making the change, be sure to restart the web server.

As a script you could run something like:


# Apache vhosts for puppetserver and enterprise console
# SSLProtocol             ALL -SSLv2 -SSLv3
for file in /etc/puppetlabs/httpd/conf.d/puppet*.conf ; do
   sed -i 's/-SSLv2$/-SSLv2 -SSLv3/g' "$file"
done

PuppetDB

For PuppetDB, you’ll need to add a ssl-protocols directive and just explicitly allow TLSv1, TLSv1.1 and TLSv1.2. The command should help point you in the right direction. Remember to restart PuppetDB after making this change.


# Puppetdb
# ssl-protocols = TLSv1, TLSv1.1, TLSv1.2
if ! grep ssl-protocols /etc/puppetlabs/puppetdb/conf.d/jetty.ini ; then
   sed -i '/ssl-port =/ a\\n#SSL protocols to use\nssl-protocols = TLSv1, TLSv1.1, TLSv1.2' /etc/puppetlabs/puppetdb/conf.d/jetty.ini
fi

MCollective

Disabling SSLv3 for MCollective is bit tricker, because it’s done with ActiveMQ. In a Puppet Enterprise installation, the ActiveMQ configuration files are managed by Puppet. This means the module needs to be modified. After modification, be sure your Puppet agent runs to apply the changes.

If you don’t want to wait for fixes (and a complete security rollup release), you can apply this change to the module on your Puppet Enterprise master yourself. However, due to a bug within Apache ActiveMQ, if a misconfiguration is made for the protocol choice, it will fall back to SSLv3 only.


# ActiveMQ/Mcollective
# Note: This is adding an explicit set of enabled protocols with these entries
# After this is applied, activemq.xml should contain:
# 
if [ -f /opt/puppet/share/puppet/modules/pe_mcollective/templates/activemq.xml.erb ] ; then
   sed -i -e 's|uri="://0.0.0.0:61616"/|uri="://0.0.0.0:61616?transport.enabledProtocols=TLSv1,TLSv1.1,TLSv1.2"/|g' -e 's|uri="://0.0.0.0:"/|uri="://0.0.0.0:?transport.enabledProtocols=TLSv1,TLSv1.1,TLSv1.2"/|g' /opt/puppet/share/puppet/modules/pe_mcollective/templates/activemq.xml.erb
fi

Mike Stahnke is director of engineering services at Puppet Labs.

Share via:
Tagged:

The security is grumbly that port 443 on the puppet master says "SSL3_GET_RECORD:wrong version number" instead of "sslv3 alert handshake failure" when he runs his scan. What should I tell him about puppet's use of 443 that will make him grumble at someone else?

The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.