Regenerating all certificates in a Puppet deployment

In some cases, you might need to regenerate the certificates and security credentials (private and public keys) that are generated by Puppet’s built-in certificate authority (CA).

For example, you might have a Puppet master you need to move to a different network in your infrastructure, or you might have experienced a security vulnerability that makes existing credentials untrustworthy.

Note: If you’re visiting this page to remediate your Puppet Enterprise deployment due to CVE-2014-0160, also known as Heartbleed, see this announcement for additional information and links to more resources. Before applying these instructions, please note that this is a non-trivial operation that contains some manual steps and will require you to replace certificates on every agent node managed by your Puppet master.
Important: The information on this page describes the steps for regenerating certs in an open source Puppet deployment. If you use Puppet Enterprise do not use the information on this page, as it will leave you with an incomplete replacement and non-functional deployment. Instead, PE customers must refer to one of the following pages:
Regardless of your situation, regenerating your certs involves the following three steps, described in detail in the sections below:
  1. On your master, you’ll clear the certs and security credentials, regenerate the CA, and then regenerate the certs and security credentials.

  2. You’ll clear and regenerate certs and security credentials for any extensions.

  3. You'll clear and regenerate certs and security credentials for all agent nodes.

CAUTION: This process destroys the certificate authority and all other certificates. It is meant for use in the event of a total compromise of your site, or some other unusual circumstance. If you just need to replace a few agent certificates, use the puppet cert clean command on your Puppet master and then follow step 3 for any agents that need to be replaced.

Step 1: Clear and regenerate certs on your Puppet master

On the Puppet master hosting the CA:

  1. Back up the SSL directory, which is in /etc/puppetlabs/puppet/ssl/. If something goes wrong, you can restore this directory so your deployment can stay functional. However, if you needed to regenerate your certs for security reasons and couldn’t, get some assistance as soon as possible so you can keep your site secure.
  2. Stop the agent service:
    sudo puppet resource service puppet ensure=stopped
    
  3. Stop the master service.
    For Puppet Server, run:
    sudo puppet resource service puppetserver ensure=stopped
  4. Delete the SSL directory:
    sudo rm -r /etc/puppetlabs/puppet/ssl
  5. Regenerate the CA and master's cert:
    sudo puppetserver ca setup

    You will see this message: Notice: Signed certificate request for ca.

  6. Generate the Puppet master’s new certs:
    sudo puppet master --no-daemonize --verbose
    
  7. When you see the message Notice: Starting Puppet master <VERSION>, type CTRL + C.
  8. Start the Puppet master service by running:
    sudo puppet resource service puppetserver ensure=running
  9. Start the Puppet agent service by running this command:
    sudo puppet resource service puppet ensure=running
    
Results
At this point:
  • You have a new CA certificate and key.

  • Your Puppet master has a certificate from the new CA, and it can field new certificate requests.

  • The Puppet master rejects any requests for configuration catalogs from nodes that haven’t replaced their certificates. At this point, it is all of them except itself.

  • When using any extensions that rely on Puppet certificates, like PuppetDB, the Puppet master won’t be able to communicate with them. Consequently, it might not be able to serve catalogs, even to agents that do have new certificates.

Step 2: Clear and regenerate certs for any extension

You might be using an extension, like PuppetDB or MCollective, to enhance Puppet. These extensions probably use certificates from Puppet’s CA in order to communicate securely with the Puppet master. For each extension like this, you’ll need to regenerate the certificates it uses.

Many tools have scripts or documentation to help you set up SSL, and you can often just re-run the setup instructions.

PuppetDB

We recommend PuppetDB users first follow the instructions in Step 3: Clear and regenerate certs for agents, below, because PuppetDB re-uses Puppet agents’ certificates. After that, restart the PuppetDB service. See Redo SSL setup after changing certificates for more information.

MCollective

MCollective often uses SSL certificates from Puppet’s CA. If you are replacing your Puppet CA and are using the same certs for MCollective, refer to the standard deployment guide and re-do any steps involving security credentials. You’ll generally need to replace client certificates, your server keypair, and the ActiveMQ server’s keystore and truststore.

Important: As of Puppet agent 5.5.4, MCollective is deprecated and will be removed in a future version of Puppet agent. If you use Puppet Enterprise, consider migrating from MCollective to Puppet orchestrator. If you use open source Puppet, migrate MCollective agents and filters using tools like Bolt and PuppetDB’s  Puppet Query Language.

Step 3: Clear and regenerate certs for Puppet agents

To replace the certs on agents, you’ll need to log into each agent node and do the following steps.

  1. Stop the agent service. On *nix:
    sudo puppet resource service puppet ensure=stopped
    On Windows, with Administrator privileges: 
    puppet resource service puppet ensure=stopped
  2. Locate Puppet’s SSL directory and delete its contents.

    The SSL directory can be determined by running puppet config print ssldir --section agent

  3. Restart the agent service. On *nix:
    sudo puppet resource service puppet ensure=running
    On Windows, with Administrator privileges:
    puppet resource service puppet ensure=running

    When the agent starts, it generates keys and requests a new certificate from the CA master.

  4. If you are not using autosigning, log in to the CA master server and sign each agent node’s certificate request.
    To view pending requests, run:
    sudo puppetserver ca list
    To sign requests, run:
    sudo puppetserver ca sign --certname <NAME>

    After an agent node’s new certificate is signed, it's retrieved within a few minutes and a Puppet run starts.

Results

After you have regenerated all agents’ certificates, everything will be fully functional under the new CA.

Note: You can achieve the same results by turning these steps into Bolt tasks or plans. See the Bolt documentation for more information.