Puppet Enterprise 2018.1

PE components use SSL certificates to communicate securely, and PE provides its own certificate authority (CA) to generate and verify credentials. You can add an external certificate authority to this configuration to meet the security requirements of your environment.

There are two options for configuring an external CA:
  • Completely replace the PE CA with your own CA. Your CA must be a root CA.

  • Use the PE CA as an intermediate of your own root CA.

In both of these scenarios, you must first install PE, and then replace the certificate files.

To use an external certificate authority with high availabiltiy, first install PE, then replace the certificate files, and finally configure HA as needed, depending on whether you're upgrading or setting up a new HA replica.

Replace the PE CA

The following tasks detail the procedures for replacing the PE CA with your own.

Prepare the certificates and security credentials

First, prepare the certificates and security credentials to use with your external CA.

Before you begin
You must have:
  • The complete X.509 Certificate Authority certificate chain for the external party CA, in PEM format

  • An X.509 Certificate, signed by the external party CA, in PEM format, with matching private and public keys, for each PE service which uses a certificate

  • An X.509 Certificate, signed by the external party CA, in PEM format, with matching private and public keys, for each Puppet agent node

  1. On your master, inspect the inventory of certificates signed using PE’s built-in CA. You need a cert, private key, and public key for each. Run puppet cert list --all.
    You see results for the following:
    • Per-node certificates for the Puppet master (and any agent nodes)
    • pe-internal-puppet-console-mcollective-client
    • pe-internal-mcollective-servers
    • pe-internal-peadmin-mcollective-client
  2. Each of these need to be replaced with new certificates signed by your external CA. The tasks below will explain how to find and replace these credentials. For each certificate you create, you'll need the following information.
    1. For the X.509 extension, Extended Key Usage, add serverAuth, clientAuth.
    2. Comment out the entry, nsCertType=server.
    3. Add any Puppet master DNS alt names to the cert. PE always uses the default alt name puppet.
    Note: pe-orchestration-services and pe-console-services use the agent certificate for the node they run on.
    Tip: For ease of use, we recommend naming ALL of your certificate and security credential files exactly the same way they are named by PE and replace them as such on the master; for example, use the cp command to overwrite the file contents of the certs generated by the PE CA. This will ensure that PE will recognize the file names and not overwrite any files when you perform Puppet runs. In addition, this will prevent you from needing to touch various config files, and thus, limit the chances of problems arising.

    The remainder of this doc assumes you'll use identical files names.

Replace the Puppet master certificates and security credentials

After you create the necessary certificates and security credentials with your external CA, you're ready to replace the Puppet master and console certificates and security credentials.

  1. On the Puppet master, clear the cached catalog: rm -f /opt/puppetlabs/puppet/cache/client_data/catalog/<PUPPET MASTER FQDN>.json
  2. In the console, click Classification, and in the PE Infrastructure group, select the PE Certificate Authority group.
  3. On the Rules tab, in the row for the Puppet master, click Unpin, and commit changes.
  4. Click Classification, and in the PE Infrastructure group, select the PE Master group.
  5. On the Configuration tab, in the puppet_enterprise::profile::master class, set the enable_ca_proxy parameter value to false.
  6. Click Add parameter, and commit changes.
  7. On the Puppet master, run Puppet.
  8. Stop all PE services.
       puppet resource service puppet ensure=stopped
       puppet resource service pe-puppetserver ensure=stopped
       puppet resource service pe-activemq ensure=stopped
       puppet resource service mcollective ensure=stopped
       puppet resource service pe-puppetdb ensure=stopped
       puppet resource service pe-postgresql ensure=stopped
       puppet resource service pe-console-services ensure=stopped
       puppet resource service pe-nginx ensure=stopped
       puppet resource service pe-orchestration-services ensure=stopped
    
  9. Copy your new certs and security credentials for the Puppet master to the relevant locations.
    External CA Generated File Location on Host Name of File
    CA's certificate/etc/puppetlabs/puppet/ssl/certsca.pem
    CA's CRL/etc/puppetlabs/puppet/sslcrl.pem
    CA's CRL/etc/puppetlabs/puppet/ssl/caca_crl.pem
    Host's Cert/etc/puppetlabs/puppet/ssl/certs<PUPPET MASTER FQDN>.pem
    Host's Public Keyetc/puppetlabs/puppet/ssl/public_keys<PUPPET MASTER FQDN>.pem
    Host's Private Keyetc/puppetlabs/puppet/ssl/private_keys<PUPPET MASTER FQDN>.pem
    cp <PATH TO NEW PUPPET MASTER FQDN>.cert /etc/puppetlabs/puppet/ssl/certs/<PUPPET MASTER FQDN>.pem
    cp <PATH TO NEW PUPPET MASTER FQDN>.private_key /etc/puppetlabs/puppet/ssl/private_keys/<PUPPET MASTER FQDN>.pem
    cp <PATH TO NEW PUPPET MASTER FQDN>.public_key /etc/puppetlabs/puppet/ssl/public_keys/<PUPPET MASTER FQDN>.pem
    cp <PATH TO NEW CA CRL>.pem /etc/puppetlabs/puppet/ssl/crl.pem
    cp <PATH TO NEW CA CRL>.pem /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem
    cp <PATH TO NEW CA CRT>.pem /etc/puppetlabs/puppet/ssl/certs/ca.pem

Replace the pe-orchestration-services certificates and security credentials

After you replace the Puppet master certificates and security credentials, you're ready to replace the pe-orchestration-services certificates and security credentials.

  1. In /etc/puppetlabs/orchestration-services/ssl/ add the new certs and security credentials for the Puppet master, and create the .pk8 cert.
    
    cp /etc/puppetlabs/puppet/ssl/certs/<PUPPET MASTER FQDN>.pem /etc/puppetlabs/orchestration-services/ssl/<PUPPET MASTER FQDN>.cert.pem
    cp /etc/puppetlabs/puppet/ssl/public_keys/<PUPPET MASTER FQDN>.pem /etc/puppetlabs/orchestration-services/ssl/<PUPPET MASTER FQDN>.public_key.pem
    cp /etc/puppetlabs/puppet/ssl/private_keys/<PUPPET MASTER FQDN>.pem /etc/puppetlabs/orchestration-services/ssl/<PUPPET MASTER FQDN>.private_key.pem
    openssl pkcs8 -topk8 -inform PEM -outform DER -in /etc/puppetlabs/orchestration-services/ssl/<PUPPET MASTER FQDN>.private_key.pem -out /etc/puppetlabs/orchestration-services/ssl/<PUPPET MASTER FQDN>.private_key.pk8 -nocrypt
    chown -R pe-orchestration-services:pe-orchestration-services /etc/puppetlabs/orchestration-services/ssl/
    			
  2. Ensure pe-orchestration-services can access the new credentials
    chown -R pe-orchestration-services:pe-orchestration-services /etc/puppetlabs/orchestration-services/ssl/

Replace the console and console-services certificate and security credentials

After you replace the pe-orchestration-services certificates and security credentials, you're ready to replace the console and console-services certificate and security credentials.

  1. Replace the console's certs and security credentials.
    Note: For a split install these are the Puppet agent certs, found in the following locations on the console server:
    • /etc/puppetlabs/puppet/ssl/certs/<AGENT FQDN>.pem
    • /etc/puppetlabs/puppet/ssl/private_keys/<AGENT FQDN>.pem
    • /etc/puppetlabs/puppet/ssl/public_keys/<AGENT FQDN>.pem
    External CA Generated File Location on HostName of FileNotes
    PE console cert /etc/puppetlabs/puppetdb/ssl /etc/puppetlabs/puppet/ssl/<PUPPET MASTER CERT>.pem On a monolithic install, copy the Puppet master’s agent cert to this location; on split install, copy the Puppet agent’s cert to this location
    PE console private key /etc/puppetlabs/puppet/ssl /etc/puppetlabs/puppet/ssl/<PUPPET MASTER PRIVATE KEY>.pem On a monolithic install, copy the Puppet master’s agent cert to this location; on split install, copy the Puppet agent’s cert to this location
    PE console public key /etc/puppetlabs/puppet/ssl /etc/puppetlabs/puppet/ssl/<PUPPET MASTER PUBLIC KEY>.pem On a monolithic install, copy the Puppet master’s agent cert to this location; on split install, copy the Puppet agent’s cert to this location
  2. In /opt/puppetlabs/server/data/console-services/certs/, add the new certs and security credentials for the Puppet master, and create the .pk8 cert.
    cp /etc/puppetlabs/puppet/ssl/certs/<PUPPET MASTER FQDN>.pem /opt/puppetlabs/server/data/console-services/certs/<PUPPET MASTER FQDN>.cert.pem
    cp /etc/puppetlabs/puppet/ssl/private_keys/<PUPPET MASTER FQDN>.pem /opt/puppetlabs/server/data/console-services/certs/<PUPPET MASTER FQDN>.private_key.pem
    cp /etc/puppetlabs/puppet/ssl/public_keys/<PUPPET MASTER FQDN>.pem /opt/puppetlabs/server/data/console-services/certs/<PUPPET MASTER FQDN>.public_key.pem
    openssl pkcs8 -topk8 -inform PEM -outform DER -in /opt/puppetlabs/server/data/console-services/certs/<PUPPET MASTER FQDN>.private_key.pem -out /opt/puppetlabs/server/data/console-services/certs/<PUPPET MASTER FQDN>.private_key.pk8 -nocrypt
    chown -R pe-console-services:pe-console-services /opt/puppetlabs/server/data/console-services/certs/
    			  
  3. Ensure the console can access the new credentials.
    chown -R pe-console-services:pe-console-services /opt/puppetlabs/server/data/console-services/certs

Replace the Puppet Master certificate and security credentials in PostgreSQL

You need to replace the Puppet Master certificate and security credentials in PostgreSQL.

  1. Add the Puppet master's certificate and security credentials to the PostgreSQL certs directory, located at /opt/puppetlabs/server/data/postgresql/9.6/data/certs/.
    cp /etc/puppetlabs/puppet/ssl/certs/<PUPPET MASTER FQDN>.pem /opt/puppetlabs/server/data/postgresql/9.6/data/certs/_local.cert.pem
    cp /etc/puppetlabs/puppet/ssl/private_keys/<PUPPET MASTER FQDN>.pem /opt/puppetlabs/server/data/postgresql/9.6/data/certs/_local.private_key.pem
    chmod 600 /opt/puppetlabs/server/data/postgresql/9.6/data/certs/*
  2. Ensure PostgreSQL can access the new credentials.
    chown pe-postgres:pe-postgres /opt/puppetlabs/server/data/postgresql/9.6/data/certs/*		 

Replace the PuppetDB certificates and security credentials

After you replace the Puppet master certificate and security credentials in PostgreSQL, you replace the PuppetDB certificates and security credentials.

  1. Replace PuppetDB's certs and security credentials, and create the .pk8 cert.
    Note: For a split install these are the agent certs, found in the same locations as the Puppet master certs.
    • /etc/puppetlabs/puppet/ssl/certs/<AGENT FQDN>.pem
    • /etc/puppetlabs/puppet/ssl/private_keys/<AGENT FQDN>.pem
    • /etc/puppetlabs/puppet/ssl/public_keys/<AGENT FQDN>.pem
    External CA Generated FileLocation on HostName of FileNotes
    PuppetDB cert /etc/puppetlabs/puppetdb/ssl /etc/puppetlabs/puppetdb/ssl/<PUPPET MASTER CERT>.pem On a monolithic install, copy the Puppet master’s agent cert to this location; on split install, copy the Puppet agent’s cert to this location
    PuppetDB private key /etc/puppetlabs/puppet/ssl /etc/puppetlabs/puppetdb/ssl/<PUPPET MASTER PRIVATE KEY>.pem On a monolithic install, copy the Puppet master’s agent cert to this location; on split install, copy the Puppet agent’s cert to this location
    PuppetDB public key /etc/puppetlabs/puppet/ssl /etc/puppetlabs/puppetdb/ssl/<PUPPET MASTER PUBLIC KEY>.pem On a monolithic install, copy the Puppet master’s agent cert to this location; on split install, copy the Puppet agent’s cert to this location
    cp /etc/puppetlabs/puppet/ssl/certs/<PUPPET MASTER FQDN>.pem /etc/puppetlabs/puppetdb/ssl/<PUPPET MASTER FQDN>.cert.pem
    cp /etc/puppetlabs/puppet/ssl/private_keys/<PUPPET MASTER FQDN>.pem /etc/puppetlabs/puppetdb/ssl/<PUPPET MASTER FQDN>.private_key.pem
    cp /etc/puppetlabs/puppet/ssl/public_keys/<PUPPET MASTER FQDN>.pem /etc/puppetlabs/puppetdb/ssl/<PUPPET MASTER FQDN>.public_key.pem
    openssl pkcs8 -topk8 -inform PEM -outform DER -in /etc/puppetlabs/puppetdb/ssl/<PUPPET MASTER FQDN>.private_key.pem -out /etc/puppetlabs/puppetdb/ssl/<PUPPET MASTER FQDN>.private_key.pk8 -nocrypt
    
  2. Ensure PuppetDB can access the new credentials: chown -R pe-puppetdb:pe-puppetdb /etc/puppetlabs/puppetdb/ssl

Replace the MCollective certificates and security credentials

The last set of certificates and security credentials you need to replace are for MCollective.

  1. Generate new credentials for each MCollective cert, then replace the cert, private key, and public key for each of them.
    External CA Generated FileLocation on HostName of File
    MCollective internal servers cert/etc/puppetlabs/puppet/ssl/certspe-internal-mcollective-servers.pem
    MCollective internal servers priv key/etc/puppetlabs/puppet/ssl_private_keyspe-internal-mcollective-servers.pem
    MCollective internal servers pub key/etc/puppetlabs/puppet/ssl_public_keyspe-internal-mcollective-servers.pem
    MCollective client cert/etc/puppetlabs/puppet/ssl/certspe-internal-peadmin-mcollective-client.pem
    MCollective client priv key/etc/puppetlabs/puppet/ssl_private_keyspe-internal-peadmin-mcollective-client.pem
    MCollective client pub key/etc/puppetlabs/puppet/ssl_public_keyspe-internal-peadmin-mcollective-client.pem
    MCollective internal console client cert/etc/puppetlabs/puppet/ssl/certspe-internal-puppet-console-mcollective-client.pem
    MCollective internal console client priv key/etc/puppetlabs/puppet/ssl_private_keyspe-internal-puppet-console-mcollective-client.pem
    MCollective internal console client pub key/etc/puppetlabs/puppet/ssl_public_keyspe-internal-puppet-console-mcollective-client.pem
    cp <PATH TO NEW PE-INTERNAL-MCOLLECTIVE-SERVERS>.cert /etc/puppetlabs/puppet/ssl/certs/pe-internal-mcollective-servers.pem
    cp <PATH TO NEW PE-INTERNAL-MCOLLECTIVE-SERVERS>.private_key /etc/puppetlabs/puppet/ssl/private_keys/pe-internal-mcollective-servers.pem
    cp <PATH TO NEW PE-INTERNAL-MCOLLECTIVE-SERVERS>.public_key /etc/puppetlabs/puppet/ssl/public_keys/pe-internal-mcollective-servers.pem
    cp <PATH TO NEW PE-INTERNAL-MCOLLECTIVE-CLIENT>.cert /etc/puppetlabs/puppet/ssl/certs/pe-internal-peadmin-mcollective-client.pem
    cp <PATH TO NEW PE-INTERNAL-MCOLLECTIVE-CLIENT>.private_key /etc/puppetlabs/puppet/ssl/private_keys/pe-internal-peadmin-mcollective-client.pem
    cp <PATH TO NEW PE-INTERNAL-MCOLLECTIVE-CLIENT>.public_key /etc/puppetlabs/puppet/ssl/public_keys/pe-internal-peadmin-mcollective-client.pem
    cp <PATH TO NEW PE-INTERNAL-PUPPET-CONSOLE-MCOLLECTIVE-CLIENT>.cert /etc/puppetlabs/puppet/ssl/certs/pe-internal-puppet-console-mcollective-client.pem
    cp <PATH TO NEW PE-INTERNAL-PUPPET-CONSOLE-MCOLLECTIVE-CLIENT>.private_key /etc/puppetlabs/puppet/ssl/private_keys/pe-internal-puppet-console-mcollective-client.pem
    cp <PATH TO NEW PE-INTERNAL-PUPPET-CONSOLE-MCOLLECTIVE-CLIENT>.public_key /etc/puppetlabs/puppet/ssl/public_keys/pe-internal-puppet-console-mcollective-client.pem
    
  2. On the Puppet master, navigate to /etc/puppetlabs/activemq/ and remove the following two files
    • broker.ts
    • broker.ks

Edit puppet.conf, restart services, and run Puppet

Once all the certificates and security credentials are replaced, you need to edit puppet.conf, restart services, and run Puppet.

  1. On the Puppet master, navigate to /etc/puppetlabs/puppet/ and open puppet.conf.
  2. In the [agent] section, add certificate_revocation=false.
  3. Restart or start the various services that help control Puppet and then run Puppet so that the certs and security credentials can be properly distributed.
    puppet resource service pe-postgresql ensure=running
    puppet resource service pe-puppetserver ensure=running
    puppet resource service pe-activemq ensure=running
    puppet resource service mcollective ensure=running
    puppet resource service pe-puppetdb ensure=running
    puppet resource service pe-console-services ensure=running
    puppet resource service pe-nginx ensure=running
    puppet resource service pe-orchestration-services ensure=running
    puppet resource service puppet ensure=running
    
  4. From the Puppet master node, use the command line to kick off a Puppet run: puppet agent -t.
    During this run, Puppet will copy the credentials you replaced into their final locations, regenerate the ActiveMQ truststore and keystore, and restart the pe-activemq and mcollective services.

    To ensure the certs have been properly distributed, got so the Inventory page in the console and click the Puppet master node in the list. Then, on the Reports tab, review the information for the node.

Add agent nodes using your external CA

After you replace the PE CA with your external CA, you need to replace the agent certificates and security credentials on your managed nodes.

To determine the proper locations for the certificate and security credential files used by the agent, use the following commands:

  • Certificate: puppet agent --configprint hostcert
  • Private key: puppet agent --configprint hostprivkey
  • Public key: puppet agent --configprint hostpubkey
  • Certificate Revocation List: puppet agent --configprint hostcrl
  • Local copy of the CA's certificate: puppet agent --configprint localcacert
  1. Install Puppet Enterprise on the node, if it isn't already installed.
  2. Using the same external CA you used for the Puppet master, create a cert and private key for your agent node.
  3. Using the commands above, locate the files you will need to replace on the agent.
  4. Copy the agent's certificate, private key, and public key into place. Do the same with the external CA's CRL and CA certificate.
  5. On your agent, restart the puppet service.

Your node should now be able to do agent runs, and its reports will appear in the console. (You can accelerate this by letting Puppet run once, waiting a few minutes for the node to be added to the MCollective group in the console, and then running puppet agent -t.)

Use the PE CA as an intermediate CA

PE can operate as an intermediate CA to an external root CA. (It can't be an intermediate to an intermediate.) In this configuration, the PE CA is left enabled and it can automatically accept CSRs, distribute certificates to agents, and use the standard puppet cert command to sign certificates.

Before you begin
  1. You must have an intermediate CA certificate from your external root CA.

  2. Back up your infrastructure.

  3. Stop all Puppet services.
    puppet resource service puppet ensure=stopped
    puppet resource service pe-puppetserver ensure=stopped
    puppet resource service pe-activemq ensure=stopped
    puppet resource service mcollective ensure=stopped
    puppet resource service pe-puppetdb ensure=stopped
    puppet resource service pe-postgresql ensure=stopped
    puppet resource service pe-console-services ensure=stopped
    puppet resource service pe-nginx ensure=stopped
    puppet resource service pe-orchestration-services ensure=stopped
    puppet resource service pxp-agent ensure=stopped
    
  4. On the master, remove existing .pem files: find /etc/puppetlabs/puppet/ssl -name '*.pem' -delete

There are some limitations to this configuration:
  • Agent-side CRL checking is not possible, although Puppet Server still verifies the CRL.

  • The CA certificate bundle (the external root CA combined with the intermediate CA certificate) must be distributed to the agents manually, ideally before Puppet runs.

Warning: This procedure destroys all certificates currently in use in your infrastructure and prevents normal operation until certificates are restored. Time performing this task accordingly.

Configure PE to use the new certificates

Place and configure certificates on your Puppet Server.

Perform these steps on your Puppet Server.

  1. Put these files in the locations indicated: 
    FileLocationNotes
    intermediate CA certificate/etc/puppetlabs/puppet/ssl/ca/ca_crt.pem
    intermediate CA key/etc/puppetlabs/puppet/ssl/ca/ca_key.pemCan't have a passphrase, because the PE CA can't provide one when using the key.
    root CA certificate/etc/puppetlabs/puppet/ssl/ca/root_crt.pemCan be placed anywhere, because Puppet Server doesn't use it directly, but the rest of these instructions assume you placed it as shown.
    Note: All certificate files must have their ownership set to pe-puppet:pe-puppet and have permissions of 0600.
  2. Generate the CA bundle, by combining the root CA and intermediate CA certificates into one PEM file.
    cd /etc/puppetlabs/puppet/ssl/ca
    cat ca_crt.pem root_crt.pem > ../certs/ca.pem
  3. Update the CA serial file: printf '%04x' 3 > /etc/puppetlabs/puppet/ssl/ca/serial
  4. Generate a certificate for Puppet Server to use, including any dns_alt_names that the server must service.
    puppet cert generate puppetserver.my.domain.net --dns_alt_names=puppetserver,puppet
  5. Configure a CRL file from the root CA.

    If you have a pre-generated CRL from the root CA, install it into /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem.

    If you don't have a pre-generated CRL from the root CA, create one by running puppet cert generate fakehost and then revoking this certificate with puppet cert clean fakehost.

  6. Re-generate the certificate inventory file.
    puppet cert reinventory

Reconfigure PE infrastructure

After replacing the public key infrastructure, you must regenerate all required certificates.

Perform these steps on your master.

  1. Set certificate revocation to false in the main section of puppet.conf.
    [main]
    certificate_revocation = false
  2. Remove the cached cataog: rm -f /opt/puppetlabs/puppet/cache/client_data/catalog/<PUPPET MASTER FQDN>.json
  3. Reconfigure PE: puppet infrastructure configure
  4. Run Puppet: puppet agent -t 

Update agents to use the new CA

Prepare agents to use the new CA configuration by copying the CA bundle in place and configuring certificate revocation.

Before you begin
Back up the /etc/puppetlabs/puppet/ssl/ directory. If something goes wrong, you might need to restore these directories so your deployment remains functional.
Tip: If you were regenerating certs for security reasons and the process failed, contact Support as soon as you restore service so we can help you secure your site.
Complete these steps on each agent, unless otherwise indicated.
  1. Stop Puppet services:
    puppet resource service puppet ensure=stopped
    puppet resource service mcollective ensure=stopped
    puppet resource service pxp-agent ensure=stopped
    # Stop pe-puppetserver if the agent is a compile master
    puppet resource service pe-puppetserver ensure=stopped
  2. Copy the CA bundle you created to /etc/puppetlabs/puppet/ssl/certs/ca.pem.
    Note: If you copy this file into place before the first Puppet run, you won't recieve any errors. If you attempt a Puppet run prior to this file being present, you receive errors because the auto-distributed ca.pem file doesn't include the root CA.
  3. On your master, set certificate revocation to false in the main section of puppet.conf:
    [main]
    certificate_revocation = false
  4. Remove the certificate file, certificate revocation list (CRL), and the agent's cached catalog:
    find /etc/puppetlabs/puppet/ssl -name "$(puppet config print certname).pem" -or -name crl.pem -delete
    rm -f "/opt/puppetlabs/puppet/cache/client_data/catalog/$(puppet config print certname).json"
  5. Restart the agent service: puppet resource service puppet ensure=running
  6. If you aren't using autosigning, from the console or your master's command line, sign the agent's certificate signing request.
  7. From the console or your master's command line, run Puppet: puppet agent -t
The agent performs a full catalog run, restarts the PXP agent service, and resumes its role in your deployment.
Back to top
The page rank or the 1 our of 5 rating a user has given the page.
The email address of the user submitting feedback.
The URL of the page being ranked/rated.