Backing up and restoring PE

Keep regular backups of your Puppet Enterprise (PE) infrastructure. Backups allow you to more easily migrate to new primary server hardware, migrate your PE installation after a major OS upgrade, troubleshoot your installation, and quickly recover from system failures.

Restriction:

This information is intended for standard installations without disaster recovery. While you can backup large and extra-large installations, you might not be able to perform a complete recovery. For large and extra-large installations, consider Disaster recovery or alternative backup or snapshot options.

If your installation includes disaster recovery, you must Restore your infrastructure before provisioning a new replica.

Restriction:

You can use the puppet-backup command to back up and restore your primary server. You can't use this command to back up compilers.

The puppet-backup command does not backup or restore secret keys. You must backup and restore these separately and securely.

Customizing backup and restore scope

By default, the PE backup and restore commands include your PE configuration, PE certificates, Puppet code, and PuppetDB. However, you can use the --scope option to customize which data is backed up or restored.

By default, the puppet-backup command backs up (or restores) the following data:
  • Your PE configuration, including license, classification, and RBAC settings. However, the configuration backup data does not include Puppet gems or Puppet Server gems.
  • PE CA certificates and the full SSL directory.
  • The Puppet code deployed to your code directory at the time of the backup.
  • PuppetDB data, including facts, catalogs and historical reports.
CAUTION: The puppet-backup command does not include secret keys. You must back up this data separately and securely.

If you want to have discrete backup files, or if you want to back up some parts of your infrastructure more often than others, you can use --scope command line option to limit the scope of a backup or restore. The --scope option accepts one or more of certs, code, config, or puppetdb. If unspecified, the default value is all. For details about what is and isn't included in each scope, refer to Directories and data in backups.

For example, if you have frequent code changes, you might back up your Puppet code more often than you back up the rest of your infrastructure. When you limit the backup scope, the backup file contains only the parts of your infrastructure that you specify. Be sure to identify the scope in your backup file's name so you know what each file contains.

Restriction:

In PE 2023.0 and 2021.0 - 2021.7.2, backup operations with a scope of certs might fail to back up the certificate authority (CA) root key and certificates because of a change in the default CA directory (cadir). If the CA root key and certificates are not backed up, run a full backup without specifying a scope. Similarly, if you run the puppet-backup restore command with a scope of certs, the restore operation might fail. As a workaround, run a full restore operation.

This workaround is necessary because the CA default directory (cadir) was changed starting with Puppet 7, but the update was not immediately implemented in the puppet-backup create and puppet-backup restore commands. For more information about the directory change, see New CA directory location. The directory location is documented in cadir.

When you restore your primary server, you must restore your Puppet configuration, certificates, code, and PuppetDB data. However, you can restore each aspect from different files, either by using backup files that have limited scopes or by limiting the restore scope. For example, by limiting the scope when you run the puppet-backup restore command, you could restore Puppet code, configuration, and certificates from one backup file, and then restore PuppetDB from a different backup file.

Important: When you restore your primary server, you must restore all four data sets: configuration, certificates, code, and PuppetDB. However, you can't restore data that isn't contained in the backup file you're restoring from. For example, a backup file that only contains PuppetDB data can only be used to restore PuppetDB data. In this case you'll need to run the puppet-backup restore command multiple times, restoring a different file each time, until you have restored all four data sets (configuration, certificates, code, and PuppetDB).

Back up your infrastructure

The Puppet Enterprise (PE) backup process creates a copy of your primary server, including configuration, certificates, code, and PuppetDB. Backup can take several hours depending on the size of PuppetDB.

Before you begin
If you want to encrypt your backup, you must import your GPG public key to your primary server.
To create a complete set of backup data, you need to backup your infrastructure's secret keys and use the puppet-backup command to backup your PE configuration, PE certificates, Puppet code, and PuppetDB data. For details about the data included in backup files, refer to Customizing backup and restore scope.
  1. Make sure pg_repack doesn't run during the backup process by stopping the pe_databases module timers: 
    systemctl stop pe_databases-*.timer
  2. Run the puppet-backup command on your primary server. The default command is: 
    sudo puppet-backup create --dir=<BACKUP_DIRECTORY>
    You can use these parameters:
    • --dir=<BACKUP_DIRECTORY>: Required. Specify a separate disk as your backup directory.
      Important: Store backups in a secure, centralized location that you can access if your primary server fails.
    • --name=<BACKUP_NAME>: Specify the backup file's name. The default name is pe_backup with a timestamp indicating when the backup file was created, such as: pe_backup-<TIMESTAMP>.tgz.
    • --pe-environment=<ENVIRONMENT>: Specify an environment to back up. To ensure the configuration is recovered correctly, this must be the environment where your primary server is located. The default value is production.
    • --scope=<SCOPE_LIST>: Specify the data you want the backup file to contain. This is used for Customizing backup and restore scope. The default scope is all. If you only want to backup specific data, specify one or more of: certs, code, config, or puppetdb.
      Tip: If you specify --scope, it is a good idea to specify a --name that describes the file's scope.
    • --gpgkey=<KEY_ID>: Specify a GPG key ID to use to encrypt the backup file.
    • --force: Bypass validation checks and ignore warnings.
  3. Back up the secret keys directory and, if applicable, the secret key for the LDAP service.
    CAUTION:

    The puppet-backup command does not include secret keys. You must back up this data separately.

    Secure the keys as you would any sensitive information.

    • The secret keys directory is located at: /etc/puppetlabs/orchestration-services/conf.d/secrets/
    • (If applicable) The LDAP service key is located at: /etc/puppetlabs/console-services/conf.d/secrets/keys.json
  4. Restart the pe_databases module timers: 
    systemctl start pe_databases-catalogs.timer pe_databases-facts.timer pe_databases-other.timer
Results
Each time you use puppet-backup to create a new backup, PE creates a single backup file containing everything you're backing up (defined by the --scope). PE writes backup files to /var/puppetlabs/backups, unless you specify a different location in the puppet-backup command. The file name follows the default naming convention (pe_backup-<TIMESTAMP>.tgz), unless you specified a different name in the puppet-backup command.
Remember: The puppet-backup command does not backup secret keys. You must back up these separately.

Restore your infrastructure

Use the Puppet Enterprise (PE) restore process to migrate your primary server to a new host or to recover from system failure.

Before you begin

You must have created backup files, as described in Back up your infrastructure.

To decrypt an encrypted backup, you must import the GPG key pair (both the public and private keys) that you used for encryption to your new primary server.

Important: Make sure pg_repack doesn't run prior to or during the restore process.
  1. If you are restoring to a previously-existing primary server, you must purge any existing PE installation before reinstalling and restoring. To do this:
    1. On your primary server, run this command to uninstall PE: 
      sudo /opt/puppetlabs/bin/puppet-enterprise-uninstaller -p -d
    2. Ensure that these directories are no longer present on the system: /opt/puppetlabs/ and /etc/puppetlabs/
  2. Install PE on the primary server you want to restore to. You must install the same PE version used to create your backup files.
    1. If you don't have the PE installer script on the machine you want to restore to, download the installer tarball and unpack it by running: tar -xf <TARBALL_FILENAME>
    2. Navigate to the directory containing the install script. Normally, the installer script is located in the PE directory that is created when you unpacked the tarball.
    3. To install PE, run: sudo ./puppet-enterprise-installer
  3. On your primary server, run the puppet-backup restore command to restore your PE infrastructure. The default command is: 
    sudo puppet-backup restore <BACKUP-FILENAME>
    You can use these parameters:
    • <BACKUP-FILENAME>: Required. Specify the pe_backup file you want to restore from. The location, name, and contents of your backup files are determined when you Back up your infrastructure.
    • --pe-environment=<ENVIRONMENT>: Specify an environment to restore. The default value is production.
    • --scope=<SCOPE_LIST>: Specify the data you want to restore. This is used for Customizing backup and restore scope. The default scope is all. If you only want to restore specific data, specify one or more of: certs, code, config, or puppetdb.
    • --force: Bypass validation checks and ignore warnings.
    Important: You might need to run the puppet-backup restore command again if either the backup file or your restore command had limited --scope. For more information, refer to Customizing backup and restore scope.
  4. Restore your secret key backups. These keys are used to encrypt and decrypt sensitive data in the inventory service, orchestrator, and the LDAP service (if enabled).
    • Inventory service and orchestrator keys directory location is: /etc/puppetlabs/orchestration-services/conf.d/secrets/
    • LDAP service key file location is: /etc/puppetlabs/console-services/conf.d/secrets/keys.json
    CAUTION: The puppet-backup restore command does not include secret keys. You must restore this data separately.
  5. Make sure the inventory service's secret key ownership is configured as: pe-orchestration-services:pe-orchestration-services
  6. If the LDAP service is enabled, make sure the LDAP service's secret key ownership is configured as: pe-console-services:pe-console-services
  7. Restart pe-orchestration-services and pe-console-services.
  8. If you restored PE onto a primary server with a different hostname than the original installation, and you have not configured the dns_alt_names setting in the pe.conf file, you need to redirect your agents to the new primary server. One way to do this is by running a task with the Bolt task runner:
    1. If Bolt isn't already installed, download and install Bolt.
    2. To update the puppet.conf file to point all agents at the new primary server, run these commands: 
      bolt task run puppet_conf action=set section=agent setting=server value=<RESTORE_HOSTNAME> --targets <COMMA-SEPARATED_LIST_OF_NODES>
bolt task run puppet_conf action=set section=main setting=server value=<RESTORE_HOSTNAME> --targets <COMMA-SEPARATED_LIST_OF-NODES>
    3. On the newly restored primary server, run puppet agent -t --no-use_cached_catalog to apply the changes. Then run the same command again to restart services.
      Important: You must run the command twice.
    4. On each agent node, run puppet agent -t --no-use_cached_catalog to test the connection to the new primary server.
  9. If Code Manager was enabled when you created your backup file, deploy your code by running these commands: 
    puppet-access login
puppet code deploy --all --wait
  10. If your installation includes disaster recovery, provision a new replica.

Directories and data in backups

These directories and data are included in PE backups.

A default puppet-backup command captures all scopes, meaning all directories and data described in the table below. However, you can use the --scope option to limit the contents of backup files, or to restore data from multiple backup files, as described in Customizing backup and restore scope. In this case, the --scope option indicates which directories and data to back up or restore.

Scope Directories and databases
certs (PE certificates)
  • PE CA certificates
  • /etc/puppetlabs/puppet/ssl/
code (Puppet code) This scope captures the Puppet code deployed to your code directory at the time of the backup. Specifically:
  • /etc/puppetlabs/code/
  • /etc/puppetlabs/code-staging/
  • /opt/puppetlabs/server/data/puppetserver/filesync/storage/
  • /opt/puppetlabs/server/data/orchestration-services/data-dir
  • /opt/puppetlabs/server/data/orchestration-services/code
config (PE configuration) This scope captures your PE configuration, including license, classification, and RBAC settings. Some directories and data are excluded, such as Puppet gems, Puppet Server gems, and directories captured in other scopes. Specifically, it includes:
  • The orchestrator, RBAC, and classifier databases
  • The contents of /etc/puppetlabs/, except:
    • The /code and /code-staging directories, which are included in the code scope.
    • The /puppet/ssl directory, which is included in the certs scope.
  • The contents of /opt/puppetlabs/, except:
    • /puppet
    • /server/pe_build
    • /server/data/packages
    • /server/apps
    • /server/data/postgresql
    • /server/data/enterprise/modules
    • /server/data/puppetserver/vendored-jruby-gems
    • /bin
    • /client-tools
    • /server/share
    • /server/data/puppetserver/filesync/storage
    • /server/data/puppetserver/filesync/client
    • /server/data/orchestration-services/data-dir and /server/data/orchestration-services/code, which are included in the code scope.
puppetdb (PuppetDB)
  • PuppetDB data, including facts, catalogs, and historical reports
  • The contents of /opt/puppetlabs/server/data/puppetdb, except the /stockpile directory.