Migrate 3.x data to 4.x

The Continuous Delivery for PE 4.x series runs on a managed Kubernetes cluster, so you must migrate your 3.x series data to the new installation.

The migration process has four parts:
  1. Upgrade your existing 3.x installation to version 3.13.8.
  2. Create a new installation of Continuous Delivery for PE version 4.x.
  3. From the version 3.x root console, run the migration task.
  4. Restart the 4.x installation, which is now populated with your 3.x data.

What is migrated?

The migration task copies and migrates to 4.x:
  • The database, which contains all your Continuous Delivery for PE 3.x installation's information, users, integrations, history, and artifacts (except job commands and job logs).
    Note: If you configured your 4.x installation to use an externally managed PostgreSQL database, this step is skipped.
  • The object store, which contains all of your 3.x installation's job commands and job logs.
    Note: The 4.x series replaces external Artifactory and Amazon S3 object storage with a built-in highly available object storage system. Your 3.x object store is migrated to the built-in 4x object store.
  • Any of these configuration settings, if in use on 3.x:
    CD4PE_REPO_CACHE_RETRIEVAL_TIMEOUT_MINUTES
CD4PE_LDAP_GROUP_SEARCH_SIZE_LIMIT
CD4PE_ENABLE_REPO_CACHING
CD4PE_HTTP_CONNECTION_TIMEOUT_SEC
CD4PE_HTTP_READ_TIMEOUT_SEC
CD4PE_HTTP_WRITE_TIMEOUT_SEC
CD4PE_HTTP_REQUEST_TIMEOUT_SEC
CD4PE_INCLUDE_GIT_HISTORY_FOR_CD4PE_JOBS
CD4PE_JOB_GLOBAL_TIMEOUT_MINUTES
CD4PE_JOB_HTTP_READ_TIMEOUT_MINUTES
The migration task updates in the 4.x installation:
  • The webhooks between your source control system and the Continuous Delivery for PE backend endpoint URL.
    Note: This step is optional when running the migration task. The migration task is designed safely run multiple times, so you can make sure your 4.x installation is ready for full-time use before you enable webhook updates.
The migration task does not copy or move to 4.x:
  • The 3.x root user, because you create a new root user when installing 4.x.
  • The JVM_ARGS configuration setting, if in use on 3.x.
  • These backup tables created for you by Continuous Delivery for PE during the 3.x series:
    • app-pipelines-3-0-backup.pfi, which was created during the upgrade from 2.x to 3.x
    • pe-ia-node-results-cve-2020-7944-backup.pfi and pe-ia-resource-results-cve-2020-7944-backup.pfi, which were created as part of the CVE 2020-7944 remediation in version 3.4.0

Migration cautions

A super user or the root user must perform this process.

Migrating from a local database in a 3.x installation to an external database in a 4.x installation is not supported.

Important: Once you initiate the migration task, any new data created in the 3.x installation may not be migrated to the 4.x installation. To avoid losing data, restrict access to Continuous Delivery for PE during migration.

Run the migration task

The migration task uses temporary credentials to establish a connection between your 3.x and 4.x installations, then migrates your 3.x installation data to your new 4.x installation.

Before you begin
  1. Upgrade your 3.x installation to version 3.13.8. The migration task fails if you run it on any other 3.x version.
  2. Create a new 4.x installation:
    1. Review the Puppet Application Manager (PAM) system requirements.
    2. Follow the instructions for your environment to Install PAM.
    3. For hosts with internet access, Deploy Continuous Delivery for PE. For airgapped hosts, Deploy Continuous Delivery for PE offline.
    CAUTION: Do not configure integrations or create new data in the 4.x installation until you complete migration.
  3. Make sure your 3.x installation can reach the Kubernetes API running on your 4.x installation.
  4. If your 4.x installation is behind a proxy, set HTTP_PROXY and HTTPS_PROXY as values for the cd4pe_docker_extra_params parameter on the cd4pe class for your 3.x installation, passing in the URL of your proxy.
    Note: To pass additional arguments to the Docker process running the Continuous Delivery for PE container, you must specify them as an array. For example: "-e HTTPS_PROXY=https://proxy.example.com", "-e HTTP_PROXY=https://proxy.example.com"
Once you've completed the above preparations, you're ready to run the migration task:
  1. In PAM, click Config.
  2. In the Migration area, select the We're migrating an existing Continuous Delivery for PE 3.x instance option. Scroll to the bottom of the page and click Save config.
    Enabling this setting allows data to move between your 3.x and 4.x installations.
  3. When the success message appears, click Go to new version. On the Version history page, click Deploy.
  4. To generate temporary credentials to connect your two installations, run the following script on the server where you installed 4.x. If necessary, change the value of the CD4PE_NAMESPACE variable to your installation's namespace. Most installations use the default namespace. 
    #!/bin/bash
C_DEFAULT="\e[0m"
C_GREEN="\e[0;32m"

CD4PE_NAMESPACE="default"

K8S_SERVER_URL="$(kubectl config view --minify -o jsonpath='{.clusters[*].cluster.server}')"
K8S_TOKEN="$(kubectl -n $CD4PE_NAMESPACE get secret $(kubectl -n $CD4PE_NAMESPACE get secrets | grep ^cd4pe-migration | cut -f1 -d ' ') -o jsonpath="{['data']['token']}")"
if [ -z "${K8S_TOKEN}" ]; then
  printf "Kubernetes token not found.  Check to ensure 'Enable migration of an existing CD4PE instance' is checked in the application configuration\n"
  exit -1
fi

K8S_CA_CERT="$(kubectl config view --raw --minify -o jsonpath='{.clusters[*].cluster.certificate-authority-data}')"

printf "${C_GREEN}Kubernetes API server URL:${C_DEFAULT}\n\n%s\n\n" \
  "${K8S_SERVER_URL}"

printf "${C_GREEN}Kubernetes service account token:${C_DEFAULT}\n\n%s\n\n" \
  "${K8S_TOKEN}"

printf "${C_GREEN}Kubernetes API CA certificate:${C_DEFAULT}\n\n%s\n\n" \
  "${K8S_CA_CERT}"
  5. In the Continuous Delivery for PE 3.x root console, go to Settings > Migration. Copy the temporary credentials you generated in the previous step into the appropriate fields.
  6. Click Start migration and confirm your settings.
  7. Monitor the status of the migration task in the 3.x Docker logs. Run docker logs cd4pe to access the logs.
    The migration task first migrates your database tables, then migrates your object store, and lastly updates your webhooks, if applicable. When complete, Ran migration task in <elapsed time> prints to the log.
  8. Redeploy the 4.x installation. In PAM, click Config.
  9. In the Migration area, deselect the We're migrating an existing Continuous Delivery for PE 3.x instance option. Click Save config.
  10. When the success message appears, click Go to new version. On the Version history page, locate the new version and click Deploy.
  11. Go to your 4.x installation, which now is populated with migrated data, and log in with your 3.x user credentials (not the root account).

Test the new 4.x installation

You must thoroughly test the 4.x installation to make sure the data was properly migrated and all functionality is working as expected.

  1. Test the data migration to make sure your 3.x installation's data is present in your 4.x installation. For each workspace in your 4.x installation:
    1. Review event logs for all control repos and modules.
    2. Check that all jobs and hardware connections are present.
    3. Check that all users and user groups are present.
  2. If any data is missing or looks incorrect, rerun the migration task.
  3. Test manual pipeline runs to make sure your 4.x installation is functioning properly. For each workspace in your 4.x installation, trigger a manual pipeline run for all control repos and modules. To do this, select Trigger pipeline from the Manual actions menu above each pipeline.
What to do next
Once you've confirmed your 3.x data is present in your 4.x installation and all manual pipeline runs succeed, Update webhooks.

Update webhooks

As the final step in the 3.x to 4.x data migration process, update the webhooks that connect Continuous Delivery for Puppet Enterprise (PE) to your source control system.

Follow these steps if you're updating webhooks after migrating data from 3.x to 4.x. There are separate instructions to Update webhooks in other scenarios.
  1. In the Continuous Delivery for PE 3.x root console, navigate to Settings > Endpoints and copy the backend service endpoint.
  2. In the Continuous Delivery for PE 4.x root console, click Settings > Webhooks, and enter the backend service endpoint you copied from the 3.x root console.
  3. Click Update webhooks.
Results
Continuous Delivery for PE updates all webhooks to point to the current installation.
CAUTION: After migrating data from 3.x to 4.x, once you update webhooks to point to your 4.x installation, do not create new data in the 3.x installation, and do not run the migration task again.

Post-migration guidance and troubleshooting

After migrating Continuous Delivery for Puppet Enterprise (PE) data from 3.x to 4.x, thouroughly testing your 4.x installation, and updating webhooks, you're ready to begin using 4.x with your team. However, do not immediately decommission your 3.x installation. You may need to roll back to 3.x if issues arise.

Important: Keep your 3.x installation intact until you are fully confident that all features of the new 4.x installation are working as expected. As a precaution, we strongly advise maintaining your inactive 3.x installation for the first two months of using 4.x.

Maintaining your dormant 3.x installation for an extended test period allows you to roll back to 3.x and prevent production environment disruptions if an issue arises with your 4.x migration.

Rolling back to 3.x

If necessary, you can temporarily roll back to your 3.x installation by switching the active webhooks from the 4.x installation to the 3.x installation and generating new access tokens for all PE instances connected with the 3.x installation.
  1. In your Continuous Delivery for PE 3.x installation's web UI, navigate to a control repo's pipeline and click Manage pipelines.
  2. Select Manage in the web UI and locate the Automation webhook section where the webhook (including token) for your control repo is shown.
  3. Copy the webhook and navigate to the corresponding repository in your source control system. Add the 3.x webhook to the repository and remove the 4.x webhook.
  4. Repeat this process for each control repo and module repo in your 3.x installation.
  5. Generate a new access token for each PE instance connected to the 3.x installation. In the Continuous Delivery for PE web UI, navigate to Settings > Puppet Enterprise and click More actions > Edit integration.
  6. Select Basic authorization or API token and enter the required information:
    • For Basic authorization, enter the username and password for your "Continuous Delivery" user. Continuous Delivery for PE uses this information to generate an API token for you. The username and password are not saved.
    • For API token, generate a PE access token for your "Continuous Delivery" user using puppet-access or the RBAC API v1 service, and paste it into the API token field. For information about generating PE access tokens and the RBAC API v1 service, see Token-based authentication in the PE documentation.
      Note: To avoid unintended service interruptions, create access tokens with a multi-year lifespans.

Once the 3.x webhooks and new PE access tokens are in place, you can resume using the 3.x installation.

When you're ready to return to 4.x, repeat the migration process, starting with Run the migration task.