Migrating 3.x data to 4.x

The Continuous Delivery for PE 4.x series runs on a managed Kubernetes cluster, so your 3.x series data must be migrated to the new installation.

The migration process has four main steps:

Upgrade your existing 3.x installation to version 3.13.8.
Create a new installation of Continuous Delivery for PE version 4.x.
From the version 3.x root console, run the migration task.
Restart the 4.x installation, which is now populated with your 3.x data.

What gets migrated?

The migration task copies and migrates to 4.x:

The database, which contains all your Continuous Delivery for PE 3.x installation's information, users, integrations, history, and artifacts except job commands and job logs.
Note: If you've configured your 4.x installation to use an externally managed PostgreSQL database, this step is skipped.
The object store, which contains all of your 3.x installation's job commands and job logs.
Note: The 4.x series replaces external Artifactory and Amazon S3 object storage with a built-in highly available object storage system. Your 3.x object store will be migrated to the built-in 4.x object store.

Any of the following configuration settings, if in use on 3.x:

CD4PE_REPO_CACHE_RETRIEVAL_TIMEOUT_MINUTES
CD4PE_LDAP_GROUP_SEARCH_SIZE_LIMIT
CD4PE_ENABLE_REPO_CACHING
CD4PE_HTTP_CONNECTION_TIMEOUT_SEC
CD4PE_HTTP_READ_TIMEOUT_SEC
CD4PE_HTTP_WRITE_TIMEOUT_SEC
CD4PE_HTTP_REQUEST_TIMEOUT_SEC
CD4PE_INCLUDE_GIT_HISTORY_FOR_CD4PE_JOBS
CD4PE_JOB_GLOBAL_TIMEOUT_MINUTES
CD4PE_JOB_HTTP_READ_TIMEOUT_MINUTES

The migration task updates in the 4.x installation:

The webhooks between your source control system and the Continuous Delivery for PEbackend endpoint URL.
Note: This step is optional when running the migration task. The migration task is designed to be safely run more than once, so you can make sure your 4.x installation is ready for full-time use before you enable webhook updates.

The migration task does not copy or move to 4.x:

The 3.x root user, as you created a new root user when installing 4.x.
The JVM_ARGS configuration setting, if in use on 3.x.
Backup tables created for you by Continuous Delivery for PE during the 3.x series:
- app-pipelines-3-0-backup.pfi - created during the upgrade from 2.x to 3.x
- pe-ia-node-results-cve-2020-7944-backup.pfi and pe-ia-resource-results-cve-2020-7944-backup.pfi - created as part of the CVE 2020-7944 remediation in version 3.4.0

Migration cautions

A super user or the root user must perform this process.

Migrating from a local database in a 3.x installation to an external database in a 4.x installation is not supported.

New data created in the 3.x installation during the migration process might not be migrated to the 4.x installation. If data created during the migration process is missing after migration is complete, re-run the migration task.

The migration process is potentially memory-intensive, and can exceed the default Java heap size set by Continuous Delivery for PE. If you encounter an out of memory error during the migration process, double the heap size on your 4.x installation by adding Java virtual machine arguments such as the following on the Config page of the platform admin console:

-Xms512m -Xmx1G

Run the migration task

The migration task uses temporary credentials to establish a connection between your 3.x and 4.x installations, then migrates your 3.x installation data to a new 4.x installation.

Before you begin

Upgrade your 3.x installation to version 3.13.8. The migration task fails if you run it on any other 3.x version.
Create a new 4.x installation.
1. Review the Puppet Application Manager system requirements.
2. Follow the instructions that match your environment to Install Puppet Application Manager.
3. According to your environment, either Configure and deploy Continuous Delivery for PE, or Configure and deploy Continuous Delivery for PE in an offline environment.
CAUTION: Do not perform any integrations or create any new data in the 4.x installation until migration is complete.
Make sure that your 3.x installation can reach the Kubernetes API running on your 4.x installation.
If your new 4.x installation is behind a proxy, set both HTTP_PROXY and HTTPS_PROXY as values for the cd4pe_docker_extra_params parameter on the cd4pe class for your 3.x installation, passing in the URL of your proxy.
Note: To pass additional arguments to the Docker process running the Continuous Delivery for PE container, you must specify them as an array. For example: "-e HTTPS_PROXY=https://proxy.example.com", "-e HTTP_PROXY=https://proxy.example.com"

In the platform admin console, click Config.
In the Migration area, select the We're migrating an existing Continuous Delivery for PE 3.x instance option. Scroll to the bottom of the page and click Save config.
When this setting is enabled, data is allowed to move between your 3.x and 4.x installations.
When the success message appears, click Go to new version. On the Version history page, click Deploy.

Generate the temporary credentials needed to connect the two installations. On the server where you installed 4.x, run the following script:

#!/bin/bash
C_DEFAULT="\e[0m"
C_GREEN="\e[0;32m"

CD4PE_NAMESPACE="default"

K8S_SERVER_URL="$(kubectl config view --minify -o jsonpath='{.clusters[*].cluster.server}')"
K8S_TOKEN="$(kubectl -n $CD4PE_NAMESPACE get secret $(kubectl -n $CD4PE_NAMESPACE get secrets | grep ^cd4pe-migration | cut -f1 -d ' ') -o jsonpath="{['data']['token']}")"
if [ -z "${K8S_TOKEN}" ]; then
  printf "Kubernetes token not found.  Check to ensure 'Enable migration of an existing CD4PE instance' is checked in the application configuration\n"
  exit -1
fi

K8S_CA_CERT="$(kubectl config view --raw --minify -o jsonpath='{.clusters[*].cluster.certificate-authority-data}')"

printf "${C_GREEN}Kubernetes API server URL:${C_DEFAULT}\n\n%s\n\n" \
  "${K8S_SERVER_URL}"

printf "${C_GREEN}Kubernetes service account token:${C_DEFAULT}\n\n%s\n\n" \
  "${K8S_TOKEN}"

printf "${C_GREEN}Kubernetes API CA certificate:${C_DEFAULT}\n\n%s\n\n" \
  "${K8S_CA_CERT}"

Note: If necessary, change the value of the CD4PE_NAMESPACE variable to your installation's namespace. Most installations use the default namespace.

In the Continuous Delivery for PE root console of your 3.x installation, navigate to Settings > Migration. Copy the credentials you generated in step 4 into the appropriate fields.
Click Start migration and confirm your settings.
Monitor the status of the migration task in the 3.x Docker logs. Access the logs by running docker logs cd4pe.
The migration task first migrates your database tables, then your object store, and finally updates your webhooks, if applicable. When complete, a Ran migration task in <time elapsed> message is printed to the log.
Next, redeploy the 4.x installation. In the platform admin console, click Config.
In the Migration area, deselect the We're migrating an existing Continuous Delivery for PE 3.x instance option. Click Save config.
When the success message appears, click Go to new version. On the Version history page, locate the newly created version and click Deploy. Navigate to the 4.x installation, which now is populated with migrated data, and log in with your 3.x user credentials (not the root account).

Test the new 4.x installation

It's extremely important to thoroughly test the 4.x installation to make sure that the data has been properly migrated and all functionality is working as expected before moving on.

Test the data migration. Make sure that the data from your 3.x installation is now present in your 4.x installation by completing the following for each workspace in your 4.x installation:
- Review the event logs for all control repos and modules
- Check that all jobs and hardware connections are present
- Check that all users and user groups are present
If you discover that any data is missing or looks incorrect, rerun the migration task.

Note: Each time it runs, the migration task deletes any database tables it finds in the 4.x installation and recreates the tables from the 3.x installation's current status.
Test manual pipeline runs. To make sure that the 4.x installation is functioning properly, trigger a manual pipeline run for each control repo and module in each workspace in your 4.x installation.
Run a pipeline manually by selecting Trigger pipeline from the Manual actions menu located above the pipeline.
When you've confirmed that all your 3.x data is present in the 4.x installation and manual pipeline runs complete successfully, move on to the next migration step: updating webhooks.

Update webhooks

As a final step in the 3.x to 4.x data migration process, or any time you change the location of your Continuous Delivery for PE installation, update the webhooks that connect Continuous Delivery for PE with your source control provider.

In the Continuous Delivery for PE 4.x root console, click Settings > Webhooks.
Enter the backend service endpoint of your previous Continuous Delivery for PE installation.
If you're updating webhooks as the final step in a 3.x to 4.x data migration, enter the backend service endpoint shown on the Settings > Endpoints page in the 3.x root console.
Click Update webhooks.
Continuous Delivery for PE updates all webhooks to point to the current installation.

Results

Once you update your webhooks to point to the 4.x installation, do not create new data in the 3.x installation, and do not run the migration task again.

Post-migration next steps and troubleshooting

Once your 3.x data is migrated, the 4.x installation has been thoroughly tested, and webhooks are updated, you're ready to begin using 4.x with your team. However, you should not immediately decommission your 3.x installation, as it may be necessary to roll back to 3.x if issues arise.

Keep your 3.x installation intact until you are fully confident that all features of the new 4.x installation are working as expected. As a precautionary measure, we strongly advise maintaining the inactive 3.x installation for the first two months of using 4.x.

Maintaining the dormant 3.x installation for an extended test period allows you to roll back to 3.x and prevent production environment disruptions in the event that an issue arises with your 4.x migration.

Rolling back to the 3.x installation

If necessary, you can temporarily roll back to the 3.x installation by switching the active webhooks from the 4.x installation to the 3.x installation and generating new access tokens for all PE instances connected with the 3.x installation.

In your 3.x installation, navigate to a control repo's pipeline and click Manage pipelines.
Select Manage in the web UI and locate the Automation webhook section, where the webhook (including token) for your control repo is printed.
Copy the webhook and navigate to the corresponding repository in your source control provider. Add the webhook to the repository and remove the webhook that points to the 4.x installation.
Repeat this process for each control repo and module repo in your 3.x installation.
Generate a new access token for each PE instance connected with the 3.x installation by navigating to Settings > Puppet Enterprise and clicking Edit credentials .
Select Basic authorization or API token and enter the required information:
- For Basic authorization, enter the username and password for your "Continuous Delivery" user. Continuous Delivery for PE uses this information to generate an API token for you. The username and password are not saved.
- For API token, generate a PE access token for your "Continuous Delivery" user using puppet-access or the RBAC v1 API, and paste it into the API token field. For instructions on generating an access token, see Token-based authentication.
  Note: To avoid unintended service interruptions, create an access token with a multi-year lifespan.

Once the 3.x webhooks and new PE access tokens are in place, you can resume using the 3.x installation.

When you're ready to return to 4.x, repeat the migration process from the beginning.