Migrating 3.x data to 4.x

Sections

The Continuous Delivery for PE 4.x series runs on a managed Kubernetes cluster, so your 3.x series data must be migrated to the new installation.

The migration process has three main steps:
  1. Create a new installation of Continuous Delivery for PE version 4.x.
  2. From the version 3.x root console, run the migration task.
  3. Restart the 4.x installation, which is now populated with your 3.x data.

What gets migrated?

The migration task copies and migrates to 4.x:

  • The database, which contains all your Continuous Delivery for PE 3.x installation's information, users, integrations, history, and artifacts except job commands and job logs.
    Note: If you've configured your 4.x installation to use an externally managed PostgreSQL database, this step is skipped.
  • The object store, which contains all of your 3.x installation's job commands and job logs.
    Note: The 4.x series replaces external Artifactory and Amazon S3 object storage with a built-in highly available object storage system. Your 3.x object store will be migrated to the built-in 4x object store.
  • Any of the following configuration settings, if in use on 3.x:
    CD4PE_REPO_CACHE_RETRIEVAL_TIMEOUT_MINUTES
    CD4PE_LDAP_GROUP_SEARCH_SIZE_LIMIT
    CD4PE_ENABLE_REPO_CACHING
    CD4PE_HTTP_CONNECTION_TIMEOUT_SEC
    CD4PE_HTTP_READ_TIMEOUT_SEC
    CD4PE_HTTP_WRITE_TIMEOUT_SEC
    CD4PE_HTTP_REQUEST_TIMEOUT_SEC
    CD4PE_INCLUDE_GIT_HISTORY_FOR_CD4PE_JOBS
    CD4PE_JOB_GLOBAL_TIMEOUT_MINUTES
    CD4PE_JOB_HTTP_READ_TIMEOUT_MINUTES
The migration task updates in the 4.x installation:
  • The webhooks between your source control system and the Continuous Delivery for PEbackend endpoint URL.
    Note: This step is optional when running the migration task. The migration task is designed to be safely run more than once, so you can make sure your 4.x installation is ready for full-time use before you enable webhook updates.
The migration task does not copy or move to 4.x:
  • The 3.x root user, as you created a new root user when installing 4.x.
  • The JVM_ARGS configuration setting, if in use on 3.x.
  • Backup tables created for you by Continuous Delivery for PE during the 3.x series:
    • app-pipelines-3-0-backup.pfi - created during the upgrade from 2.x to 3.x
    • pe-ia-node-results-cve-2020-7944-backup.pfi and pe-ia-resource-results-cve-2020-7944-backup.pfi - created as part of the CVE 2020-7944 remediation in version 3.4.0

Migration cautions

A super user or the root user must perform this process.

Migrating from a local database in a 3.x installation to an external database in a 4.x installation is not supported.

Once the migration task is initiated, any new data created in the 3.x installation may not be migrated to the 4.x installation. To avoid losing data, restrict access to Continuous Delivery for PE while the migration process is underway.

The migration process is potentially memory-intensive, and can exceed the default Java heap size set by Continuous Delivery for PE. If you encounter an out of memory error during the migration process, double the heap size on your 4.x installation by adding Java virtual machine arguments such as the following on the Config page of the platform admin console:
-Xms512m -Xmx1G

Run the migration task

The migration task uses temporary credentials to establish a connection between your 3.x and 4.x installations, then migrates your 3.x installation data to a new 4.x installation.

Before you begin
  1. Upgrade your 3.x installation to version 3.13.0 or newer. The migration task is only available in Continuous Delivery for PE 3.13.0 and newer versions.
  2. Follow the instructions in Install Continuous Delivery for PE or Install Continuous Delivery for PE in an offline environment to create a new 4.x installation.
    CAUTION: Do not perform any integrations or create any new data in the 4.x installation until migration is complete.
  3. Make sure that your 3.x installation can reach the Kubernetes API running on your 4.x installation.
  4. If your new 4.x installation is behind a proxy, set both HTTP_PROXY and HTTPS_PROXY as values for the cd4pe_docker_extra_params parameter on the cd4pe class for your 3.x installation, passing in the URL of your proxy.
    Note: To pass additional arguments to the Docker process running the Continuous Delivery for PE container, you must specify them as an array. For example: "-e HTTPS_PROXY=https://proxy.example.com", "-e HTTP_PROXY=https://proxy.example.com"
  1. In the platform admin console, click Config.
  2. In the Migration area, select the Enable migration of an existing CD4PE instance option. Scroll to the bottom of the page and click Save config.
    When this setting is enabled, data is allowed to move between your 3.x and 4.x installations.
  3. When the success message appears, click Go to new version. On the Version history page, click Deploy.
  4. Generate the temporary credentials needed to connect the two installations. On the server where you installed 4.x, run the following script:
    #!/bin/bash
    C_DEFAULT="\e[0m"
    C_GREEN="\e[0;32m"
    
    CD4PE_NAMESPACE="default"
    
    K8S_SERVER_URL="$(kubectl config view --minify -o jsonpath='{.clusters[*].cluster.server}')"
    K8S_TOKEN="$(kubectl -n $CD4PE_NAMESPACE get secret $(kubectl -n $CD4PE_NAMESPACE get secrets | grep ^cd4pe-migration | cut -f1 -d ' ') -o jsonpath="{['data']['token']}")"
    if [ -z "${K8S_TOKEN}" ]; then
      printf "Kubernetes token not found.  Check to ensure 'Enable migration of an existing CD4PE instance' is checked in the application configuration\n"
      exit -1
    fi
    
    K8S_CA_CERT="$(kubectl config view --raw --minify -o jsonpath='{.clusters[*].cluster.certificate-authority-data}')"
    
    printf "${C_GREEN}Kubernetes API server URL:${C_DEFAULT}\n\n%s\n\n" \
      "${K8S_SERVER_URL}"
    
    printf "${C_GREEN}Kubernetes service account token:${C_DEFAULT}\n\n%s\n\n" \
      "${K8S_TOKEN}"
    
    printf "${C_GREEN}Kubernetes API CA certificate:${C_DEFAULT}\n\n%s\n\n" \
      "${K8S_CA_CERT}"
    Note: If necessary, change the value of the CD4PE_NAMESPACE variable to your installation's namespace. Most installations use the default namespace.
  5. In the Continuous Delivery for PE root console of your 3.x installation, navigate to Settings > Migration. Copy the credentials you generated in step 4 into the appropriate fields.
  6. Click Start migration and confirm your settings.
  7. Monitor the status of the migration task in the 3.x Docker logs. Access the logs by running docker logs cd4pe.
    The migration task first migrates your database tables, then your object store, and finally updates your webhooks, if applicable. When complete, a Ran migration task in <time elapsed> message is printed to the log.
  8. Next, redeploy the 4.x installation. In the platform admin console, click Config.
  9. In the Migration area, deselect the Enable migration of an existing CD4PE instance option. Click Save config.
  10. When the success message appears, click Go to new version. On the Version history page, click Deploy. Navigate to the 4.x installation, which now is populated with migrated data, and log in with your 3.x user credentials (not the root account).
What to do next

If necessary, rerun the migration task. When you are satisfied with the data migration and ready to stop using the 3.x installation, proceed to Update webhooks.

Update webhooks

As a final step in the 3.x to 4.x data migration process, or any time you change the location of your Continuous Delivery for PE installation, update the webhooks that connect Continuous Delivery for PE with your source control provider.

  1. In the Continuous Delivery for PE 4.x root console, click Settings > Webhooks.
  2. Enter the backend service endpoint of your previous Continuous Delivery for PE installation.
    If you're updating webhooks as the final step in a 3.x to 4.x data migration, enter the backend serivce endpoint shown on the Settings > Endpoints page in the 3.x root console.
  3. Click Update webhooks.
    Continuous Delivery for PE updates all webhooks to point to the current installation.
Results
Once data migration to 4.x is completed successfully and webhooks are updated, you are ready to use Continuous Delivery for PE 4.x. Your 3.x installation is now obsolete. Uninstall or limit access to the 3.x installation and do not run the migration task again.
Puppet sites use proprietary and third-party cookies. By using our sites, you agree to our cookie policy.