Troubleshooting Continuous Delivery for PE

Use this guide to troubleshoot issues with your Continuous Delivery for Puppet Enterprise (PE) installation.

Note: For PE instances with a replica configured for disaster recovery. In the event of a partial failover, Continuous Delivery for PE is not available. Learn more at What happens during failovers in the PE documentation. To restore Continuous Delivery for PE functionality, you must promote the replica to primary server.

Look up a source control webhook

Continuous Delivery for PE creates a webhook and attempts to automatically deliver it to your source control system when you add a new control repo or module to your workspace. You can look up this webhook if you ever need to manually add or re-add it to your source control repository.

  1. In the Continuous Delivery for PE web UI, click Control Repos or Modules and select the control repo or module whose webhook you want to view.
  2. In the Pipelines area of the screen, click Manage pipelines.
  3. If necessary, select Manage in the web UI.
    Note: If you're managing the pipelines for this control repo or module as code, don't worry. After copying the webhook, do not save any changes and switch back to Manage as code before closing the window. Your pipeline code will not be affected.
  4. In the Automation webhook area, copy the full webhook address printed in the black box. This is the unique webhook created to connect this control repo or module in Continuous Delivery for PE with its corresponding repository in your source control system.
What to do next
Add the webhook to the corresponding repository in your source control system, following the source control system's documentation. In most cases, webhooks are managed in the repository's settings area.

Manually configure a Puppet Enterprise integration

When you add credentials for a PE instance, Continuous Delivery for PE attempts to look up the endpoints for PuppetDB, Code Manager, orchestrator, and node classifier, and to access the primary SSL certificate generated during PE installation. If this information can't be located, such as in cases where your PE instance uses customized service ports, you are asked to enter it manually.

This task assumes you have completed the steps in Add your Puppet Enterprise credentials and have been redirected to the manual configuration page.

  1. In the Name field, enter a unique friendly name for your Puppet Enterprise installation.

    If you need to work with multiple PE installations within Continuous Delivery for PE, these friendly names help you to differentiate which installation's resources you're managing, so choose them carefully.

  2. In the API token field, paste a PE access token for your "Continuous Delivery" user. Generate this token using puppet-access or the RBAC v1 API.
    For instructions on generating an access token, see Token-based authentication.
  3. In the four service fields, enter the endpoints for your PuppetDB, Code Manager, orchestrator, and node classifier. You can locate these endpoints with the PE console.
    1. In the PE console, go to Overview (or Status in newer PE versions) and click Puppet Services status.
    2. Copy the endpoints from the Puppet Services status monitor and paste them into the appropriate fields in Continuous Delivery for PE, omitting the https:// prefix for each endpoint, as shown in the sample below.
      Puppet Services status format Continuous Delivery for PE format
      PuppetDB Service https://sample.host.puppet:8081 sample.host.puppet:8081
      Code Manager Service https://sample.host.puppet:8140 sample.host.puppet:8170
      Important: Use port 8170 for Code Manager in Continuous Delivery for PE.
      Orchestrator Service https://sample.host.puppet:8143 sample.host.puppet:8143
      Classifier Service https://sample.host.puppet:4433 sample.host.puppet:4433
  4. Locate the master SSL certificate generated by PE during installation by running:
    curl https://<HOSTNAME>:8140/puppet-ca/v1/certificate/ca --insecure
  5. Copy the certificate in its entirety (including the header and footer) and paste it into the CA certificate field.
  6. Click Save Changes.
  7. Optional: Once the main PE integration is set up, manually configure impact analysis.
    1. Generate a dedicated PE certificate to authenticate with the endpoints used for impact analysis. On the node hosting your PE certificate authority service (typically the primary server), run the following, replacing <CERTIFICATE NAME> with the name you want to give to this certificate, such as cd4pe-impact-analysis:
      puppetserver ca generate --certname <CERTIFICATE_NAME>
    2. Complete the steps in Configure impact analysis to install the relevant modules and update classification.
    3. In the Continuous Delivery for PE web UI, navigate to Settings > Puppet Enterprise and click Edit credentials to reopen the configuration pane.
    4. Follow the instructions in step 3 above to locate the endpoint for Puppet Server. Enter it in the Puppet Server service field, omitting the https:// prefix.
      Important: To run impact analysis tasks on a compiler or load balancer instead of the primary server (strongly recommended for PE installations that use these components as part of their architecture) enter the hostname of the compiler or load balancer at :8140 (for example, loadbalancer.example.com:8140) in the Puppet Server service field.
    5. Retrieve the new PE certificate you generated earlier and its private key from your Puppet Server certificate authority service by running the following:
      cat /etc/puppetlabs/puppet/ssl/certs/<CERTIFICATE_NAME>.pem
      cat /etc/puppetlabs/puppet/ssl/private_keys/<CERTIFICATE_NAME>.pem
    6. Copy the certificate in its entirety (including the header and footer) and paste it in the Impact Analysis Certificate field. Then copy the private key in its entirety (including the header and footer) and paste it in the Impact Analysis Private Key field.
    7. Click Save Changes.

Restart Continuous Delivery for PE

Continuous Delivery for PE is run in a managed Kubernetes cluster, so restarting the pod is an appropriate first step when troubleshooting.

To restart the pod, run:
kubectl rollout restart deployment cd4pe
For more on the kubectl rollout command, see the Kubernetes documentation.

Logs

Because Continuous Delivery for PE is run in a managed Kubernetes cluster, the software's logs are accessed with the kubectl logs command.

To access the Continuous Delivery for PE logs, run:
kubectl logs deployment/cd4pe
To access your installation's PostgreSQL logs, run:
kubectl logs statefulset/postgres

Trace-level logging

Enable or disable trace-level logging by setting Enable trace logging in the Advanced configuration and tuning section of the Config page on Puppet Application Manager.

PE component errors in logs

The logs include errors for both Continuous Delivery for PE and for the numerous PE components used by the software. It’s important to realize that an error in the Continuous Delivery for PE log may actually indicate an issue with Code Manager, r10k, or another component.

For example, this log output indicates a Code Manager issue:
Module Deployment failed for PEModuleDeploymentEnvironment[nodeGroupBranch=cd4pe_lab, 
nodeGroupId=a923c759-3aa3-43ce-968a-f1352691ca02, nodeGroupName=Lab environment, 
peCredentialsId=PuppetEnterpriseCredentialsId[domain=d3, name=lab-MoM], 
pipelineDestinationId=null, targetControlRepo=null, targetControlRepoBranchName=null, 
targetControlRepoHost=null, targetControlRepoId=null]. 
Puppet Code Deploy failure: Errors while deploying environment 'cd4pe_lab' (exit code: 1): 
ERROR -> Unable to determine current branches for Git source 'puppet' (/etc/puppetlabs/code-staging/environments) 
Original exception: malformed URL 'ssh://git@bitbucket.org:mycompany/control_lab.git' 
at /opt/puppetlabs/server/data/code-manager/worker-caches/deploy-pool-3/ssh---git@bitbucket.org-mycompany-control_lab.git

For help resolving issues with PE components, see the Puppet Enterprise troubleshooting documentation.

Error IDs in web UI error messages

Occasionally, error messages shown in the Continuous Delivery for PE web UI include an error ID and instructions to contact the site administrator. For example:
Screenshot of the web UI showing an error message instructing the user to "Contact the site administrator for support along with errorid"

Errors of this kind are shown without additional detail for security purposes. Users with root access to the Continuous Delivery for PE host system can search the log for the error ID to learn more.

Duplicate job logs after reinstall

A job's log is housed in object storage after the job is complete. If you reinstall Continuous Delivery for PE but reuse the same object storage without clearing it, you might notice logs for multiple jobs with the same job number, or job logs already present when a new job has just started.

To remove duplicate job logs and prevent their creation, make sure you clear both the object storage and the database when reinstalling Continuous Delivery for PE.

Name resolution

Continuous Delivery for PE uses CoreDNS for name resolution. Many can't reach and timeout connecting errors reported in the logs are actually DNS lookup failures.

To add DNS entries to CoreDNS, run:
kubectl -n kube-system edit configmaps coredns
The system will open the configmap file in an editor. Add a hosts stanza directly after the kubernetes stanza using the following format:
hosts /etc/hosts <DOMAIN> {
  <IP ADDRESS> <HOSTNAME> [ALIASES]
  fallthrough
}
Here's an example edited configmap with a hosts stanza added:
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        hosts /etc/hosts puppetdebug.vlan {
          10.234.4.29 pe-201922-master.puppetdebug.vlan pe-201922-master
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2020-08-25T17:34:17Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "10464"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: ba2907be-0067-4382-b103-fc248974719a

Looking up information about Continuous Delivery for PE

Use kubectl commands to access information about your Continuous Delivery for PE installation.

Look up the environment variables in use

To list the environment variables in use on your installation, run:

kubectl describe deployments.apps cd4pe
Note: For information on tuning your Continuous Delivery for PE instance by setting environment variables, including adjusting HTTP and job timeout periods, changing the size of LDAP server searches, or enabling Git repository caching, see Advanced Continuous Delivery for PE configuration.

Look up Continuous Delivery for PE version

To print the version of Continuous Delivery for PE running in your installation, run:
kubectl get deployment cd4pe -o jsonpath='{.spec.template.spec.containers[0].image}' ;  printf "\n"

Drain a node

Drain the impacted nodes when performing maintenance on your Kubernetes cluster such as upgrading system packages or rebooting the system.

To drain a node in order to perform system maintenance, run:
/opt/ekco/shutdown.sh