Troubleshooting Code Manager

Code Manager requires coordination between multiple components, including source control, r10k, and the file sync service. If you have issues with Code Manager, check that these components are functioning.

Code Manager logs

Code Manager logs to the Puppet Server log. By default, this log is at: /var/log/puppetlabs/puppetserver/puppetserver.log

For more information about working with the logs, see the Puppet Server logs documentation.

For source control webhook issues, check your Git host's logs.

Check Code Manager's status

Check the status of Code Manager and file sync if your deployments are not working as expected, or if you need to verify that Code Manager is enabled before running a dependent command.

The puppet-code status command verifies that Code Manager and file sync are responding. The command returns the same information as the GET /v1/deploys/status endpoint. By default, the command returns details at the info log level. It doesn't support critical and debug log levels.

Errors that puppet-code status might report include:
Code Manager couldn’t connect to the server
Occurs if the pe-puppetserver process isn’t running.
Code Manager reports invalid configuration
Occurs if there is an invalid configuration in the code-manager.conf file, located at: /etc/puppetlabs/puppetserver/conf.d/code-manager.conf
File sync storage service reports unknown status
Occurs if the status callback timed out.

Test the connection to the control repository

The control repository controls the existence of environments, and ensures that the correct versions of all the necessary modules are installed in your environments. The primary server must be able to access and clone the control repo as the pe-puppet user.

To make sure that Code Manager can connect to the control repo, run:

puppet-code deploy --dry-run

If the connection is set up correctly, this command returns a list of all environments found in the control repo (or repos, if you have multiple sources configured). A successful response means the control repo's SSH key has the correct permissions, the Git URL is correct, and the pe-puppet user can perform the necessary operations.

If there is a problem with the connection, the command returns this message: Unable to determine current branches for Git source. It also returns a file path on the primary server that you can use for debugging the SSH key and Git URL.

Check the Puppetfile for errors

Check the environment's Puppetfile for syntax errors and verify that every module in the Puppetfile can be installed from the listed source. To do this, you need a copy of an environment's Puppetfile in a temporary directory.

On the primary server, create a temporary directory at /var/tmp/test-puppetfile and place a copy of the Puppetfile into the temporary directory. From there, you can then check the syntax and sources in your Puppetfile.

To check the Puppetfile syntax, run r10k puppetfile check from within the temporary directory. If syntax errors are detected, correct them, and run the test again. If the syntax is correct, the command returns Syntax OK.

To test the configuration of sources in your Puppetfile, perform a test installation. In your temporary directory (at /var/tmp/test-puppetfile), run the following command:
sudo -H -u pe-puppet bash -c \
'/opt/puppetlabs/puppet/bin/r10k puppetfile install'

This command attempts to install modules listed in your Puppetfile to a modules directory in your temporary directory. This test verifies if there is access to all listed module sources. Take note of all errors that occur, because issues with individual modules can cause issues for the entire environment. Errors with an individual module (such as Git URL syntax or version issues) are reported as general errors for that module. If you have modules from private Git repositories that require an SSH key to clone the module, check that you are using the SSH Git URL and not the HTTPS Git URL. After correcting the reported errors, rerun the test install again to confirm the errors are resolved.

For more information, refer to Managing environment content with a Puppetfile.

Run a deployment test

You can manually run a full r10k deployment to check your Puppetfile syntax, access to sources, and whether the deployment works through r10k.

The following command attempts a full r10k deployment based on the r10k.yaml file that Code Manager uses. This test writes to the code staging directory only and doesn't trigger file sync. Only use this for ad-hoc testing. The test deployment command is:
sudo -H -u pe-puppet bash -c \
'/opt/puppetlabs/puppet/bin/r10k deploy environment -c \
/opt/puppetlabs/server/data/code-manager/r10k.yaml -p -v debug'

If the command succeeds, the /etc/puppetlabs/code-staging directory is populated with directory-based environments and all the necessary modules for every environment.

If the command fails, the error is likely caused by Code Manager's r10k-related settings. The error messages indicate which settings are failing. For more information, refer to Code Manager settings.

Source control webhook issues

Refer to Troubleshoot a Code Manager webhook.

Monitor /v1/deploys logs

If you're experiencing errors with deployments triggered through the POST /v1/deploys webhook, you can monitor logs when you call the endpoint.

To do this, you'll need to call the POST /v1/deploys endpoint with the wait parameter while monitoring the console services log. To monitor the console-services.log file, open a terminal window and run:
tail -f /var/log/puppetlabs/console-services/console-services.log

Code deployments time out

If your environments are heavily loaded, code deployments can take a long time, and the system might time out before deployment is complete.

If your deployments are timing out too soon, increase one or more of these settings:
  • timeouts_deploy
  • timeouts_shutdown
  • timeouts_sync
  • timeouts_wait
For descriptions of these settings, refer to Code Manager parameters. For instructions on configuring these settings, refer to Customize Code Manager configuration in Hiera.
Tip: Deployment timeouts can also occur when the file sync client holds code deployments while waiting for long-running plans to finish. To resolve this, you can increase the timeouts_sync setting or allow Running plans alongside code deployments.

File sync stops when Code Manager tries to deploy code

Code Manager's code deployments involves accessing many small files. If you store your Puppet code on network-attached storage, you might experience poor performance with deployments due to a slow network or problems with back-end hardware.

If you're experiencing performance issues, try to:
  • Tune the network to accommodate the many small files.
  • Store Puppet code on local or direct-attached storage.

Classes are missing after deployment

After a successful code deployment, a class you added isn't available in the PE console.

If your code deployment succeeds, but a class you added isn't available in the console, try these steps one at a time:
  • Refresh your browser.
  • Run Puppet to refresh classes.
  • Verify that the environment directory exists on disk.
  • Check your node group's settings to make sure the group has the correct environment assigned. You might need to run Puppet or redeploy environments after changing environment settings.