PE known issues

Windows agent installation fails with a manually transferred certificate

Performing a secure installation on Windows nodes by manually transferring the primary server CA certificate fails with the connection error: Could not establish trust relationship for the SSL/TLS secure channel.

Initial agent run after upgrade can fail with many environments

In installations with many environments, where file sync can take several minutes, the orchestration service fails to reload during the first post-upgrade Puppet run. As a workaround, re-run the Puppet agent until the orchestration service loads properly. To prevent encountering this error, you can clean up unused environments before upgrading, and wait several minutes after the installer completes to run the agent.

Installing or upgrading agents using TLSv1 fails with older OpenSSL versions

Script-based agent installation or upgrade on nodes that use TLSv1 can fail if the curl version installed on the node uses OpenSSL earlier than version 1.0. This issue produces an SSL error during any curl connection to the primary server. As a workaround, add --ciphers AES256-SHA to ~/.curlrc so that curl calls always use an appropriate cipher.

Converting legacy compilers fails with an external certificate authority

Agent_cert_regen: ERROR: Failed to regenerate agent certificate on node <compiler-node.domain.com>
Agent_cert_regen: bolt/run-failure:Plan aborted: run_task 'enterprise_tasks::sign' failed on 1 target
Agent_cert_regen: puppetlabs.sign/sign-cert-failed Could not sign request for host with certname <compiler-node.domain.com> using caserver <master-host.domain.com>

Converted compilers can slow PuppetDB in geo-diverse installations

In configurations that rely on high-latency connections between your primary servers and compilers – for example, in geo-diverse installations – converted compilers running the PuppetDB service might experience significant slowdowns. If your primary server and compilers are distributed among multiple data centers connected by high-latency links or congested network segments, reach out to Support for guidance before converting legacy compilers.

Upgrading a replica fails after regenerating the master certificate

puppet infra upgrade replica <REPLICA_FQDN> --only-recreate-databases

Running client tool commands against a replica can produce errors

Running puppet-code, puppet-access, or puppet query against a replica can produce the error certificate relies on legacy Common Name field. This error is due to the replica's certificate not having a subject alt name. As a workaround, regenerate the replica certificate: puppet infrastructure run regenerate_replica_certificate target=<REPLICA_HOSTNAME>.

Puppet Server FIPS installations don’t support Ruby’s OpenSSL module

FIPS-enabled PE installations don't support extensions or modules that use the standard Ruby Open SSL library, such as hiera-eyaml or the splunk_hec module. As a workaround, you can use a non-FIPS-enabled primary server with FIPS-enabled agents, which limits the issue to situations where only the agent uses the Ruby library.

Errors when using puppet code and puppet db commands on FIPS-compliant platforms

/opt/puppetlabs/puppet/bin/gem install puppetdb_cli --bindir /opt/puppetlabs/bin/

Enabling manage_delta_rpm causes agent run failures on CentOS and RHEL 8

Enabling the manage_delta_rpm parameter in the pe_patch class causes agent run failures on CentOS and RHEL 8 due to a package name change.

Restarting or running Puppet on infrastructure nodes can trigger an illegal reflective access operation warning

When restarting PE services or performing agent runs on infrastructure nodes, you might see the warning Illegal reflective access operation ... All illegal access operations will be denied in a future release in the command-line output or logs. These warnings are internal to PE service components, have no impact on their functionality, and can be safely disregarded.

Running plans during code deployment can result in failures

If a plan is running during a code deployment, things like compiling apply block statements or downloading and running tasks that are part of a plan might fail. This is because plans run on a combination of PE services, like orchestrator and puppetserver, and the code each service is acting on might get temporarily out of sync during a code deployment.

The apply shim in pxp-agent doesn't pick up changes

Exited 1:\n/opt/puppetlabs/pxp-agent/tasks-cache/apply_ruby_shim/apply_ruby_shim.rb:39:in `<main>': undefined method `map' for nil:NilClass (NoMethodError)\n

Plans still run after failure

When pe-orchestration-services exits unexpectedly, plan jobs are sometimes still running even though they failed.

Null characters in task output cause errors

Tasks that print null bytes cause an orchestrator database error that prevents the result from being stored. This issue occurs most frequently for tasks on Windows that print output in UTF-16 rather than UTF-8.

Gateway timeout errors in the console

Using facts to filter nodes might produce either a "502 Bad Gateway" or "Gateway Timeout" error instead of the expected results.

Login length requirements apply to existing remote users

The login length requirement prevents reinstating existing remote users when they are revoked, resulting in a permissions error in the console. The requirement is meant to only apply to local users only.

Editing a hash in configuration data causes parts of the hash to disappear

When editing configuring data with hash values in the console, some parts of the hash that did not get edited disappear after committing changes, and reappear when the hash is edited again.

Patching fails with excluded yum packages

In the patching task or plan, using yum_params to pass the --exclude flag in order to exclude certain packages can result in task or plan failure if the only packages requiring updates are excluded. As a workaround, use the versionlock command (which requires installing the yum-plugin-versionlock package) to lock the packages you want to exclude at their current version. Alternatively, you can fix a package at a particular version by specifying the version with a package resource for a manifest that applies to the nodes to be patched.

Changing a file type in a control repo produces a checkout conflict error

Changing a file type in a control repository – for example, deleting a file and replacing it with a directory of the same name – generates the error JGitInternalException: Checkout conflict with files accompanied by a stack trace in the Puppet Server log. As a workaround, deploy the control repo with the original file deleted, and then deploy again with the replacement file or directory.

Enabling Code Manager and multithreading in Puppet Server deadlocks JRuby

Setting the new environment_timeout parameter to any non-zero value – including the unlimited default when Code Manager is enabled – interferes with multithreading in Puppet Server and can result in JRuby deadlocking after several hours.

Default SSH URL with TFS fails with Rugged error

Using the default SSH URL with Microsoft Team Foundation Server (TFS) with the rugged provider causes an error of "unable to determine current branches for Git source." This is because the rugged provider expects an @ symbol in the URL format.

To work around this error, replace ssh:// in the default URL with git@

ssh://tfs.puppet.com:22/tfs/DefaultCollection/Puppet/_git/control-repo

git@tfs.puppet.com:22/tfs/DefaultCollection/Puppet/_git/control-repo

GitHub security updates might cause errors with shellgit

GitHub has disabled TLSv1, TLSv1.1 and some SSH cipher suites, which can cause automation using older crypto libraries to start failing. If you are using Code Manager or r10k with the shellgit provider enabled, you might see negotiation errors on some platforms when fetching modules from the Forge. To resolve these errors, switch your configuration to use the rugged provider, or fix shellgit by updating your OS package.

Timeouts when using --wait with large deployments or geographically dispersed compilers

Because the --wait flag deploys code to all compilers before returning results, some deployments with a large node count or compilers spread across a large geographic area might experience a timeout. Work around this issue by adjusting the timeouts_sync parameter.

r10k with the Rugged provider can develop a bloated cache

If you use the rugged provider for r10k, repository pruning is not supported. As a result, if you use many short-lived branches, over time the local r10k cache can become bloated and take up significant disk space.

If you encounter this issue, run git-gc periodically on any cached repo that is using a large amount of disk space in the cachedir. Alternately, use the shellgit provider, which automatically garbage collects the repos according to the normal Git CLI rules.

Code Manager and r10k do not identify the default branch for module repositories

When you use Code Manager or r10k to deploy modules from a Git source, the default branch of the source repository is always assumed to be main. If the module repository uses a default branch that is not main, an error occurs. To work around this issue, specify the default branch with the ref: key in your Puppetfile.

After an error during the initial run of file sync, Puppet Server won't start

The first time you run Code Manager and file sync on a primary server, an error can occur that prevents Puppet Server from starting. To work around this issue:

1. Stop the pe-puppetserver service.
2. Locate the data-dir variable in /etc/puppetlabs/puppetserver/conf.d/file-sync.conf.
3. Remove the directory.
4. Start the pe-puppetserver service.

Repeat these steps on each primary server exhibiting the same symptoms, including any compilers.

Puppet Server crashes if file sync can't write to the live code directory

If the live code directory contains content that file sync didn’t expect to find there (for example, someone has made changes directly to the live code directory), Puppet Server crashes.

The following error appears in puppetserver.log:

2016-05-05 11:57:06,042 ERROR [clojure-agent-send-off-pool-0] [p.e.s.f.file-sync-client-core] Fatal error during file sync, requesting shutdown.
org.eclipse.jgit.api.errors.JGitInternalException: Could not delete file /etc/puppetlabs/code/environments/development
        at org.eclipse.jgit.api.CleanCommand.call(CleanCommand.java:138) ~[puppet-server-release.jar:na]

To recover from this error:

1. Delete the environments in code dir: find /etc/puppetlabs/code -mindepth 1 -delete
2. Start the pe-puppetserver service: puppet resource service pe-puppetserver ensure=running
3. Trigger a Code Manager run by your usual method.

Code Manager can't recover from Puppetfile typos in URL

When you have a git typo in your Puppetfile, subsequent code deploys continuously fail until you manually delete deployer caches, even after the Puppetfile error is corrected.