PE known issues

Sections

These are the known issues in PE 2019.8.

Installation and upgrade known issues

These are the known issues for installation and upgrade in this release.

Upgrade and puppet infrastructure commands fail if your master is not in the production environment

Upgrades and puppet infrastructure commands — including replica upgrade and compiler provisioning, conversion, and upgrade — fail with a Bolt::RunFailure if your master is not in the production environment.

Follow these steps to ensure upgrades and puppet infrastructure commands work as expected:
  • Verify that you've specified your non-production infrastructure environment for these parameters:
    • pe_install::install::classification::pe_node_group_environment
    • puppet_enterprise::master::recover_configuration::pe_environment
  • Run puppet infra recover_configuration --pe-environment <MASTER_ENVIRONMENT>
  • When upgrading, run the installer with the --pe_environment flag: sudo ./puppet-enterprise-installer –- --pe_environment <MASTER_ENVIRONMENT>

Upgrade fails if a PostgreSQL repack is in progress

If a PostgreSQL repack operation is in progress when you attempt to upgrade PE — for example, if you're using the puppetlabs-pe_databases module or other methods to run regular repacks of the PostgreSQL database — the upgrade can fail with the error cannot drop extension pg_repack because other objects depend on it.

If you use cron jobs to repack the database, disable them before upgrading. If you encounter this error during upgrade, disable the cron jobs and wait for the repack in progress to complete before retrying the upgrade.

If the repack does not end in a timely manner, or if you have severe time constraints for your upgrade plan, contact Puppet Support for assistance.

Upgrade fails with an unenabled replica

PE upgrade fails if you have a provisioned, but not enabled, replica. As a workaround, edit the PE Master node group configuration with these parameters for the puppet_enterprise::profile::master class:
  • puppetdb_port = [8081]
  • puppetdb_host = ["MASTERS_FQDN_EXAMPLE"]

Upgrade fails when using custom database certificate parameters

When upgrading an infrastructure with an unmanaged PostgreSQL database to this version of PE, the upgrade can fail and result in downtime if the databases use custom certificates not issued by the PE certificate authority. This is the case even if you configured the database_properties parameters in pe.conf to set a custom sslrootcert. The upgrader ignores these custom certificate settings.

To manually override the upgrader's certificate settings, run these commands, replacing <NEW CERT> with the name of your custom certificate.
rpm -iUv /opt/puppetlabs/server/data/packages/public/2019.3.0/el-7-x86_64-6.12.0/pe-modules-2019.3.0.2-1.el7.x86_64.rpm
                    
sed -i 's#/etc/puppetlabs/puppet/ssl/certs/ca.pem#/etc/puppetlabs/puppet/ssl/<NEW CERT>.pem#'
/opt/puppetlabs/server/data/environments/enterprise/modules/pe_manager/lib/puppet_x/puppetlabs/meep/configure/postgres.rb
                        
./puppet-enterprise-installer - --force
Note: The first step in the code block above includes paths specific to the PE version you're upgrading to. Replace these version information as needed.

Compiler provisioning fails if a single compiler is unresponsive

The puppet infrastructure provision compiler command fails if any compiler in your pool fails a pre-provisioning health check.

Automated Puppet runs fail after running compiler commands

After provisioning or converting compilers with puppet infrastructure commands, automated Puppet runs can fail because the Puppet service hasn't restarted. As a workaround, manually restart the Puppet service after running the command.
puppet resource service puppet ensure=stopped
puppet resource service puppet ensure=running

puppet infrastructure commands fail with an external node classifier

If you use an external node claissifier, puppet infrastructure commands, such as puppet infrastructure compiler upgrade and puppet infrastructure compiler provision, fail. If you use an external node classifier, contact Puppet Support for manual workarounds before upgrading to an affected version of PE.

Converting legacy compilers fails with an external certificate authority

If you use an external certificate authority (CA), the puppet infrastructure run convert_legacy_compiler command fails with an error during the certificate-signing step.
Agent_cert_regen: ERROR: Failed to regenerate agent certificate on node <compiler-node.domain.com>
Agent_cert_regen: bolt/run-failure:Plan aborted: run_task 'enterprise_tasks::sign' failed on 1 target
Agent_cert_regen: puppetlabs.sign/sign-cert-failed Could not sign request for host with certname <compiler-node.domain.com> using caserver <master-host.domain.com>
To work around this issue when it appears:
  1. Log on to the CA server and manually sign certificates for the compiler.
  2. On the compiler, run Puppet: puppet agent -t
  3. Unpin the compiler from PE Master group, either from the console, or from the CLI using the command: /opt/puppetlabs/bin/puppet resource pe_node_group "PE Master" unpinned="<COMPILER_FQDN>"
  4. On your master, in the pe.conf file, remove the entry puppet_enterprise::profile::database::private_temp_puppetdb_host
  5. If you have an external PE-PostgreSQL node, run Puppet on that node: puppet agent -t
  6. Run Puppet on your master: puppet agent -t
  7. Run Puppet on all compilers: puppet agent -t

Disaster recovery known issues

These are the known issues for disaster recovery in this release.

Upgrade fails with an unenabled replica

PE upgrade fails if you have a provisioned, but not enabled, replica. As a workaround, edit the PE Master node group configuration with these parameters for the puppet_enterprise::profile::master class:
  • puppetdb_port = [8081]
  • puppetdb_host = ["MASTERS_FQDN_EXAMPLE"]

Provisioning a replica fails after regenerating the master certificate

If you previously regenerated the certificate for your master, provisioning a replica can fail due to permission issues with backed up directories. As a workaround, delete the directory /opt/puppetlabs/server/data/postgresql/11/data/certs_bak and then attempt provisioning again.

Replica commands can leave the Puppet service disabled

The reinitialize replica command as well as the provision replica command, which includes reinitializing, leaves the Puppet service disabled on the replica. As a workaround, manually restart the Puppet service on the replica after running the provision or reinitialize replica commands: puppet resource service puppet ensure=running

Skipping agent configuration when enabling a replica deletes settings for the PE Agent group

If you use the --skip-agent-config flag with puppet infra enable replica or puppet infra provision replica --enable, any custom settings that you've specified for server_list and pcp_broker_list in the PE Agent node group are deleted. As a workaround, restore these values after enabling the replica.

Runs, plans, and tasks fail after promoting a replica

After promoting a replica, infrastructure nodes can't connect to the newly promoted master because the master_uris value still points to the old master.

If you have an enabled replica, in both the PE Agent and PE Infrastructure Agent node groups, in the puppet_enterprise::profile::agent class, verify that the setting for master_uris matches the setting for server_list. Both values must include both your master and replica, for example ["MASTER.EXAMPLE.COM", "REPLICA.EXAMPLE.COM"]. Setting these values ensures that agents can continue to communicate with the promoted replica in the event of a failover.

If you already promoted your replica and encountered errors, in the PE Infrastructure Agent node group, in the puppet_enterprise::profile::agent class, update the value for master_uris to point to the new master. Then, run Puppet on all infrastructure nodes.

Puppet runs can take longer than expected in failover scenarios

In disaster recovery environments with a provisioned replica, if the master is unreachable, a Puppet run using data from the replica can take up to three times as long as expected (for example, 6 minutes versus 2 minutes).

FIPS known issues

These are the known issues with FIPS-enabled PE in this release.

Puppet Server FIPS installations don’t support Ruby’s OpenSSL module

FIPS-enabled PE installations don't support extensions or modules that use the standard Ruby Open SSL library, such as hiera-eyaml or the splunk_hec module. As a workaround, you can use a non-FIPS-enabled master with FIPS-enabled agents, which limits the issue to situations where only the agent uses the Ruby library.

Errors when using puppet code and puppet db commands on FIPS-compliant platforms

When the pe-client-tools packages are run on FIPS-compliant platforms, puppet code and puppet db commands fail with SSL handshake errors. To use puppet db commands on a FIPS-compliant platforms, install the puppetdb_cli Ruby gem with the following command:
/opt/puppetlabs/puppet/bin/gem install puppetdb_cli --bindir /opt/puppetlabs/bin/
To use puppet code commands on a FIPS-compliant platforms, use the Code Manager API. Alternatively, you can use pe-client-tools on a non-FIPS-compliant platform to access a FIPS-enabled master.

Configuration and maintenance known issues

These are the known issues for configuration and maintenance in this release.

Restarting or running Puppet on infrastructure nodes can trigger an illegal reflective access operation warning

When restarting PE services or performing agent runs on infrastructure nodes, you might see the warning Illegal reflective access operation ... All illegal access operations will be denied in a future release in the command-line output or logs. These warnings are internal to PE service components, have no impact on their functionality, and can be safely disregarded.

Console is inaccessible with PE set to IPv6

If you specify IPv6, PE Java services still listens to the IPv4 localhost. This mismatch can prevent access to the console as Nginx proxies traffic to the wrong localhost.

As a workaround, set the java.net.preferIPv6Addresses property to true for console services and PuppetDB:
"puppet_enterprise::profile::console::java_args": {
  "Xms": "256m"
  "Xmx": "256m"
  "Djava.net.preferIPv6Addresses": "=true"
}
 
"puppet_enterprise::profile::puppetdb::java_args": {
  "Xms": "256m"
  "Xmx": "256m"
  "Djava.net.preferIPv6Addresses": "=true"
}

puppet infrastructure recover_configuration misreports success if specified environment doesn't exist

If you specify an invalid environment when running puppet infrastructure recover_configuration, the system erroneously reports that the environment's configuration was saved.

Orchestration services known issues

These are the known issues for the orchestration services in this release.

The function puppet plan show displays invalid plans when Code Manager is disabled

If Code Manager is disabled, the puppet plan show function and the console will still list plans that are contained in a control repo, but those plans cannot be run. Enable Code Manager to run plans.

Running plans during code deployment can result in failures

If a plan is running during a code deployment, things like compiling apply block statements or downloading and running tasks that are part of a plan might fail. This is because plans run on a combination of PE services, like orchestrator and puppetserver, and the code each service is acting on might get temporarily out of sync during a code deployment.

Master reporting HTTP error after Qualys scan

When running a Qualys scan, the master reports the error "HTTP Security Header Not Detected. Issue at Port 443".

Apply blocks fail to compile

Puppetserver might fail to compile apply blocks for plans when there are more than eight variables, or when variables have names that conflict with key names for hashes or target settings in plans.

Yaml plans display all parameters as optional in the console

Yaml plans list all parameters as having default values, regardless of whether there is a default value set in the code or not. This causes all parameters to display defaults in orchestrator APIs and show as optional in the console.

The wait_until_available function doesn’t work with multiple transports

When a target included in the TargetSpec argument to the wait_until_available plan function uses the ACE (remote) transport, the function fails immediately and will not wait for any of the targets in the TargetSpec argument.

Console and console services known issues

These are the known issues for the console and console services in this release.

Gateway timeout errors in the console

Using facts to filter nodes might produce either a "502 Bad Gateway" or "Gateway Timeout" error instead of the expected results.

Patching known issues

These are the known issues for patching in this release.

Patch task can misreport success for Windows nodes

When patching Windows nodes, running the pe_patch::patch_server task always reports success, even if there are problems installing one or more updates. Unpatched nodes continue to appear in the list of nodes with available patches.

Code management known issues

These are the known issues for Code Manager, r10k, and file sync in this release.

Reenabling lockless code deploys can fail

Reenabling lockless code deploys can fail due to the persistence of the versioned code directory. If you disable lockless deploys after enabling it, your code directory is moved back to the default location, however the versioned code directory remains in place at /etc/puppetlabs/puppetserver/code. If you later reenable lockless deploys, you might encounter the error cannot overwrite directory. If possible, manually delete the versioned code directory before reenabling lockless deploys. Or, if you've already reenabled the setting and hit the overwrite error, run puppet code deploy --all to return to a working state.

Default SSH URL with TFS fails with Rugged error

Using the default SSH URL with Microsoft Team Foundation Server (TFS) with the rugged provider causes an error of "unable to determine current branches for Git source." This is because the rugged provider expects an @ symbol in the URL format.

To work around this error, replace ssh:// in the default URL with git@

For example, change:
ssh://tfs.puppet.com:22/tfs/DefaultCollection/Puppet/_git/control-repo
to
git@tfs.puppet.com:22/tfs/DefaultCollection/Puppet/_git/control-repo

GitHub security updates might cause errors with shellgit

GitHub has disabled TLSv1, TLSv1.1 and some SSH cipher suites, which can cause automation using older crypto libraries to start failing. If you are using Code Manager or r10k with the shellgit provider enabled, you might see negotiation errors on some platforms when fetching modules from the Forge. To resolve these errors, switch your configuration to use the rugged provider, or fix shellgit by updating your OS package.

Timeouts when using --wait with large deployments or geographically dispersed compilers

Because the --wait flag deploys code to all compilers before returning results, some deployments with a large node count or compilers spread across a large geographic area might experience a timeout. Work around this issue by adjusting the timeouts_sync parameter.

r10k with the Rugged provider can develop a bloated cache

If you use the rugged provider for r10k, repository pruning is not supported. As a result, if you use many short-lived branches, over time the local r10k cache can become bloated and take up significant disk space.

If you encounter this issue, run git-gc periodically on any cached repo that is using a large amount of disk space in the cachedir. Alternately, use the shellgit provider, which automatically garbage collects the repos according to the normal Git CLI rules.

Code Manager and r10k do not identify the default branch for module repositories

When you use Code Manager or r10k to deploy modules from a Git source, the default branch of the source repository is always assumed to be master. If the module repository uses a default branch that is not master, an error occurs. To work around this issue, specify the default branch with the ref: key in your Puppetfile.

After an error during the initial run of file sync, Puppet Server won't start

The first time you run Code Manager and file sync on a master, an error can occur that prevents Puppet Server from starting. To work around this issue:

  1. Stop the pe-puppetserver service.
  2. Locate the data-dir variable in /etc/puppetlabs/puppetserver/conf.d/file-sync.conf.
  3. Remove the directory.
  4. Start the pe-puppetserver service.

Repeat these steps on each master exhibiting the same symptoms, including any compilers.

Puppet Server crashes if file sync can't write to the live code directory

If the live code directory contains content that file sync didn’t expect to find there (for example, someone has made changes directly to the live code directory), Puppet Server crashes.

The following error appears in puppetserver.log:

2016-05-05 11:57:06,042 ERROR [clojure-agent-send-off-pool-0] [p.e.s.f.file-sync-client-core] Fatal error during file sync, requesting shutdown.
org.eclipse.jgit.api.errors.JGitInternalException: Could not delete file /etc/puppetlabs/code/environments/development
        at org.eclipse.jgit.api.CleanCommand.call(CleanCommand.java:138) ~[puppet-server-release.jar:na]

To recover from this error:

  1. Delete the environments in code dir: find /etc/puppetlabs/code -mindepth 1 -delete
  2. Start the pe-puppetserver service: puppet resource service pe-puppetserver ensure=running
  3. Trigger a Code Manager run by your usual method.

Code Manager can recover from Puppetfile typos in URL

When you have a git typo in your Puppetfile, subsequent code deploys continuously fail until you manually delete deployer caches, even after the Puppetfile error is corrected.

File-sync client repo grows after frequent commits

The file-sync client repo grows rapidly when there are frequent commits to it. For example, when syncing the CA dir for HA, and many new certificates are signed are revoked quickly.

SSL and certificate known issues

These are the known issues for SSL and certificates in this release.

Automated Puppet runs can fail after running certificate regeneration commands

After regenerating certificates with puppet infrastructure commands, automated Puppet runs can fail because the Puppet service hasn't restarted. As a workaround, manually restart the Puppet service after running the commands.
puppet resource service puppet ensure=stopped
puppet resource service puppet ensure=running

Client tools known issues

These are the known issues for client tools in this release.

Running puppet query produces a cryptic error

Running puppet query with insufficient permissions produces an error similar to this:
ERROR - &{<nil>  } (*models.Error) is not supported by the TextConsumer, can be resolved by supporting TextUnmarshaler interface

Generate a token from a user account with the correct permissions and then run the query again.

How helpful was this page?

If you leave us your email, we may contact you regarding your feedback. For more information on how Puppet uses your personal information, see our privacy policy.

Puppet sites use proprietary and third-party cookies. By using our sites, you agree to our cookie policy.