Configure orchestration

After installing PE, you can change some default settings to further configure the orchestrator and pe-orchestration-services.

Orchestrator and pe-orchestration-services parameters

These are some optional parameters you can use to configure the behavior of the orchestrator and the pe-orchestration-services service.

You can modify these profile class parameters in the Puppet Enterprise (PE) console on the Classes tab for the PE Orchestrator infrastructure node group.

puppet_enterprise::profile::agent::pxp_enabled
Boolean used to enable or disable the Puppet Execution Protocol (PXP) service.
Set to true to enable the PXP service, which is required to use the orchestrator and run Puppet from the console.
Set to false to disable the PXP service. If false, you can’t use the orchestrator or the Run Puppet button in the console.
Must be true to Configure PXP agent parameters.
Default: true
puppet_enterprise::profile::bolt_server::concurrency
An integer that determines the maximum number of simultaneous task or plan requests the orchestrator can make to bolt-server.
This setting only limits task or plan executions on nodes with SSH or WinRM transport methods, because these are the only tasks and plans requiring requests to bolt-server.
Default: 100 requests
CAUTION: Do not set a concurrency limit that is higher than the bolt-server limit. This can cause timeouts that lead to failed task runs.
puppet_enterprise::profile::orchestrator::allowed_pcp_status_requests
An integer that defines how many times an orchestrator job allows status requests to time out before a job is considered failed. Status requests wait 12 seconds between timeouts, so multiply the value of the allowed_pcp_status_requests by 12 to determine how many seconds the orchestrator waits on targets that aren’t responding to status requests.
Default: 35 timeouts
puppet_enterprise::profile::orchestrator::default_deploy_timeout
An integer specifying how long a deploy job can run on a single node before being force stopped.
CAUTION: Force stopping jobs can result in incomplete Puppet runs, partial configuration changes, and other issues. When setting the default timeout limit, consider average job scope, run time, and your infrastructure's capacity (such as concurrency limits).
Default: 1800 (30 minutes)
Deploy jobs triggered by puppet infra commands always have a one year timeout limit, regardless of the value of default_deploy_timeout.
puppet_enterprise::profile::orchestrator::default_plan_timeout
An integer specifying how long a plan can run before being force stopped. This represents a timeout limit for the entire plan, not individual plan jobs.
CAUTION: Force stopping plans can result in incomplete Puppet runs, partial configuration changes, and other issues. When setting the default timeout limit, consider average plan scope, run time, and your infrastructure's capacity (such as concurrency limits).
Default: 3600 (60 minutes)
Plans triggered by puppet infra commands always have a one year timeout limit, regardless of the value of default_plan_timeout.
puppet_enterprise::profile::orchestrator::default_task_node_timeout
An integer specifying how many seconds a task can run on a single node before being force stopped.
CAUTION: Force stopping tasks can result in incomplete Puppet runs, partial configuration changes, and other issues. When setting the default timeout limit, consider average task scope, run time, and your infrastructure's capacity (such as concurrency limits).
Default: 2400 (40 minutes)
Tasks triggered by puppet infra commands always have a one year timeout limit, regardless of the value of default_task_node_timeout.
puppet_enterprise::profile::orchestrator::global_concurrent_compiles
An integer specifying how many concurrent compile requests can be outstanding to the primary server across all orchestrator jobs.
Default: 8 requests
puppet_enterprise::profile::orchestrator::java_args
Specifies the Java heap size, which is the amount of JVM memory that each Java process is allowed to request from the OS for orchestration services to use.
The value is formatted as a JSON hash, where the maximum and minimum are usually the same. For example: {"Xmx": "256m", "Xms": "256m"}
Default: 704 MB
puppet_enterprise::profile::orchestrator::job_prune_threshold
An integer of 2 or greater, which specifies the number of days to retain job reports.
This parameter sets the corresponding parameter job-prune-days-threshold.
While job_prune_threshold itself has no default value, job-prune-days-threshold has a default of 30 (30 days).
puppet_enterprise::profile::orchestrator::jruby_max_active_instances
An integer that determines the maximum number of JRuby instances that the orchestrator creates to execute plans. Because each plan uses one JRuby to run, this value is effectively the maximum number of concurrent plans. Setting the orchestrator heap size (java_args) automatically sets the jruby_max_active_instances using the formula $java_args ÷ 1024. If the result is less than one, the default is one JRuby instance.
Default: 1 instance
Note: The jruby_max_active_instances pool for the orchestrator is separate from the Puppet Server pool. Refer to JRuby max active instances for more information.
puppet_enterprise::profile::orchestrator::max_connections_per_route_authenticated
An integer representing the maximum number of concurrent HTTP-client connections allowed for each route when requests include a client certificate.
Default: 20
puppet_enterprise::profile::orchestrator::max_connections_per_route_unauthenticated
An integer representing the maximum number of concurrent HTTP-client connections allowed for each route when requests do not include a client certificate.
Default: 20
puppet_enterprise::profile::orchestrator::max_connections_total_authenticated
An integer representing the maximum number of concurrent HTTP-client connections allowed for all routes when requests include a client certificate.
Default: 20
puppet_enterprise::profile::orchestrator::max_connections_total_unauthenticated
An integer representing the maximum number of concurrent HTTP-client connections allowed for all routes when requests do not include a client certificate.
Default: 20
puppet_enterprise::profile::orchestrator::pcp_timeout
An integer representing how long, in seconds, an agent can spend attempting to connect to a PCP broker during a Puppet run triggered by the orchestrator. If the agent can’t connect to the broker in the specified time frame, the Puppet run times out.
Default: 30
puppet_enterprise::profile::orchestrator::run_service
A Boolean used to enable (true) or disable (false) orchestration services.
Default: true
puppet_enterprise::profile::orchestrator::socket_timeout
An integer specifying, in milliseconds, the maximum wait time before the orchestrator closes an HTTP connection when no data is available on the socket.
Default: 120000
puppet_enterprise::profile::orchestrator::task_concurrency
An integer defining the maximum number of task or plan actions that can be executed simultaneously.
The defined concurrency limit applies to the nodes targeted by individual task or plan jobs. For example, if task_concurrency is set to 200 and four separate task jobs that each target 200 nodes are initiated, then task actions start executing on all 800 nodes.
If the number of nodes targeted by an individual task or plan job exceeds the concurrency limit, actions on some of the targeted nodes will be queued. For example, if task_concurrency is set to 200, and an individual task job is run on 300 targeted nodes, then task actions start executing on 200 nodes while the remaining 100 nodes wait in queue.
Default: 1000 actions per job
puppet_enterprise::profile::plan_executor::versioned_deploys
A Boolean used for Running plans alongside code deployments.
Set to true to enable versioned deployments of environment code.
Default: false
Important: Setting this to true disables the file sync client's locking mechanism that usually enforces a consistent environment state for your plans. This can cause Puppet functions and plans that call other plans to behave unexpectedly if a code deployment occurs while a plan is running.

For information about how the orchestrator works, what you can do with it, and additional parameters and configuration options, refer to Orchestrating Puppet runs, tasks, and plans.

For PXP agent parameters, refer to Configure PXP agent parameters.

Configure PXP agent parameters

Puppet Execution Protocol (PXP) is a message format used to request task execution and receive task statuses. PXP agents runs the PXP service. You can configure pxp_agent parameters with Hiera or in the PE console.

puppet_enterprise::profile::agent::pxp_enabled
Boolean used to enable or disable the Puppet Execution Protocol (PXP) service.
Set to true to enable the PXP service, which is required to use the orchestrator and run Puppet from the console.
Set to false to disable the PXP service. If false, you can’t use the orchestrator or the Run Puppet button in the console.
Default: true
puppet_enterprise::pxp_agent::ping_interval
An integer specifying the frequency, in seconds, that PXP agents ping PCP brokers. If the broker doesn't respond, the agent tries to reconnect.
Default: 120
puppet_enterprise::pxp_agent::pxp_logfile
The path, as a string, to the PXP agent log file. This file can be used to debug orchestrator issues.
The default value varies by OS.
  • *nix: "/var/log/puppetlabs/pxp-agent/pxp-agent.log"
  • Windows: "C:\Program Data\PuppetLabs\pxp-agent\var\log\pxp-agent.log"
puppet_enterprise::pxp_agent::spool_dir_purge_ttl
A string representing the amount of time to retain records of Puppet or task runs on agents.
Format the value as a string consisting of a number and one of the following suffixes: m (minutes), h (hours), or d (days).
Default: "14d" (14 days)
puppet_enterprise::pxp_agent::task_cache_dir_purge_ttl
A string representing the amount of time that task files are cached after use.
Format the value as a string consisting of a number and one of the following suffixes: m (minutes), h (hours), or d (days).
Default: "14d" (14 days)
puppet_enterprise::pxp_agent::broker_proxy
Optional. Set a proxy URI to use to connect to the pcp-broker to listen for task and Puppet run requests.
puppet_enterprise::pxp_agent::master_proxy
Optional. Set a proxy URI to use to connect to the primary server to download task implementations.
puppet_enterprise::pcp_max_message_size_mb
An integer specifying the maximum message size, in MB, for pcp_broker, pxp_agent, and the orchestrator.
The maximum message size cannot be higher than the default size of 64 MB. You can only specify a smaller value.
Default: 64
Important: Don't change the pcp_max_message_size_mb parameter if you send or receive large payloads, because this can cause errors for large task and plan run parameters and output.

Manage ARP table overflow

In larger deployments that use the PCP broker, you might encounter Address Resolution Protocol (ARP) table overflows.

Overflows occur when the ARP table (which is a local cache of IP-to-MAC-address resolutions) becomes full and starts evicting old entries. When long-established, but frequently-used, entries are evicted, network traffic increases to restore them. This increases network latency and CPU load on the broker.
Here is an example of a typical ARP table overflow log message:
[root@s1 peadmin]# tail -f /var/log/messages
Aug 10 22:42:36 s1 kernel: Neighbour table overflow.
Aug 10 22:42:36 s1 kernel: Neighbour table overflow.
Aug 10 22:42:36 s1 kernel: Neighbour table overflow.

To resolve this issue, you need to increase sysctl settings related to ARP tables.

For example, these settings are appropriate for networks hosting up to 2000 agents:
# Set max table size
net.ipv6.neigh.default.gc_thresh3=4096
net.ipv4.neigh.default.gc_thresh3=4096
# Start aggressively clearing the table at this threshold
net.ipv6.neigh.default.gc_thresh2=2048
net.ipv4.neigh.default.gc_thresh2=2048
# Don't clear any entries until this threshold
net.ipv6.neigh.default.gc_thresh1=1024
net.ipv4.neigh.default.gc_thresh1=1024