As your Puppet Enterprise infrastructure scales up to 4,000 nodes and beyond, add load-balanced compile masters to your monolithic installation to increase the number of agents you can manage.
Each compile master increases capacity by 1,500 to 3,000 nodes, until you exhaust the capacity of PuppetDB or the console.
How compile masters work
A single master can't efficiently process requests and compile code for more than 4,000 nodes. When you reach this scale, expand your infrastructure by adding compile masters to share the workload and compile catalogs faster.
In a configuration that uses compile masters, the main master is known as the master of masters (MoM).
Components and services running on compile masters
All compile masters contain a Puppet Server and a file sync client.
When triggered by a web endpoint, file sync takes changes from the working directory on the MoM and deploys the code to a live code directory. File sync then deploys that code to all your compile masters, ensuring that all masters in a multi-master configuration remain in sync. By default, compile masters check for code updates every five seconds.
The certificate authority (CA) service is disabled on compile masters. A proxy service running on the compile master Puppet Server directs CA requests to the MoM, which hosts the CA in default installations.
Compile masters also have:
- The MCollective client,
peadmin
- The repository for agent installation,
pe_repo
- The controller profile used with PE client tools
- Puppet Communications Protocol (PCP) brokers to enable orchestrator scale
Logs for compile masters are located at /var/log/puppetlabs/puppetserver/
.
Logs for PCP brokers on compile masters are located at /var/log/puppetlabs/puppetserver/pcp-broker.log
.
Install compile masters
To install a compile master, you first install an agent and then classify that agent as a compile master.
Configure compile masters for orchestrator scale
As your installation grows, you can scale orchestrator jobs by configuring connections between your agents, load balancers, and compile masters.
The orchestrator communicates with PCP brokers on compile masters over port 8143, and sends job-related messages to the brokers, which are then relayed by the brokers to Puppet Execution Protocol (PXP) agents. As you add compile masters, you also scale the number of PCP brokers that can send orchestration messages to agents.
When adding compile masters, you should load balance PXP agent connections to the PCP brokers running on them. PXP agents connect to PCP brokers running on compile masters over port 8142. Status checks on compile masters must be sent to port 8140, using https://<hostname>:8140/status/v1/simple
.
Tasks also require a configuration. If you want to download tasks from a Puppet master, you configure puppet_enterprise::profile::agent::master_uris
to a list of load balancer host names.
- Ensure port 8143 is open on the master of masters or on any workstations used to run orchestrator jobs.
- Configure infrastructure agents to connect to the MoM.
- Configure agents to connect to your load balancer.
Using load balancers with compile masters
In a large environment installation (LEI), agent requests are distributed among a pool of compile masters in order to increase the environment's total capacity.
You can configure a simple TCP load balancer to route traffic between agents and compile masters on port 8140.
Specifics on how to configure a load balancer infrastructure falls outside the scope of this document, but examples of how to leverage haproxy
for this purpose can be found in the HAproxy module documentation.
Load balancing for orchestration scale
PCP brokers run on compile masters and connect to PXP agents over port 8142. PCP brokers are built on websockets and require many persistent connections. If you're not using HTTP health checks, we recommend using a round robin or random load balancing algorithm for PXP agent connections to PCP brokers, because PCP brokers don't operate independent of the orchestrator and will isolate themselves if they become disconnected. You can check connections with the /status/v1/simple
endpoint for an error state.
You must also configure your load balancer to avoid closing long-lived connections that have little traffic. In the HAproxy module, you can set the timeout tunnel
to 15m
since PCP brokers disconnect inactive connections after 15 minutes.
Using health checks
The Puppet REST API exposes a status endpoint that can be leveraged from a load balancer health check to ensure that unhealthy hosts do not receive agent requests from the load balancer.
The master service responds to unauthenticated HTTP GET requests issued to https://<hostname>:8140/status/v1/simple
. The API responds with an HTTP 200 status code if the service is healthy.
If your load balancer doesn't support HTTP health checks, a simpler alternative is to check that the host is listening for TCP connections on port 8140. This ensures that requests aren't forwarded to an unreachable instance of the master, but it does not guarantee that a host is pulled out of rotation if it's deemed unhealthy, or if the service listening on port 8140 is not a service related to Puppet.
Optimizing workload distribution
Due to the diverse nature of the network communications between the agent and the master, we recommend that you implement a load balancing algorithm that distributes traffic between compile masters based on the number of open connections. Load balancers often refer to this strategy as "balancing by least connections."
Configuring pe_repo
for agent installation
When installing a new agent from a load balanced pool of compile masters, the agent configuration points to whichever compile master handled the request, instead of the load balancer itself. To ensure agents take advantage of pooled masters, specify the pe_repo::compile_master_pool_address
in the PE Master node group.