As your Puppet Enterprise infrastructure scales up to 4,000 nodes and beyond, add load-balanced compile masters to your monolithic installation to increase the number of agents you can manage.
Each compile master increases capacity by 1,500 to 3,000 nodes, until you exhaust the capacity of PuppetDB or the console.
How compile masters work
A single master can process requests and compile code for up to 4,000 nodes. When you exceed this scale, expand your infrastructure by adding compile masters to share the workload and compile catalogs faster.
https://<hostname>:8140/status/v1/simple
.
Components and services running on compile masters
All compile masters contain a Puppet Server and a file sync client.
When triggered by a web endpoint, file sync takes changes from the working directory on the MoM and deploys the code to a live code directory. File sync then deploys that code to all your compile masters, ensuring that all masters in a multi-master configuration remain in sync. By default, compile masters check for code updates every five seconds.
The certificate authority (CA) service is disabled on compile masters. A proxy service running on the compile master Puppet Server directs CA requests to the MoM, which hosts the CA in default installations.
Compile masters also have:
- The repository for agent installation,
pe_repo
- The controller profile used with PE client tools
- Puppet Communications Protocol (PCP) brokers to enable orchestrator scale
Logs for compile masters are located at /var/log/puppetlabs/puppetserver/
.
Logs for PCP brokers on compile masters are located at /var/log/puppetlabs/puppetserver/pcp-broker.log
.
Using load balancers with compile masters
When using more than one compile master, a load balancer can help distribute the load between the compile masters and provide a level of redundancy.
Specifics on how to configure a load balancer infrastructure falls
outside the scope of this document, but examples of how to leverage haproxy
for this purpose can be found in the HAproxy
module documentation.
Load balancing
PCP brokers run on compile
masters and connect to PXP agents
over port 8142. PCP brokers are
built on websockets and require many persistent connections. If you're not using
HTTP health checks, we recommend using a round robin or random load balancing
algorithm for PXP agent
connections to PCP brokers,
because PCP brokers don't operate
independent of the orchestrator and will isolate themselves if they become
disconnected. You can check connections with the /status/v1/simple
endpoint for an error state.
You must also configure your load balancer to avoid closing
long-lived connections that have little traffic. In the HAproxy module, you can set
the timeout tunnel
to 15m
since PCP brokers disconnect inactive connections after 15
minutes.
Using health checks
The Puppet REST API exposes a status endpoint that can be leveraged from a load balancer health check to ensure that unhealthy hosts do not receive agent requests from the load balancer.
The master
service responds to unauthenticated HTTP GET requests issued to https://<hostname>:8140/status/v1/simple
.
The API responds with an HTTP 200 status code if the service is healthy.
If your load balancer doesn't support HTTP health checks, a simpler alternative is to check that the host is listening for TCP connections on port 8140. This ensures that requests aren't forwarded to an unreachable instance of the master, but it does not guarantee that a host is pulled out of rotation if it's deemed unhealthy, or if the service listening on port 8140 is not a service related to Puppet.
Optimizing workload distribution
Due to the diverse nature of the network communications between the agent and the master, we recommend that you implement a load balancing algorithm that distributes traffic between compile masters based on the number of open connections. Load balancers often refer to this strategy as "balancing by least connections."
Install compile masters
To install a compile master, you first install an agent and then classify that agent as a compile master.
After installing compile masters, you must configure them to appropriately route communication between your master of masters and agent nodes.
Configure compile masters
Compile masters must be configured to appropriately route communication between your master of masters and agent nodes.
-
Install compile masters and load balancers.
-
If you need DNS altnames for your load balancers, add them to the master.
-
Ensure port 8143 is open on the master of masters or on any workstations used to run orchestrator jobs.