Installing compilers

As your Puppet Enterprise infrastructure scales up to 4,000 nodes and beyond, add load-balanced compilers to your installation to increase the number of agents you can manage.

Each compiler increases capacity by 1,500 to 3,000 nodes, until you exhaust the capacity of PuppetDB or the console.

How compilers work

A single primary server can process requests and compile code for up to 4,000 nodes. When you exceed this scale, expand your infrastructure by adding compilers to share the workload and compile catalogs faster.

Important: Compilers must run the same OS major version, platform, and architecture as the primary server.
Compilers act as PCP brokers, conveying messages between the orchestrator and Puppet Execution Protocol (PXP) agents. PXP agents connect to PCP brokers running on compilers over port 8142. Status checks on compilers must be sent to port 8140, using https://<hostname>:8140/status/v1/simple.
A diagram of compiler connections.

Components and services running on compilers

Compilers typically run Puppet Server and PuppetDB services, as well as a file sync client. Older, legacy-style compilers must be converted in order to add PuppetDB.

When triggered by a web endpoint, file sync takes changes from the working directory on the primary server and deploys the code to a live code directory. File sync then deploys that code to all your compilers. By default, compilers check for code updates every five seconds.

The certificate authority (CA) service is disabled on compilers. A proxy service running on the compiler Puppet Server directs CA requests to the primary server, which hosts the CA in default installations.

Compilers also have:

  • The repository for agent installation, pe_repo
  • The controller profile used with PE client tools
  • Puppet Communications Protocol (PCP) brokers to enable orchestrator scale

Logs for compilers are located at /var/log/puppetlabs/puppetserver/.

Logs for PCP brokers on compilers are located at /var/log/puppetlabs/puppetserver/pcp-broker.log. Logback configuration for PCP broker logs is part of the Orchestration services settings.

Using load balancers with compilers

When using more than one compiler, a load balancer can help distribute the load between the compilers and provide a level of redundancy.

Specifics on how to configure a load balancer infrastructure falls outside the scope of this document, but examples of how to leverage haproxy for this purpose can be found on the HAproxy module Forge page.

Calculating load balancing

For load balancing between the Puppet agent and the Puppet primary server, implement a load balancing algorithm that distributes traffic among compilers based on the number of open connections. Traffic is directed to the compiler with the smallest number of open connections. This strategy is known as “balancing by least connections.”

For load balancing between PCP brokers and PXP agents, a different method is used. PXP agents establish TCP connections to PCP brokers over port 8142. PCP brokers are built on web sockets and require many persistent connections. PCP brokers depend on maintaining connectivity to the Puppet orchestrator, but if the brokers become disconnected from the orchestrator, the brokers can fail at the HTTP level while still accepting TCP connections. Follow these guidelines:
  • If you are using HTTP health checks, use a "least connections" algorithm to distribute load evenly.
  • If you are not using HTTP health checks, use a round robin or random load balancing algorithm to avoid directing all traffic to an unhealthy PCP broker. You can check connections for possible errors by using the /status/v1/simple endpoint.

Using health checks

The Puppet REST API exposes a status endpoint that can be used for load balancer health checks to ensure that unhealthy hosts don’t receive agent requests.

To check the health of your hosts, issue HTTP GET requests to the following endpoints:

  • For Puppet agent traffic on port 8140, use https://<hostname>:8140/status/v1/simple/server. This endpoint checks the health of the pe-puppetserver service and returns an HTTP 200 status when the server is fully operational. It does not check the health of other services such as the broker-service.
  • For pxp-agent traffic on port 8142, use https://<hostname>:8140/status/v1/simple/broker-service. This endpoint checks the health of the broker-service and returns an HTTP 200 status code when the service is fully operational.

If your load balancer doesn't support HTTP health checks, you can use a TCP connection tests on port 8140 to check whether a host is listening. This test is useful for verifying that the port is open, but does not confirm whether the host is running a Puppet service.

Load balancing for multi-region installations

If you have load balancers in multiple regions, use a global DNS proximity-based service address.

Tip: For guidance about deploying PE in global, multi-region, or multi-network segment scenarios, see the Multi-region Reference Architectures article.

When using a centralized Puppet deployment with multiple regional proxies or load balancers, create a global DNS proximity-based service address for Puppet and use that to route agents to the appropriate regional load balancer based on their location. Set the global DNS proximity-based address as the compiler pool address in Hiera.

For example, in your common.yaml file:
pe_repo::compile_master_pool_address: "<PUPPET-GLOBAL-SERVICE-ADDRESS>"
Some suitable global DNS proximity-based service address implementations include:
  • BIG-IP DNS
  • Route 53 Geolocation routing in AWS
  • TCP Proxy Global Load Balancing in GCP
  • Traffic Manager in Azure

Install compilers

Installing a compiler adds the specified node to the PE Infrastructure Agent and PE Compiler node groups and installs the PuppetDB service on the node.

Before you begin
The node you want to provision as a compiler must have a Puppet agent installed, or you must be able to connect to a non-agent node with SSH.

Ensure that you have a valid admin RBAC token. For instructions, see Token-based authentication.

Important: Before you install compilers in multi-region installations, contact Support for guidance . If your primary server and compilers are connected with high-latency links or congested network segments, you might experience better PuppetDB performance with legacy compilers.

To install a FIPS-compliant compiler, install the compiler on a supported platform with FIPS mode enabled. The node must be configured with sufficient available entropy or the installation process fails.

  1. Configure the agent on infrastructure nodes to connect to the primary server.
    1. In the console, click Node groups, and in the PE Infrastructure group, select the PE Agent > PE Infrastructure Agent group.
    2. If you manage your load balancers with agents, on the Rules tab, pin load balancers to the group.
      Pinning load balancers to the PE Infrastructure Agent group ensures that they communicate directly with the primary server.
    3. On the Classes tab, find the puppet_enterprise::profile::agent class and specify these parameters:
      Parameter Value
      manage_puppet_conf Specify true to ensure that your setting for server_list is configured in the expected location and persists through Puppet runs.
      pcp_broker_list Hostname for your primary server and replica, if you have one. Hostnames must include port 8142, for example ["PRIMARY.EXAMPLE.COM:8142", "REPLICA.EXAMPLE.COM:8142"].
      primary_uris Hostname for your primary server and replica, if you have one, for example ["PRIMARY.EXAMPLE.COM", "REPLICA.EXAMPLE.COM"]. This setting assumes port 8140 unless you specify otherwise with host:port.
      server_list
    4. Remove any values set for pcp_broker_ws_uris.
    5. Commit changes.
    6. Run Puppet on all agents classified into the PE Infrastructure Agent group.
  2. Pin the node that you want to provision to the PE Infrastructure Agent group, and then run Puppet on the node (run puppet agent -t).
  3. On your primary server, logged in as root, run the following command to provision a single compiler: 
    puppet infrastructure provision compiler <COMPILER_FQDN>
    This command accepts a maximum of one compiler FQDN; this command can't provision multiple compilers at once. Additionally, you can specify these optional parameters:
    • dns-alt-names: Comma-separated list of any alternative names that agents use to connect to the compiler. The installation uses puppet by default.
      Important: If your puppet.conf file includes a dns_alt_names entry, you must include the dns-alt-names parameter when provisioning your compiler.
    • no-dns-alt-names: Prevents the installer from setting the default alternative name, puppet. Use this parameter if you don't allow alternative names (as indicated by allow-subject-alt-names: false in your ca.conf file).
    • use-ssh: Enables installing on a node that doesn't have a Puppet agent currently installed. You must be able to connect to the node with SSH. You can pair this flag with additional SSH parameters. Run puppet infrastructure provision --help for details.
  4. Verify that the contents of the global layer Hiera file on the new compiler, located at /etc/puppetlabs/puppet/hiera.yaml, match the contents of the global layer Hiera file on the primary server.
    • If necessary, update hiera.yaml on the compiler to match hiera.yaml on the primary server.
    • If you use code to manage the contents of hiera.yaml on the primary server, ensure that the new compiler is also classified to manage the contents of its own hiera.yaml file.
What to do next
Configure compilers to appropriately route communication between your primary server and agent nodes.

Configure compilers

Compilers must be configured to appropriately route communication between your primary server and agent nodes.

Before you begin
  • Install compilers and load balancers.
  • If you need DNS altnames for your load balancers, add them to the primary server.
  • Ensure port 8143 is open on the primary server or on any workstations used to run orchestrator jobs.
Restriction: This procedure is not intended for installations with load balancers in multiple locations. To configure compilers in multi-region installations, refer to Load balancing for multi-region installations.
  1. Configure pe_repo::compile_master_pool_address to send agent install requests to the load balancer.
    1. In the console, click Node groups, and in the PE Infrastructure group, select the PE Master group.
    2. On the Configuration data tab, select the pe_repo class, and set the value of the compile_master_pool_address parameter to the load balancer hostname. If you are using a single compiler, set the compile_master_pool_address value to the compiler's fully qualified domain name (FQDN).
    3. Click Add data and commit changes.
    4. Run Puppet on the compiler, and then on the primary server.
  2. Configure Puppet agents to connect orchestration (PXP) agents to compilers through the load balancer. You can configure these settings in the console or with Hiera.
    1. In the console, click Node groups, and in the PE Infrastructure group, select the PE Agent group.
    2. On the Classes tab, find the puppet_enterprise::profile::agent class and specify parameters:
      Parameter Value
      manage_puppet_conf Specify true to ensure that your setting for server_list is configured in the expected location and persists through Puppet runs.
      pcp_broker_list Specify hostnames for your load balancers. This setting configures PXP agents. Hostnames must include port 8142, for example: ["LOADBALANCER1.EXAMPLE.COM:8142","LOADBALANCER2.EXAMPLE.COM:8142"]
      primary_uris Specify hostnames for your load balancers, for example: ["LOADBALANCER1.EXAMPLE.COM","LOADBALANCER2.EXAMPLE.COM"]

      This setting configures PXP agents and assumes port 8140 unless you specify otherwise, such as: "LOADBALANCER1.EXAMPLE.COM:<PORT>"
      server_list Specify hostnames for your load balancers, for example: ["LOADBALANCER1.EXAMPLE.COM","LOADBALANCER2.EXAMPLE.COM"]

      This setting configures Puppet agents and assumes port 8140 unless you specify otherwise, such as: "LOADBALANCER1.EXAMPLE.COM:<PORT>"
    3. Remove any values set for pcp_broker_ws_uris.
    4. Commit changes.
    5. Run Puppet on the primary server, and then run Puppet on all agents or install new agents.
      This Puppet run configures both Puppet agents and PXP agents to connect to the load balancer.

Convert existing compilers

If you have legacy compilers, you can improve their usability and scalability by adding PuppetDB. In addition to installing the PuppetDB service, converting an existing compiler adds the node to the PE Compiler node group and unpins it from the PE Master node group.

Before you begin
Open port 5432 from compilers to your primary server or, in extra-large installations, your PE-PostgreSQL node.
Important: Contact Support for guidance before converting compilers in multi-region installations. If your primary server and compilers are connected with high-latency links or congested network segments, you might experience better PuppetDB performance with legacy compilers.
On your primary server logged in as root, run: . 
puppet infrastructure run convert_legacy_compiler compiler=<COMPILER_FQDN-1>,<COMPILER_FQDN-2>
Tip: To convert all compilers:
puppet infrastructure run convert_legacy_compiler all=true
What to do next
Run puppet infrastructure tune on your primary server and adjust tuning for compilers as needed.