Static catalogs

A static catalog includes additional metadata that identifies the desired state of a node’s file resources that have source attributes pointing to puppet:/// locations. This metadata can refer to a specific version of the file, rather than the latest version, and can confirm that the agent is applying the appropriate version of the file resource for the catalog. As most of the metadata is provided in the catalog, Puppet agents make fewer requests to the primary server.

When to use static catalogs

When a primary server compiles a non-static catalog, the catalog does not specify a particular version of its file resources. When the agent applies the catalog, it always retrieves the latest version of that file resource, or uses a previously retrieved version if it matches the latest version’s contents. Note that this potential problem affects file resources that use the source attribute. File resources that use the content attribute are not affected, and their behavior does not change in static catalogs.

When a Puppet manifest depends on a file whose contents change more frequently than the Puppet agent receives new catalogs — for example, if the agent is caching catalogs or cannot reach a primary server over the network — a node might apply a version of the referenced file that does not match the instructions in the catalog.

As a result, the agent’s Puppet runs might produce different results each time the agent applies the same catalog. This can cause problems because Puppet expects a catalog to produce the same results each time it is applied, regardless of any code or file content updates on the primary server.

Additionally, each time a Puppet agent applies a non-static cached catalog that contains file resources sourced from puppet:/// locations, the agent requests file metadata from the primary server, even though nothing changed in the cached catalog. This causes the primary server to perform unnecessary resource-intensive checksum calculations for each such file resource.

Static catalogs avoid these problems by including metadata that refers to a specific version of the resource’s file. This prevents a newer version from being incorrectly applied, and avoids having the agent request the metadata on each Puppet run.

This type of catalog is called static because it contains all of the information that an agent needs to determine whether the node’s configuration matches the instructions and static state of file resources at the point in time when the catalog was compiled.

Static catalog features

A static catalog includes file metadata in its own section and associates it with the catalog's file resources.

In a non-static catalog, the Puppet agent requests metadata and content for the file from Puppet Server. The server generates a checksum for the file and provides that file as it currently exists on the primary server.

With static catalogs enabled, the primary server generates metadata for each file resource sourced from a puppet:/// location and adds it to the static catalog, and adds a code_id to the catalog that associates such file resources with the version of their files as they exist at compilation time. For example: 

file { '/etc/application/config.conf':
  ensure => file,
  source => 'puppet:///modules/module_name/config.conf'
}

Inlined metadata is part of a FileMetadata object in the static catalog that is divided into two new sections: metadata for metadata associated with individual files, and recursive_metadata for metadata associated with many files. To use the appropriate version of the file content for the catalog, Puppet Server also adds a code_id parameter to the catalog. The value of code_id is a unique string that Puppet Server uses to retrieve the version of file resources in an environment at the time when the catalog was compiled.

When applying a file resource from a static catalog, an agent first checks the catalog for that file’s metadata. If it finds some, Puppet uses the metadata to call the static_file_content API endpoint on Puppet Server and retrieves the file’s contents, also called the code_content. If the catalog does not contain metadata for the resource, Puppet requests the file resource’s metadata from the primary server, compares it to the local file if it exists, and requests the resource’s file from the primary server in its current state — if the local file doesn’t exist or differs from the primary server's version.

Configuring code_id and the static_file_content endpoint

When requesting the file’s content via the static catalog’s metadata, the Puppet agent passes the file’s path, the catalog’s code_id, and the requested environment to Puppet Server's static_file_content API endpoint. The endpoint returns the appropriate version of the file’s contents as the code_content.

If static catalogs are enabled but Puppet Server's static catalog settings are not configured, the code_id parameter defaults to a null value and the agent uses the file_content API endpoint, which always returns the latest content. To populate the code_id with a more useful identifier and have the agent use the static_file_content endpoint to retrieve a specific version of the file’s content, you must specify scripts or commands that provide Puppet with the appropriate results.

Puppet Server locates these commands using the code-id-command and code-content-command settings in the puppetserver.conf file. Puppet Server runs the code-id-command each time it compiles a static catalog, and it runs the code-content-command each time an agent requests file contents from the static_file_content endpoint.

The Puppet Server process must be able to execute these scripts. Puppet Server also validates their output and checks their exit codes. Environment names can contain only alphanumeric characters and underscores (_). The code_id can contain only alphanumeric characters, dashes (-), underscores (_), semicolons (;), and colons (:). If either command returns a non-zero exit code, Puppet Server logs an error and returns the error message and a 500 response code to the API request.

Puppet Server validates the standard output of each of these scripts, and if the output’s acceptable, it adds the results to the catalog as their respective parameters’ values. You can use any versioning or synchronization tools, as long as you write scripts that produce a valid string for the code_id and code content using the catalog’s code_id and file’s environment.

The following examples demonstrate how Puppet Server passes arguments to the code-id-command and code-content-command scripts and how Puppet Server uses the results to return a specific version of a file resource.

For files in an environment managed by Git, you would use something like the following code-id-command script, with the environment name passed in as the first command-line argument:
#!/bin/bash
set -e
if [[ -z "$1" ]]; then
  echo Expected an environment >&2
  exit 1
fi
cd /etc/puppetlabs/code/environments/"$1" && git rev-parse HEAD

As long as the script’s exit code is zero, Puppet Server uses the script’s standard output as the catalog’s code_id.

With a code-content-command script, Puppet Server passes the environment name as the first command-line argument, the code_id as the second argument, and the path to the file resource from its content_uri as the third argument:
#!/bin/bash
set -e
if [[ $# < 3 ]]; then
  echo Expected environment, code-id, file-path >&2
  exit 1
fi
cd /etc/puppetlabs/code/environments/"$1" && git show "$2":"$3"

The script’s standard output becomes the file’s code_content, provided the script returns a non-zero exit code.

Enabling or disabling static catalogs

The global static_catalogs setting is enabled by default. However, the default configuration does not include the code-id-command and code-content-command scripts or settings needed to produce static catalogs, and even when configured to produce static catalogs Puppet Server does not inline metadata for all types of file resources.

Note that Puppet Server does not produce static catalogs for an agent under the following circumstances:

  • If the static_catalogs setting is false in:
    • The Puppet Server’s puppet.conf file.
    • The environment.conf file for the environment under which the agent is requesting a catalog.
    • The agent’s puppet.conf file.
  • If the Server’s code-id-command and code-content-command settings and scripts are not configured, or if the code-id-command returns an empty string.

Additionally, Puppet Server only inlines metadata for file resources if all of the following conditions are true:

  • It contains a source parameter with a Puppet URI, such as source => 'puppet:///path/to/file'.
  • It contains a source parameter that uses the built-in modules mount point.
  • The file it sources is inside the following glob, relative to the environment’s root directory: */*/files/**. For example, Puppet Server inlines metadata into static catalogs for file resources sourcing module files located by default in /etc/puppetlabs/code/environments/<ENVIRONMENT>/modules/<MODULE NAME>/files/**.