Monitoring current infrastructure state

Expand menu Collapse menu

Node run statuses
Filtering nodes on the Overview page
1. Filter by node run status
2. Filter by node fact
Monitor PE services
1. View the Puppet Services status monitor
2. puppet infrastructure status command

When nodes fetch their configurations from the Puppet master, they send back inventory data and a report of their run. This information is summarized on the Overview page in the console.

The Overview page displays the most recent run status of each of your nodes so you can quickly find issues and diagnose their causes. You can also use this page to gather essential information about your infrastructure at a glance, such as how many nodes your Puppet master is managing, and whether any nodes are unresponsive.

Node run statuses

The Overview page displays the run status of each node following the most recent Puppet run. There are 10 possible run statuses.

Nodes run in enforcement mode

With failures: This node’s last Puppet run failed, or Puppet encountered an error that prevented it from making changes.; The error is usually tied to a particular resource (such as a file) managed by Puppet on the node. The node as a whole might still be functioning normally. Alternatively, the problem might be caused by a situation on the Puppet master, preventing the node's agent from verifying whether the node is compliant.
With corrective changes: During the last Puppet run, Puppet found inconsistencies between the last applied catalog and this node’s configuration, and corrected those inconsistencies to match the catalog.
Note: Corrective change reporting is only available on agent nodes running PE 2016.4 and later. Agents running earlier versions will report all change events as "with intentional changes."
With intentional changes: During the last Puppet run, changes to the catalog were successfully applied to the node.
Unchanged: This node's last Puppet run was successful, and it was fully compliant. No changes were necessary.

Nodes run in no-op mode

Note: No-op mode reporting is only available on agent nodes running PE 2016.4 and later. Agents running earlier versions will report all no-op mode runs as "would be unchanged."

With failures: This node’s last Puppet run in no-op mode failed, or Puppet encountered an error that prevented it from simulating changes.
Would have corrective changes: During the last Puppet run, Puppet found inconsistencies between the last applied catalog and this node’s configuration, and would have corrected those inconsistencies to match the catalog.
Would have intentional changes: During the last Puppet run, catalog changes would have been applied to the node.
Would be unchanged: This node’s last Puppet run was successful, and the node was fully compliant. No changes would have been necessary.

Nodes not reporting

Unresponsive: The node hasn't reported to the Puppet master recently. Something might be wrong. The cutoff for considering a node unresponsive defaults to one hour, and can be configured.; Check the run status table to see the timestamp for the last known Puppet run for the node and an indication of whether its last known run was in no-op mode. Correct the problem to resume Puppet runs on the node.
Have no reports: Although Puppet Server is aware of this node's existence, the node has never submitted a Puppet report for one or more of the following reasons: it's a newly commissioned node; it has never come online; or its copy of Puppet is not configured correctly.
Note: Expired or deactivated nodes are displayed on the Overview page for seven days. To extend the amount of time that you can view or search for these nodes, change the node-ttl setting in PuppetDB. Changing this setting affects resources and exported resources.

Special categories

In addition to reporting the run status of each node, the Overview page provides a secondary count of nodes that fall into special categories.

Intended catalog failed: During the last Puppet run, the intended catalog for this node failed, so Puppet substituted a cached catalog, as per your configuration settings.; This typically occurs when you have compilation errors in your Puppet code. Check the Puppet run's log for details.; This category is shown only if one or more agents fail to retrieve a valid catalog from Puppet Server.
Enforced resources found: During the last Puppet run in no-op mode, one or more resources was enforced, as per your use of the noop => false metaparameter setting.; This category is shown only if enforced resources are present on one or more nodes.

How Puppet determines node run statuses

Puppet uses a hierarchical system to determine a single run status for each node. This system gives higher priority to the activity types most likely to cause problems in your deployment, so you can focus on the nodes and events most in need of attention.

During a Puppet run, several activity types might occur on a single node. A node's run status reflects the activity with the highest alert level, regardless of how many events of each type took place during the run. Failure events receive the highest alert level, and no change events receive the lowest.


Run status	Definitely happened	Might also have happened
	Failure	Corrective change, intentional change, no change
	Corrective change	Intentional change, no change
	Intentional change	No change
	No change

For example, during a Puppet run in enforcement mode, a node with 100 resources receives intentional changes on 30 resources, corrective changes on 10 resources, and no changes on the remaining 60 resources. This node's run status is "with corrective changes."

Node run statuses also prioritize run mode (either enforcement or no-op) over the state of individual resources. This means that a node run in no-op mode is always reported in the Nodes run in no-op column, even if some of its resource changes were enforced. Suppose the no-op flags on a node's resources are all set to false. Changes to the resources are enforced, not simulated. Even so, because it is run in no-op mode, the node's run status is "would have intentional changes."

Filtering nodes on the Overview page

You can filter the list of nodes displayed on the Overview page by run status and by node fact. If you set a run status filter, and also set a node fact filter, the table takes both filters into account, and shows only those nodes matching both filters.

Clicking Remove filter removes all filters currently in effect.

The filters you set are persistent. If you set run status or fact filters on the Overview page, they continue to be applied to the table until they're changed or removed, even if you navigate to other pages in the console or log out. The persistent storage is associated with the browser tab, not your user account, and is cleared when you close the tab.

Important: The filter results count and the fact filter matching nodes counts are cached for two minutes after first retrieval. This reduces the total load on PuppetDB and decreases page load time, especially for fact filters with multiple rows. As a result, the displayed counts might be up to two minutes out of date.

Filter by node run status

The status counts section at the top of the Overview page shows a summary of the number of nodes with each run status as of the last Puppet run. Filter nodes by run status to quickly focus on nodes with failures or change events.

In the status counts section, select a run status (such as with corrective changes or have no reports) or a run status category (such as Nodes run in no-op).

Filter by node fact

You can create a highly specific list of nodes for further investigation by using the fact filter tool.

For example, you can check that nodes you've updated have successfully changed, or find out the operating systems or IP addresses of a set of failed nodes to better understand the failure. You might also filter by facts to fulfill an auditor's request for information, such as the number of nodes running a particular version of software.

Click Filter by fact value. In the Fact field, select one of the available facts. An empty fact field is not allowed.

Tip: To see the facts and values reported by a node on its most recent run, click the node name in the Run status table, then select the node’s Facts tab.

Select an Operator:


Operator	Meaning	Notes
=	is
!=	is not
~	matches a regular expression (regex)	Select this operator to use wildcards and other regular expressions if you want to find matching facts without having to specify the exact value.
!~	does not match a regular expression (regex)
>	greater than	Can be used only with facts that have a numeric value.
>=	greater than or equal to	Can be used only with facts that have a numeric value.
<	less than	Can be used only with facts that have a numeric value.
<=	less than or equal to	Can be used only with facts that have a numeric value.

In the Value field, enter a value. Strings are case-sensitive, so make sure you use the correct case.
The filter will display an error if you use an invalid string operator (for example, selecting a numeric value operator such as >= and entering a non-numeric string such as pilsen as the value) or enter an invalid regular expression.
Note: If you enter an invalid or empty value in the Value field, PE takes the following action in order to avoid a filter error:
- Invalid or empty Boolean facts are processed as false, and results are retrieved accordingly.
- Invalid or empty numeric facts are processed as 0, and results are retrieved accordingly.
- Invalid or incomplete regular expressions invalidate the filter, and no results are retrieved.
Click Add.
As needed, repeat these steps to add additional filters. If filtering by more than one node fact, specify either Nodes must match all rules or Nodes may match any rule.

Monitor PE services

Puppet Enterprise includes tools for monitoring the status of core services including the activity, classifier, and RBAC services, Puppet Server, and PuppetDB. You can monitor these services on the command line, and from within the console.

View the Puppet Services status monitor

The Puppet Services status monitor provides a visual overview of the current state of key services, and can be used to quickly determine whether an unresponsive or restarting service is causing an issue with your deployment.

In the console, click Overview.
Click Puppet Services status to open the monitor.

A checkmark appears next to Puppet Services status if all applicable services are accepting requests. In the event that no data is available, a question mark appears next to the link. If one or more services is restarting or not accepting requests, a warning icon appears.

`puppet infrastructure status` command

The puppet infrastructure status command displays errors and alerts from PE services, including the activity, classifier, and RBAC services, Puppet Server, and PuppetDB.

If high availability is enabled in your environment, the command reports separately on the primary master, primary master replica, and compile masters.