AST query language

Works with: strings, numbers, timestamps, Booleans, arrays, multi, path.

Matches if: the field's actual value is exactly the same as the provided value.

• Most fields are strings.

• Some fields are Booleans.

• Arrays match if any one of their elements matches.

• Path matches are a special kind of array, and must be exactly matched with this operator.

Works with: numbers, timestamps, multi.

Matches if: the field is greater than the provided value.

Works with: numbers, timestamps, multi.

Matches if: the field is less than the provided value.

Works with: numbers, timestamps, multi.

Matches if: the field is greater than or equal to the provided value.

Works with: numbers, timestamps, multi.

Matches if: the field is less than or equal to the provided value.

Works with: strings, multi.

Matches if: the field's actual value matches the provided regular expression. The provided value must be a regular expression represented as a JSON string:

• The regexp must not be surrounded by the slash characters (/rexegp/) that delimit regexps in many languages.

• Every backslash character must be escaped with an additional backslash. Thus, a sequence like \d would be represented as \d, and a literal backslash (represented in a regexp as a double-backslash \) would be represented as a quadruple-backslash (\\).

The following example would match if the certname field's actual value resembled something like www03.example.com:

["~", "certname", "www\\d+\\.example\\.com"]

Note: Regular expression matching is performed by the database backend, so the available regexp features are determined by PostgreSQL. For best results, use the simplest and most common features that can accomplish your task.

Works with: paths.

Matches if: each array element, which must be a PostgreSQL regular expression or an integer, matches each element of the path. Integers only match array indexes, regular expressions that only contain integer digits like "123" do not match array indexes, and all other regular expressions, including something like "[12]3", match both array indexes and map keys.

The following example would match any network interface names starting with "eth":

["~>", "path", ["networking", "eth.*", "macaddress"]]

If you want to match any index for an array path element, you can use regular expressions, as the element acts like a string:

["~>", "path", [<array_fact>, ".*"]]

Limitations: with the current implementation an anchored expression like "^sda.*" may never match an array element. Currently those expressions will match for queries against the fact-contents, but for now, that should not be considered reliable across PuppetDB upgrades.

Works with: fields that may be null.

Matches if: the field's value is null (when second argument is true) or the field is not null, or has a real value (when second argument is false).

The following example would return events that do not have an associated line number:

["null?", "line", true]

Similarly, the below query would return events that do have a specified line number:

["null?", "line", false]

Matches if: all of its arguments would match. Accepts any number of query strings as its arguments.

Matches if: at least one of its arguments would match. Accepts any number of query strings as its arguments.

Matches if: its argument would not match. Accepts a single query string as its argument.

To reduce the keypairs returned for each result in the response, you can use extract:

["extract", ["hash", "certname", "transaction_uuid"],
  ["=", "certname", "foo.com"]]

When only extracting a single column, the [] are optional:

["extract", "transaction_uuid",
  ["=", "certname", "foo.com"]]

When applying an aggregate function over a group_by clause, an extract statement takes the form:

["extract", [["function", "count"], "status"],
  ["=", "certname", "foo.com"],
  ["group_by", "status"]]

Extract can also be used with a standalone function application:

["extract", [["function", "count"]], ["~", "certname", ".\*.com"]]

or

["extract", [["function", "count"]]]

The function operator is used to call a function on the result of a subquery. Supported functions are described below.

The group_by operator must be applied as the last argument of an extract, and takes one or more column names as arguments. For instance, to get event status counts for active certname by status, you can query the events endpoint with:

["extract", [["function", "count"], "status", "certname"],
  ["group_by", "status", "certname"]]

To get the average uptime for your nodes:

["extract", [["function", "avg", "value"]], ["=", "name", "uptime_seconds"]]

Dot notation is also supported for extracting a subtree of JSON fields. For example you can query the inventory endpoint with

["extract", ["trusted.certname", "facts.system_uptime"]]

To get a response with only the elements you've asked for

{
    "trusted.certname": "mbp.local",
    "facts.system_uptime.uptime": {
        "days" : 0,
        "hours" : 1,
        "uptime" : "1:52 hours",
        "seconds" : 6733
    }
}

A dotted field, which repseents a path into a JSON tree is made up of components separated by dots (.), for example facts.kernel. Any path component can be double-quoted, for example facts."x.y".z, in which case the name will include all of the characters after the first double-quote, and before the next double-quote that is itself not preceded by a backslash and is followed by either a dot, or the end of the field. So the previous example facts."x.y".z represents the three components, facts, x.y, and z. In AST queries, any double-quotes will have to be properly JSON escaped. So in an extract the path x."y.z" becomes [extract "x."y.z"", ...].

There is currently no way to represent a field component that contains a dot and ends in a backslash. For example, a fact named x.y\ must be quoted, given the dot, but as just mentioned, quoted fields cannot end in a backslash.

Note: the match() operator described here is deprecated and is likely to be retired or altered in a backward-incompatible way in a future release.

In some cases (e.g. inventory endpoint) dotted fields can also contain a match() component, for example facts.partitions.match("sd.*") The match pattern must be a PostgreSQL regular expression, and must begin with match, open paren, double quote, and it will end at the next double quote, close paren that is not preceded by a backslash and is followed by either a dot, or the end of the field. The regex then, has essentially the same syntax as a double quoted field. And similarly, there is currently no way to specify a match regular expression that ends in a backslash.

With the current implementation, the match() component's behavior is not well defined, likley to be surprising, and likely to change in the future, so we recommend avoiding it for now, but please do contact us if you are currently using it, or would like to use an operator with better semantics, so we can incorporate that information into future plans.

As an example of the potentially surprising behavior, the appearance of any match() operator in a dotted field can cause the entire field, not just the match() segment, to be handled as a regular expression in an awkward manner.

The from operator allows you to choose the entity that you want to query and provide optional query and paging clauses to filter those results. This operator can be used at the top-level context of a query:

["from", "nodes", ["=", "certname", "myserver"]]

The from operator can also be used in a subquery for setting the context when using the in operator.

When querying a particular endpoint, such as /pdb/query/v4/nodes, the endpoint provides the context for the query. Querying the root endpoint requires specifying a context explicitly.

Implicit queries work like most operators, and simply require you to specify the related entity and the query to use:

["subquery", "<ENTITY>", <SUBQUERY STATEMENT>]

The <ENTITY> is the particular entity you are subquerying on, however not all entities are implicitly relatable to all other entities, as not every relationship makes sense. Consult the documentation for the chosen <ENTITY> for details on what implicit relationships are supported.

In PuppetDB, we keep a map of how different entities relate to each other, and therefore no data beyond the entity is needed in this case. This is different from explicit subqueries, where you must specify how two entities are related. Implicit subqueries can be used to join any two entities that have a certname field. Additional relationships are described in the endpoint-specific documentation as applicable.

While implicit subqueries can make your syntax succinct, not all relationships are mapped internally. For these more advanced subqueries, you need to specify exactly the fields that a subquery should join on. This is where an explicit subquery can be useful.

Explicit subqueries are unlike the other operators listed above. They always appear together in one of the following forms:

["in", ["<FIELDS>"], ["extract", ["<FIELDS>"], <SUBQUERY STATEMENT>] ]

The second new methodology uses from to set the context, and now looks like this:

["in", ["<FIELDS>"], ["from", <ENTITY>, ["extract", ["<FIELDS>"], <SUBQUERY>] ] ]

That is:

• The in operator results in a complete query string. The extract operator and the subqueries do not.

• An in statement must contain one or more fields and an extract statement.

• An extract statement must contain one or more fields and a subquery statement.

These statements work together as follows (working "outward" and starting with the subquery):

• The subquery collects a group of PuppetDB objects (specifically, a group of resources, facts, fact-contents, or nodes). Each of these objects has many fields.

• The extract statement collects the value of one or more fields across every object returned by the subquery.

• The in statement matches if its field values are present in the list returned by the extract statement.

The complete in statement described in the table above would match any object that shares a certname with a node that has Class[Apache]. This could be combined with a Boolean operator to get a specific fact from every node that matches the in statement.