Introducing Puppet query language (PQL)

After hearing for years about the inscrutability of our existing AST-based query language, the PuppetDB team is proud to announce the release of our new query language PQL, first released as experimental in PuppetDB 4.0 and Puppet Enterprise 2016.1.

PQL is a more human-friendly way to query PuppetDB. Coupled with a new Puppet function for querying PuppetDB from within Puppet manifests, PQL is intended to both simplify querying from the CLI and facilitate more interesting integrations of PuppetDB and traditional Puppet applications.

I'd like to highlight some benefits of PQL over AST and encourage you to try it out. For additional information, see our PQL tutorial and the PQL reference guide.

PQL is succinct

PQL can more concisely (and we hope, more readably) express almost any query in the AST language. The following examples can be executed against the new and experimental /pdb/query/v4 endpoint or by using the puppet query command available in the puppet-client-tools package. The PuppetDB client tool will be included in PE 2016.3.

  • Retrieve counts of nodes by Puppet version:

    AST:

    ["from", "facts",
      ["extract", ["value", ["function", "count"]],
      ["=", "name", "puppetversion"], ["group_by", "value"]]]

    PQL:

    facts[value, count()] {name = "puppetversion" group by value}
  • Retrieve all events from the latest reports with type “User”, title "logstash", and message "current_value present, should be absent (noop)":

    AST:

    ["from", "events",
      ["and", ["=", "resource_type", "User"],
              ["~", "resource_title", "logstash"],
              ["=", "latest_report?", true],
              ["=", "message", "current_value present, should be absent (noop)"]]]'

    PQL:

    events{resource_type = "User"
                and resource_title = "logstash"
                and message = "current_value present, should be absent (noop)"}
  • Retrieve a list of certnames for which the latest report status is "failed":

    AST:

    ["from", "reports",
      ["extract", "certname",
        ["and", ["=", "latest_report?", true],
                ["=", "status", "failed"]]]]

    PQL:

    reports[certname]{latest_report? = true and status = "failed"}

The only AST features that are not currently covered by PQL are the experimental event-counts and aggregate-event-counts endpoints, and the distinct_resources query parameter.

PQL is composable (like AST)

One of the principal design goals for PQL was composability of queries. Users must be able to design a complex query out of small, independently working and testable components. In the AST language, you can take a working query against the reports endpoint:

["=", "status", "failed"]

and combine it with a second query against the facts endpoint:

["=", "name", "operatingsystem"]

so that the only facts returned correspond to nodes represented in nodes with reports matched by the first query:

["and", ["=", "name", "operatingsystem"],
        ["in" "certname",
          ["extract", "certname",
            ["select_reports", ["=", "status", "failed"]]]]]

Restricting the output of the query above to only fact values is a matter of wrapping the same query with an extract clause:

["extract", "value",
  ["and", ["=", "name", "operatingsystem"],
          ["in" "certname"
            ["extract", "certname"
              ["select_reports", ["=", "status", "failed"]]]]

This kind of composability comes fairly naturally to AST-based languages, and we have carried the same benefit to PQL. To illustrate, the corresponding PQL queries corresponding to the examples above would be

reports{status = "failed"}

facts{name = "operatingsystem"}

facts{name = "operatingsystem" and certname in reports[certname]{status = "failed"}}

facts[value]{name = "operatingsystem" and
                     certname in reports[certname]{status = "failed"}}

You can call PQL from Puppet

Alongside PQL, we've introduced a function defined in the PuppetDB terminus package that queries PuppetDB from within a Puppet manifest. This function can be used to similar effect as in Erik Dalén’s puppet-puppetdbquery module, and in some cases, Puppet's exported resources. The function is called puppetdb_query, and using it is as simple as this:

$last_node_query = 'nodes{order by report_timestamp desc limit 1}'
$latest_node = puppetdb_query($last_node_query)[0]['certname']
Notify {"hello PQL":
    message => "My last report was from $latest_node.",
}

PQL is expanding

While PQL currently applies only to PuppetDB, the language will ultimately find its way into additional Puppet products, including the node manager and application orchestrator. The AST language is not going away any time soon (under-the-hood PQL still compiles to the AST language), but over time, PQL will become a primary interface between a number of our services.

Finally, PQL is under active development as it pertains to PuppetDB. While we have several language features for consideration on our short-term roadmap, including easier descendence into structured facts and resource parameters, more succinct querying for resource type/title combinations, and broader coverage of common uses by implicit subqueries, there is no substitute for real user input. If you're interested in discussing any aspect of PQL, what works, what doesn't, what's confusing, or new features you'd like to see, please reach out to us on the puppet-users mailing list, in #puppet or #puppet-dev on freenode, or via our bug tracker.

Wyatt Alt is a software engineer at Puppet.

Learn more

Puppet sites use proprietary and third-party cookies. By using our sites, you agree to our cookie policy.