PuppetDB: Release notes

• Fixed an issue with report garbage collection where a partition would become partially detached and block future garbage collection progress. Garbage collection will now finalize the partition detach operation and remove the table. (GitHub #4013)

• Fixed an issue with report garbage collection where a partition would be detached, but the table was never deleted. Garbage collection will now identify and clean-up these tables. (GitHub #4013)

• Released support and packages for Debian 12 (bookworm)

• Added a database constraint to prevent duplicate catalogs. If your database contains any duplicate catalogs, only the most recent catalog for each certname will be kept.

Austin Blatt and Rob Browning.

• Ship with updated dependencies (clojure, and pgjdbc)

• Ensure producer_timestamp is logged with legacy commands.

Austin Blatt, Cas Donoghue, and Rob Browning.

• Fixed an issue with negated regex queries (!~) on JSON fields (dotted fact paths and resource parameters), which were not matching the full negation of the regex match when the key was missing from some of the JSON maps.

Austin Blatt and Rob Browning.

• Dependencies have been updated

Austin Blatt, Cas Donoghue, Jonathan Newman, Rob Browning, and Steve Axthelm

• Update trapperkeeper-webserver-jetty9 to 4.5.2 to address CVE-2023-44487, CVE-2023-36478, GHSA-58qw-p7qm-5rvh, GHSA-hmr7-m48g-48f6, GHSA-3gh6-v5v9-6v9j

• Update Bouncy Castle FIPS to v1.0.2.4 to resolve CVE-2022-45156 and CVE-2023-33202

• Update jvm-ssl-utils to 3.5.2 to address a stack overflow in certificates with tags.

Austin Blatt, Nick Burgan-Illig, Joshua Partlow, and Rob Browning

• Some PQL queries with numerous or clauses should no longer cause PuppetDB to run out of memory. Previously they could allocate an exorbitant amount of RAM. (GitHub #3874)

Austin Blatt, Nick Burgan-Illig, Jonathan Newman, Eric Newton, Joshua Partlow, Steve Axthelm, and Rob Browning

Austin Blatt, Nick Burgan-Illig, and Rob Browning

• queries with thousands of in array entries would cause performance problems in query compilation. (PDB-3171)

• PuppetDB should no longer crash on reload (SIGHUP) in some cases (e.g. after startup but before processing any commands). (PDB-5215)

Austin Blatt, Jonathan Newman, Joshua Partlow, Rob Browning, and Nick Lewis

• Ordering query results by certain fields (e.g. the trusted facts field) should no longer cause an error. (PDB-5568)

• The org.postgresql/postgresql driver has been updated to version 42.4.3 to address CVE-2022-41946, which is an information exposure vulnerability that could expose database information to a local system user. (PDB-5570)

• The org.ini4j/ini4j library has been upgraded to 0.5.4 to address CVE-2022-41404, a Denial of Service (DoS) vulnerability. (PDB-5571)

Austin Blatt and Rob Browning

• The timing of garbage collection operations can be controlled more selectively. The gc-interval controls the timing of a set of operations, and now the timing of each of those operations can be specified individually. (PDB-5547)

• Queries should no longer be able to block report garbage collection indefinitely. If you do not use the PostgreSQL module and have the recommended, separate [read-database] username, then you may need to make adjustments.

  Specifically, the the normal (write) [database] username must have the right to terminate the [read-database] username's queries. The recommended configuration has been updated to include a suitable grant puppetdb_read to puppetdb. (PDB-5559)

• The coordination of database migrations will now disallow [read-database] user connections as intended. If you do not use the PostgreSQL module, have the recommended, separate [read-database] username, and have enabled migration coordination via a migrator-username then you may need to make adjustments.

  Specifically, the the normal migrator-username must have the ability to terminate the [read-database] username's connections, which the recommended configuration accomplishes by granting the write user's role to the migrator via the grant puppetdb to puppetdb_migrator, allowing the migrator to terminate the read user's connections indirectly via the grant puppetdb_read to puppetdb, also in the recommended configuration. (PDB-5559)

• PuppetDB now drops expired partitions (e.g. reports) more effectively. The changes will cause PostgreSQL to log messages like this: "FATAL: terminating connection due to administrator command". Previously PuppetDB could cause database deadlocks that might indefinitely prevent the expired partitions from being dropped. (PDB-5559)

April Murphy, Arthur Lawson, Austin Blatt, Cas Donoghue, Jonathan Newman, Justin Stoller, Nick Lewis, and Rob Browning

• PuppetDB will now log additional information when processing each command, including the producer timestamp, and when available, a prefix of the fingerprint. (PDB-5524)

Austin Blatt, Jonathan Newman, Justin Stoller, and Rob Browning

• The org.postgresql/postgresql driver has been updated to version 42.4.1 to address CVE-2022-31197, which is an SQL injection risk that according to the CVE report, can only be exploited if an attacker controls the database to the extent that they can adjust relevant tables to have "malicious" column names. (PE-34250)

April Murphy, Jonathan Newman, Rob Browning, and Stel Abrego

• Query logging has been improved when log-queries is set to true. Now queries are logged with their UUID before they are parsed which makes debugging easier when there are PQL parsing issues. (PDB-5482)

• Fixed a bug that was introduced in 6.19.0 and 7.10.1 that caused valid queries using the ~> operator inside of an extract clause to fail. The patch also fixes a UX issue where providing the wrong amount of arguments in a ~> clause would result in a cryptic error. (PE-33977)

• Fixed a bug that was introduced in 6.21.0 and 7.10.1 which caused upgrade failures with PostgreSQL hot standbys. The method of disabling the jit has been changed to avoid the problem. (PDB-5483)

Arjen Zonneveld, Austin Blatt, Jon-Paul Lindquist, Jonathan Newman, Katlin Anderson, Maggie Dreyer, Rob Browning, and Stel Abrego

• PuppetDB should require much less time and memory when parsing some PQL queries, , for example queries including many or clauses like nodes {x or y or ...}. Previously 5000 clauses could not be parsed with an 8GB heap, and much smaller queries still required exorbitant amounts of memory and CPU time. (PDB-5260)

• Ubuntu 20.04, RedHat 8 (FIPS) (Puppet Enterprise only), and SUSE Linux Enterprise 15 are now supported.

Austin Blatt and Rob Browning

• PuppetDB will no longer run a garbage collection on startup. This may substantially reduce the time required before PuppetDB begins accepting commands and queries. (PDB-5422)

• The fact path GC now runs no more than once every 24 hours by default. This should be much less expensive in most cases, in exchange for a potentially slower response to the disappearance of individual fact paths. (PDB-5423)

• PuppetDB will no longer process incoming commands during the initial sync. This may allow the sync to finish more quickly, decreasing startup time (Puppet Enterprise only). (PDB-5386)

• PostgreSQL introduced a query JIT in version 11, and enabled it by default in 12. At the moment, it causes some queries to be dramatically more expensive, and PuppetDB was affected, so it now disables the JIT for all of its queries. (PDB-5452)

Austin Blatt, Rob Browning, and Stel Abrego

This releases contains a fix for CVE-2022-21724 (PDB-5449), a pgjdbc exploit. In order to use this exploit, an attacker must have permission to create JDBC connections via JDBC URL's while the application is running. Since PuppetDB only creates JDBC connections internally from configuration upon startup, this CVE is of very low risk to our users. In order for a PuppetDB user to be successfully attacked, the malicious actor would have to first acquire at least the user privileges of the PuppetDB user. Due to the CVE's high severity rating, we believe it's appropriate to release a fix regardless.

Austin Blatt, Rob Browning, and Stel Abrego

• Added support for Debian 11. (PDB-5390)

• Improved performance of the "deactivate node" command. (PDB-5378)

• Improved performance of the fact-contents endpoint. Testing against a database of 10,000 mocked nodes, there was an observed 84% decrease in time taken to complete a difficult query. This optimization has a known issue with PostgreSQL JIT compilation. (PDB-5259)

• Fixed a bug with HA sync (Puppet Enterprise only) regarding /pdb/query/v4/<entity>/<certname> style queries that caused replicas to falsely report that the sync transferred 0 nodes. (PDB-5381)

• Fixed error handling issues in the command endpoint. Previously, providing a certname that was an empty string or null would cause PuppetDB to crash and prevent prior restarts from exiting maintenance mode. Upon other errors such as missing required parameters, the command endpoint would return a status 500 HTML page or cryptic internal error data. This patch ensures the command endpoint will always return a standard { "error": <description> } JSON response upon any ingestion error and ingestion errors at the command endpoint will not cause PuppetDB to crash. (PDB-5282)

Austin Blatt, Rob Browning, and Stel Abrego

PuppetDB should no longer delete the last day of reports a day earlier than specified by report-ttl in some cases. (PDB-5351)

Austin Blatt, Rob Browning, and Stel Abrego

This release is part of both a Puppet Platform and PE release that resolves CVEs, see Puppet's CVE announcements for more information.

Puppet Agent 6.25.1 and 7.12.1 (PUP-11209) introduced a new catalog resource field in order to resolve CVE-2021-27025. This field was not handled properly by older versions of PuppetDB and will result in catalogs not being stored in PuppetDB. Before upgrading any agents in your installation to 6.25.1 or 7.12.1, you must first upgrade your PuppetDB(s) to this version. (PDB-5338)

Austin Blatt, Bogdan Irimie, Rob Browning, Sebastian Miclea, and Stel Abrego

• If a query with an extract clause contains a misspelled option, the clause is completely ignored resulting in a misleading response body.

  ["from", "reports",
    ["extract", [["function", "count", "certname"]],
      ["null?", "type", false],
      ["groupy_by", "certname"]]]

  will return all the reports because the extract cause will be ignored ( it contains groupy_by instead of group_by). Instead of returning nil for a malformed extract clause (when converting the query to sql plan), try to identify the misspelled part and log an appropriate error message. (PDB-4731)

• When querying for trusted facts on inventory endpoint with a query like:

  inventory[] { trusted.extensions.foo = "bar"}

  instead of facts.trusted.extensions.foo, the index wasn't hit. The change introduced by this ticket ensures that an index is hit if the query is made with just trusted.[fact]. (PDB-4985)

Austin Blatt, Oana Tanasoiu, Rob Browning, and Sebastian Miclea

• TLSv1.3 is enabled by default, it will be preferred over TLSv1.2 when possible. PDB-5255

• Adds a new optional query field origin that allows users initiating a query to identify their query, which will assist with debugging any query-related issues. PDB-5216

• Fixes multiple bug when grouping by a dotted fact path. For example, previously double quotes were required - now the must be omitted. PDB-5214 PDB-4628

• The query optimizer that attempts to drop unneeded joins is now enabled by default, but that can be changed by setting the PDB_QUERY_OPTIMIZE_DROP_UNUSED_JOINS environment variable. (PDB-5131)

• The to_string function should now be allowed in facts and fact-contents query fields. (PDB-5104)

• Queries that include a limit, offset, or order_by, in a subquery that uses from like this:

  ["from", "nodes",
   ["in", "certname",
    ["from", "reports", ["extract", "certname"],
     ["limit", 1]]]]

  should no longer crash with an error that looks like this:

  'type' is not a queryable object for nodes...

  (PDB-5026)

• The delete-reports subcommand now restarts the puppetdb service after deleting reports. (PDB-5142)

Andrei Filipovici, Austin Blatt, Ethan J. Brown, Filipovici-Andrei, Heston Hoffman, Maggie Dreyer, Oana Tanasoiu, Rob Browning, and Sebastian Miclea

• Fixed an issue where someone with the ability to query PuppetDB could arbitrarily write, update, or delete data CVE-2021-27021 PDB-5138

• Significantly reduced the memory usage by the puppetdb terminus to process commands. PDB-5107

• Some command processing operations should require less work and require fewer round trips to the database. PDB-5128

• If the read-only user has database permissions that it does not need, PuppetDB will log errors. PDB-5145

• (PE only) Fixed an issue causing unnecessary factset sync PDB-5021

• Lock timeouts will be parsed correctly now. Previously, if a lock timeout had been set either via the experimental PDB_GC_DAILY_PARTITION_DROP_LOCK_TIMEOUT_MS variable, or other means, PuppetDB might fail to interpret the value correctly, and as a result, fail to prune older data correctly. (PDB-5141)

• All reports can be queried by including type = "any" as a query filter. PDB-4766

• The ssl-setup command (which is also invoked by the PuppetDB package installation scripts) should handle ssl-related filesystem permissions more carefully. Previously it might reset them when it shouldn't have, and/or leave them briefly with incorrect, potentially overly permissive values. PDB-2590

• Added a new query parameter explain that will return the query plan and actual run time statistics for the query. PDB-5055

• Add ability to disable storage of resource events PDB-3635

• Fixed a bug in the ssl-setup command that would insert a duplicate setting into the jetty.ini config. PDB-5084

• PuppetDB will no longer return HTML formatted stack traces from the API endpoint, now only the error message will be returned. The full error can still be found it the logs if needed. PDB-5063

• Trailing characters after a query, which were usually from mismatched ] in an AST query, will no longer be ignored. Instead an error will be returned to alert the user to the potential error in their query. PDB-2488

• Added two new users connection-migrator-username and connection-username in database.ini config file. The new users are used to establish connections to the database when the connection username is different from the database username (this is the case for managed PostgreSQL in Azure) PDB-4934

• A new metric (:concurrent-depth), which counts the number of /cmd API requests that are waiting to write to the disk, was added. PDB-4268

• A new metric (new-fact-time) was added under puppetlabs.puppetdb.storage. This metric measures the time it takes to persist facts for a never before seen certname. PDB-3418

• The performance dashboard is now accessible on the HTTPS port. In PE, if the PuppetDB is using a certificate allowlist, users can authenticate their connection with an rbac token as a URL parameter. PDB-3159

• (PE only) PuppetDB will synchronize with another instance more efficiently now. Previously it would synchronize each entity (factsets, reports, etc.) incrementally, holding open PostgreSQL queries/transactions throughout the entire process, which could take a long time. Those transactions could substantially harm database performance, increase table fragmentation, and if entangled with something like pglogical, increase transient storage requirements (by blocking WAL log reclaimaton). Now instead, the queries should completed up-front, as quickly as possible. PDB-2420

• Changed the index on certname for report table partitions to be an index on (certname, end_time) to improve performance of certain queries from the PE console. PDB-5003

• The /metrics/v2 endpoint is now avaialble for external (non-localhost) connections, but requires authentication. This can be configured in a new configuration file auth.conf. PDB-4811

• The optimize_drop_unused_joins query parameter can now optimize queries that contain a single count function. PDB-4984

• Added a query bulldozer which is spawned during periodic GC when PuppetDB attempts to drop partitioned tables. The bulldozer will cancel any queries blocking the GC process from getting the AccessExclusiveLocks it needs in order to drop a partition. The PDB_GC_QUERY_BULLDOZER_TIMEOUT_MS setting allows users to disable the query-bulldozer if needed. PDB-4948

• Previously an attempt to stop (or restart) PuppetDB might appear to succeed, even though some of its components were actually still running. That's because PuppetDB wasn't actually waiting for some of the internal tasks to finish as had been expected. Now PuppetDB should block during stop or restart until all of the components have actually shut down. This issue is a likely contributor to some cases where PuppetDB appeared to restart/reload successfully, but sync never started working again. PDB-4974

• Various security fixes. (PDB-5000)

• Puppet Enterprise (PE) only: fixed an issue where PuppetDB wouldn't exit maintenance mode if garbage collection was disabled and sync was enabled. (PDB-4975)

• PuppetDB no longer retries queries internally, suppressing some transient connection errors. Instead, it immediately returns an error code. You can restore the previous behavior by setting the PDB_USE_DEPRECATED_QUERY_STREAMING_METHOD environment variable. See the configuration information for further details.

• PuppetDB won't hold an extra database connection open while generating query responses. Previously it would create and hold an extra connection open during the initial phase of the response. You can restore the previous behavior by setting the PDB_USE_DEPRECATED_QUERY_STREAMING_METHOD environment variable. See the configuration information for further details.

• Running PuppetDB with PostgreSQL 9.6 or 10 is no longer supported. Use PostgreSQL 11 or greater instead.