PuppetDB will be a critical component of your Puppet deployment, as agent nodes will be unable to request catalogs if it goes down. Therefore, you should make sure it can handle your site’s load and is resilient against failures.
When scaling any service, there are several possible performance and reliability bottlenecks. These can be dealt with in turn as they become problems.
PuppetDB will be limited by the performance of your PostgreSQL server. You can increase performance by making sure your database server has an extremely fast disk, plenty of RAM, a fast processor, and a fast network connection to your PuppetDB server. You may also need to look into database clustering and load balancing.
It’s also possible that the default PostgreSQL configuration on your system will be very conservative. If so, the PgTune tool can suggest settings that may be more appropriate for the relevant host.
Database administration is beyond the scope of this guide, but the following links may be helpful:
PuppetDB is limited by the amount of memory available to it, which is set in the init script’s config file. If PuppetDB runs out of memory, it will start logging
OutOfMemoryError exceptions and delaying command processing. Unlike many of the bottlenecks listed here, this one is fairly binary: PuppetDB either has enough memory to function under its load, or it doesn’t. The exact amount needed will depend on the number of nodes, the similarity of the nodes, the complexity of each node’s catalog, and how often the nodes check in.
The more frequently your Puppet nodes check in, the heavier the load on your PuppetDB server.
You can reduce the need for higher performance by changing the
runinterval setting in every Puppet node’s puppet.conf file. (Or, if running Puppet agent from cron, by changing the frequency of the cron task.)
The frequency with which nodes should check in will depend on your site’s policies and expectations — this is as much a cultural decision as it is a technical one. A possible compromise is to use a wider default check-in interval, but implement MCollective’s
puppetd plugin to trigger immediate runs when needed.
PuppetDB can take advantage of multiple CPU cores to handle the commands in its queue. Each core can run a worker thread. By default, PuppetDB will use half of the cores in its machine.
You can increase performance by running PuppetDB on a machine with many CPU cores and then tuning the number of worker threads:
Although a single PuppetDB and PostgreSQL server probably can handle all of the load at the site, you may want to run multiple servers for the sake of resilience and redundancy. To configure high-availability PuppetDB, you should:
PuppetDB uses its own embedded SSL processing, which is usually not a performance problem. However, truly large deployments will be able to squeeze out more performance by terminating SSL with Apache or NGINX instead. If you are using multiple PuppetDB servers behind a reverse proxy, we recommend terminating SSL at the proxy server.
Instructions for configuring external SSL termination are currently beyond the scope of this guide. However, we expect that if your site is big enough for this to be necessary, you have probably done it with several other services before.