Published on 3 July 2019 by

Working as a support engineer at Puppet can present a unique perspective on the issues encountered by users and an opportunity to solve or prevent them from occurring. In this instance, I noticed that there were a number of Puppet Enterprise customers who were surprised to find themselves with an expired Certificate Authority and a nonfunctional Puppet infrastructure, as well as a less than stellar experience in resolving the issue. This led me to develop a module for use with Bolt to detect and more easily solve this.

If you have been operating a Puppet Enterprise installation for several years, install and use the check_ca_expiry task today!

Background

The Certificate Authority (CA) is the part of Puppet Server that handles signing and revoking agent certificates. It includes a signed certificate that is issued to every node to prove that it has been authenticated by the CA. When this certificate expires, the CA will no longer trust agents, which effectively renders Puppet inoperable. Fortunately, there is a way to generate a new certificate using the existing private key, which essentially renews the existing certificate.

The old way

Our previous recommendation was to use the certregen module, but this is now deprecated with the introduction of the puppetserver ca command in Puppet 6. Because this heavily relied on an SSH wrapper, it wasn't ideal for Windows nodes and could only be run on the master, meaning you needed ssh access from your master to all agents.

Enter Bolt

The use case for Bolt in this situation was a no-brainer. Extending the CA is a one-time action that needs to be agentless, as the services required to communicate with agents will be nonfunctional if the CA has expired. It also supports both ssh and WinRM transport protocols and can be performed from any workstation with the appropriate level of access. Thus was born the ca_extend module.

Demonstration

First, let’s check the expiration date of our CA cert.

$ bolt task run ca_extend::check_ca_expiry --nodes pe-201815-master,pe-201815-agent,pe-201815-compile --run-as root
Finished on pe-201815-master:

    "status": "valid",
    "expiry date": "May 21 20:19:55 2024 GMT"

Finished on pe-201815-compile:

    "status": "valid",
    "expiry date": "May 21 20:19:55 2024 GMT"

Finished on pe-201815-agent:

    "status": "valid",
    "expiry date": "May 21 20:19:55 2024 GMT"

Let’s pretend that date has already passed or is right around the corner. We can use the Plan to extend the CA cert and configure the master, compilers, and infrastructure nodes such as separate PuppetDB nodes to use it. Include any separate infrastructure nodes in the compile_masters parameter.

$ bolt plan run ca_extend::extend_ca_cert master=pe-201815-master compile_masters=pe-201815-compile --run-as root
Starting: plan ca_extend::extend_ca_cert
Starting: command 'echo "test" | base64 -w 0 - &>/dev/null' on localhost
Finished: command 'echo "test" | base64 -w 0 - &>/dev/null' with 0 failures in 0.0 sec
INFO: Stopping puppet and pe-puppetserver services on pe-201815-master
Starting: task service on pe-201815-master
Finished: task service with 0 failures in 0.85 sec
Starting: task service on pe-201815-master
Finished: task service with 0 failures in 1.95 sec
INFO: Extending certificate on master pe-201815-master
Starting: task ca_extend::extend_ca_cert on pe-201815-master
Finished: task ca_extend::extend_ca_cert with 0 failures in 2.92 sec
INFO: Configuring master pe-201815-master to use new certificate
Starting: task ca_extend::configure_master on pe-201815-master
Finished: task ca_extend::configure_master with 0 failures in 95.72 sec
Starting: task service on pe-201815-master
Finished: task service with 0 failures in 1.64 sec
INFO: Configuring compile master(s) pe-201815-compile to use new certificate
Starting: file upload from /tmp/tmp.CuWtz3dmfx to /etc/puppetlabs/puppet/ssl/certs/ca.pem on pe-201815-compile
Finished: file upload from /tmp/tmp.CuWtz3dmfx to /etc/puppetlabs/puppet/ssl/certs/ca.pem with 0 failures in 0.59 sec
Starting: task run_agent on pe-201815-compile
Finished: task run_agent with 0 failures in 44.34 sec
INFO: CA cert decoded and stored at /tmp/tmp.CuWtz3dmfx
INFO: Run plan 'ca_extend::upload_ca_cert' to distribute to agents
Finished: plan ca_extend::extend_ca_cert in 148.06 sec

The new cert is dumped to /tmp/ on our machine, which we can ship off to agents with another plan. This plan will detect whether the agent is *nix or Windows and upload the cert to the appropriate directory.

$ bolt plan run ca_extend::upload_ca_cert cert=/tmp/tmp.CuWtz3dmfx --nodes pe-201815-agent --run-as root
Starting: plan ca_extend::upload_ca_cert
Starting: plan ca_extend::get_agent_facts
Starting: install puppet and gather facts on pe-201815-agent
Finished: install puppet and gather facts with 0 failures in 10.06 sec
Finished: plan ca_extend::get_agent_facts in 10.06 sec
Starting: plan facts
Starting: task facts on pe-201815-agent
Finished: task facts with 0 failures in 4.47 sec
Finished: plan facts in 4.48 sec
Starting: file upload from /tmp/tmp.CuWtz3dmfx to /etc/puppetlabs/puppet/ssl/certs/ca.pem on pe-201815-agent
Finished: file upload from /tmp/tmp.CuWtz3dmfx to /etc/puppetlabs/puppet/ssl/certs/ca.pem with 0 failures in 0.64 sec
Finished: plan ca_extend::upload_ca_cert in 15.21 sec
{
  "success": {
    "pe-201815-agent": {
      "_output": "Uploaded '/tmp/tmp.CuWtz3dmfx to 'pe-201815-agent:/etc/puppetlabs/puppet/ssl/certs/ca.pem'"


}

It’s that simple. Try it yourself! Go check the expiration date of your CA cert today.

Adrian Parreiras Horta is a support engineer at Puppet.

Learn more

Share via:
Tagged:
The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.