Managing GitHub Protocol Changes' Impact on Your Puppet Code Deployments
Get an overview of GitHub protocol changes and how it impacts Puppet.
Table of Contents:
What Are GitHub Protocol Changes?
In March 2022, Github underwent permanent changes to improve Git protocol security. Those included removing insecure DSA keys and legacy ciphers, updating requirements on RSA keys, and adding more secure key signature algorithms ECDSA and ED25519.
Many users of Puppet Code Manager and r10k were affected by this change. Code Manager and r10k both use libgit2 version 1.9.0, which doesn't yet support the upgraded security of RSA with SHA-2. That means affected Puppet users may need to update their authentication methods before then to ensure proper code deployment.
Resolving 2022 GitHub Protocol Issues in Puppet
Given that Puppet previously advised in managing environments with a control repository, it is likely many users will have already created RSA 256 keys as recommended. Any affected public keys which start sha-rsa will stop working from March 15, at which point running puppet code deploy or `r10k deploy environment will result in failures to deploy and an error like this:
$ Found 1 environments. $[ $ { $ "environment": "production", $ "error": { $ "details": { $ "corrected-name": "production" $ }, $ "kind": "puppetlabs.code-manager/deploy-failure", $ "msg": "Errors while deploying environment 'production' (exit code: 1):\nERROR\t -> Unable to determine current branches for Git source 'puppet' (/etc/puppetlabs/code-staging/environments)\nOriginal exception:\nERROR: You're using an RSA key with SHA-1, which is no longer allowed. Please use a newer client or a different key type.\nPlease see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.\n\n at /opt/puppetlabs/server/data/code-manager/worker-caches/deploy-pool-2/git@github.com-davidsandilands-control-repo.git\n" $ }, $ "id": 15, $ "status": "failed" $ } $]
You will need to recreate the affected key and add it as a deployment key in GitHub.
For Open Source Puppet (OSP) r10k, you will create new keys:
ssh-keygen -t ed25519 -P '' -f /root/.ssh/id_ed25519
Then you will update the private key setting at /etc/puppetlabs/r10k/r10k.yaml
If you are using Puppet Enterprise on the primary Puppet server with Code Manager, then you should create the keys by running:
# ssh-keygen -t ed25519 -P '' -f /etc/puppetlabs/puppetserver/ssh/id-control_repo.ed25519
Then as per Configure Code Manager, you should update the private key setting either in the web console or your Hiera data.
In order to ensure correct permissions are applied to the keys if you are using Puppet Enterprise, run:
# puppet infrastructure configure
If you are using a Replica server on Puppet Enterprise, you should copy to your Replica server both the private key /etc/puppetlabs/puppetserver/ssh/id-control_repo.ed25519 and the public key /etc/puppetlabs/puppetserver/ssh/id-control_repo.ed25519.pub.
Follow the process for managing deploy keys on GitHub to add the content as a read-only deployment key to your control repo on the public key /etc/puppetlabs/puppetserver/ssh/id-control_repo.ed25519.pub or /root/.ssh/id_ed25519.pub.
Libssh2 support for RSA with SHA-2 was added by the Libssh2 project to the Libssh2 library on January 6 and this update will be included in a future release of Puppet Enterprise in order to allow use of RSA 2 SSH keys again.
The good news is that GitHub’s moves to stronger ciphers and keys will ensure the safety and security of code and deployment, and is an excellent opportunity to increase the security of your code and infrastructure.
