Troubleshooting Hiera

NOTE: This article is targeted at versions of Puppet that use Hiera 3 or higher — generally Puppet 4, or Puppet Enterprise 2015.2 and later versions. Some of this information will also apply to older versions.

This blog post follows from my prior post, Hiera hierarchies and the custom facts everyone needs. In this new post, I hope to provide a basis for understanding how Hiera works under the hood, and the steps to troubleshoot when Hiera doesn't work as you might have anticipated.

Background

In my last article, I recommended a hierarchy along these lines:

:hierarchy:

  - nodes/%{::trusted.certname}
  - team/%{::team}
  - application/%{::application}
  - datacenter/%{::datacenter}
  - common

Let me explain what this actually means. Imagine this hypothetical collection of Hiera data:

[root@master hieradata]# tree
.
├── application
│   ├── commercesite.yaml
│   └── mobilesite.yaml
├── common.yaml
├── datacenter
│   ├── infrastructure.yaml
│   ├── portland.yaml
│   └── sydney.yaml
├── nodes
│   ├── centos6a.pdx.puppet.vm.yaml
│   ├── centos6b.syd.puppet.vm.yaml
│   ├── master.inf.puppet.vm.yaml
│   └── server2012r2b.syd.puppet.vm.yaml
└── team
    ├── camelot.yaml
    └── developers.yaml

Now imagine a node with these facts (obviously this is simplified):

[root@centos6a ~]# facter -p
application => commercesite
datacenter => portland
fqdn => centos6a.pdx.puppet.vm
team => developers

The resulting set of yaml files that will be loaded by Hiera will be:

centos6a.pdx.puppet.vm.yaml
developers.yaml
commercesite.yaml
portland.yaml
common.yaml

Anytime there is an entry in Puppet code like this:

hiera(mypasswd)

The files will be evaluated looking for an entry like this:

mypasswd: hellokitty4!

The files will be evaluated top to bottom for a key value pair for mypasswd. If multiple files exist that contain the same key, as in the example below, the first entry discovered (the one nearest the top) will take precedence.

commercesite.yaml

mypasswd: hellokitty4!

common.yaml

mypasswd: SuperSecureReallyCR45yP@ssw0rd!@#

For Puppet code like this, however:

hiera_array(mygroups)

Puppet will continue to evaluate all the Hiera files in the hierarchy and build an array out of the values discovered. For instance this:

commercesite.yaml

mygroups:
  commerceadmins
  commercedev
  

common.yaml

mygroups:
  admins
  backupuser

Would result in a final array of:

mygroups = [‘commerceadmins’, ‘commercedev’, ‘admins’, ‘backupuser’]

Hiera_hash works in a very similar way, but collecting hashes instead of an array.

Puppet 3 added an additional consideration with automatic lookups. Now any parameterized class like the one below will act as if there is a hiera() lookup for every parameter, specifically in the form of classname::variablename. For $servers this would be ntp::servers.

init.pp

class ntp (
$autoupdate        = $ntp::params::autoupdate,
$broadcastclient   = $ntp::params::broadcastclient,
$config            = $ntp::params::config,
$servers           = $ntp::params::servers,
$service_enable    = $ntp::params::service_enable,

) {...

This greatly simplifies the ability to reuse Puppet code without needing to make explicit Hiera calls everywhere (which typically don’t exist in Forge modules). This value however will be overridden if the parameter is defined by an ENC of the Puppet Enterprise Node Classifier.

The next wrinkle is the addition of multiple backends in the hiera.yaml. The default backend for Hiera is yaml; however, numerous additional backends can be added, the most popular of which is eyaml. It looks like this:

hiera.yaml

---

:backends: - eyaml - yaml

When you use an additional backend, Heira's workload effectively doubles because it must walk the entire tree of the first before moving on to the second backend.

Note: Yaml can contain regular content alongside encrypted content, and the eyaml backend is capable of processing both. This could mean that Puppet is doing the extra work of two passes for no benefit (assuming you are using the same datadir and file extension). If that is the case, removing yaml as a backend in hiera.yaml is advised.

The next consideration is how Hiera actually gets evaluated. We’re used to Hiera being evaluated from within the Puppet Server during catalog compilation. However, Hiera can be accessed directly from the Hiera executable, or through the Puppet agent. Because we have the all-in-one Puppet agent, the Ruby installation is bundled directly with Puppet; this installation of Ruby could exist in parallel with a local install of Ruby. All three of these methods to call Hiera will have different locations for the Ruby gems they utilize.

  1. System Ruby. It’s not recommended to install any Puppet- or Hiera-related gems in this location. Many organizations have installed hiera-eyaml here to have access to eyaml executable in the default path, however it’s recommended to utilize the version in the Puppet agent gem location instead.

    Default location on Centos 7: /usr/share/gems

  2. Puppet Ruby with agent gems. This backends the hiera executable, puppet apply and puppet lookup and /opt/puppetlabs/puppet/bin/eyaml among others.

    Default location on Puppet Agent 1.7.1: /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0

  3. Puppet Ruby with Puppet Server gems. This backends a normal Puppet run.

    Default location on Puppet Enterprise 2016.4: /opt/puppetlabs/server/data/puppetserver/jruby-gems

If you’re troubleshooting a custom backend (like eyaml) that is installed as a Ruby gem, you’ll likely want it to be in at least locations 2 and 3 above. Generally it’s preferred to avoid system ruby entirely. Each location will have a different command to install:

System Ruby

[root@master hieradata]# gem install hiera-eyaml

Puppet Agent Ruby

[root@master hieradata]# /opt/puppetlabs/puppet/bin/gem install hiera-eyaml

Puppet Server

[root@master hieradata]# /opt/puppetlabs/bin/puppetserver gem install hiera-eyaml

Note: A note on the Puppet Server gem install. The JRuby Puppet instances that actually compile catalogs as a part of Puppet Server run as pe-puppet, not root. If only root has access to this gem location, these gems will not be loaded.

Troubleshooting steps

Let’s see how to figure out what's going on when Hiera doesn’t work as expected. In general, understanding how Hiera works from the previous section will let you follow the process through and find the issue. This section will provide several tools and highlight some common issues. If one of these common issues isn’t the problem, the general strategy is to follow the process via which Puppet makes calls to Hiera, as described in the prior section, and validate each step to find the culprit.

Common issues

This section highlights the most common issues I encounter.

  • Yaml errors. Make sure to use YAML lint. YAML is incredibly finicky. You probably won’t notice a missing space, but YAML will. That is to say, it will just silently ignore what you entered. There are a bunch of YAML linting tools available, that can be used online or integrated into your workflow. I highly recommend building a pre-commit hook to run YAML lint against any commits to your code to systematically avoid most syntax errors. One of the most commonly used linting tools is puppet-syntax, which you can find here: https://github.com/gds-operations/puppet-syntax. It handles Puppet and ERB templates as well as yaml.

  • Restart pe-puppetserver. Any update to hiera.yaml will require pe-puppetserver to be restarted. Updates to Hiera files under your datadir do not require a restart. As of Puppet Enterprise 2016.5, it will be possible to issue service pe-puppetserver reload to reload the files without requiring a complete restart of Puppet Server.

  • Facts issues. Oftentimes, for whatever reason, a node doesn’t actually have the facts you think it does. Double check. A good way to isolate code and other Hiera issues is to place a value in common.yaml. If that loads but the more specific one does not, then it’s not a Puppet code issue.

  • Missing or incorrectly installed Ruby gems. This happens quite often with hiera-eyaml. Either it’s been installed only into the agent’s gem path but not for the Puppet Server, or the permissions are wrong. Quite often, I’ve seen security-conscious organizations set a more restrictive umask than 0022. In that case, /opt/puppetlabs/bin/puppetserver gem install hiera-eyaml will complete successfully, but it will install the package without enough permissions for the pe-puppet user to access them. This issue is being fixed in Puppet Enterprise 2016.5 by automatically running the puppetserver gem command with the appropriate umask.

Tools

Here are a few tools and techniques to help identify which parts of Hiera are or are not working:

  • YAML Lint. I’m serious about this. You can copy and paste your yaml into this website if you have no better way to do it (although I probably wouldn’t want to send passwords that way): http://www.yamllint.com/. As I mentioned above, Puppet Syntax is a great tool for integrating into a continuous integration workflow.

  • Puppet Hiera lookup. This uses the agent gems, so may not be indicative of Puppet Server issues. It will be helpful to verify the hiera.yaml configuration. If you’re utilizing hiera-eyaml, it will also verify that the keys are configured correctly, and that the correct yaml files are being utilized. Verification looks like this:

    puppet lookup --debug --explain --node

    The output will show the actual keys found, and the debug information will show which files it attempted to look for, and where ultimately the keys were found:

[root@master vagrant]# puppet lookup --debug --explain --node centos6a.pdx.puppet.vm ntp::servers | tail
Debug: hiera(): Cannot find datafile /etc/puppetlabs/code/environments/test/hieradata/environment/test.yaml, skipping
Debug: hiera(): Looking for data source datacenter/portland
Debug: hiera(): Found ntp::servers in datacenter/portland
Data Binding "hiera"
  Found key: "ntp::servers" value: [
    "0.us.pool.ntp.org",
    "1.us.pool.ntp.org",
    "2.us.pool.ntp.org",
    "3.us.pool.ntp.org"
  ]
  • Ben Ford’s hiera_explain application. This creates a pretty clean dump of the files Hiera is using and the lookup results for Hiera items that the node is using. You can install it as a rubygem into the Puppet agent gem repo:

    /opt/puppetlabs/puppet/bin/gem install hiera_explain

    And run it with:

    /opt/puppetlabs/puppet/bin/hiera_explain

    Here is a screenshot of it in action:

  • Use puppetserver ruby to load Hiera as pe-puppet inside of puppetserver. This will often catch configuration or permissions issues:

    sudo -u pe-puppet /opt/puppetlabs/bin/puppetserver ruby /opt/puppetlabs/bin/hiera --debug environment=production

  • Run Puppet Server in the foreground (be prepared for a giant log). This will tell you exactly what is going on, but will take a while to parse through.

    • First stop the pe-puppetserver service.
    • Run '/opt/puppetlabs/bin/puppetserver foreground --debug | tee /tmp/debuglog.txt'.
    • Wait for Puppet Server to say it’s ready.
    • Do a 'puppet agent -t' on the client.
    • Once the run as completed, kill that running job and then restart the pe-puppetserver service. Review the /tmp/debuglog.txt file for the Hiera calls.

      There will be numerous lookups in the output. I’ve extracted the example for ntp::servers below. Hopefully you can follow from the background section how Hiera goes through its steps to find a value:

    2016-12-01 18:05:05,469 DEBUG [qtp385824407-32] [puppetserver] Puppet Performing a hiera indirector lookup of ntp::servers with options {:variables=>Scope(Class[Ntp]), :merge=>#<Puppet::Pops::FirstFoundStrategy:0x434d84f6 @options_t=#<Puppet::Pops::Types::PStructType:0x1bd0cd6d @elements=[#<Puppet::Pops::Types::PStructElement:0x2c77cae1 @key_type=#<Puppet::Pops::Types::POptionalType:0x33291ad @type=#<Puppet::Pops::Types::PStringType:0x1e148f03 @size_type=nil, @values=["strategy"]>>, @value_type=#<Puppet::Pops::Types::POptionalType:0x52a0fae2 @type=#<Puppet::Pops::Types::PPatternType:0x4588a6e2 @patterns=[#<Puppet::Pops::Types::PRegexpType:0x2f9ac6f7 @pattern="first", @regexp=/first/>]>>>]>, @options={}>}
    2016-12-01 18:05:05,470 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): Using Hiera 1.x backend API to access instance of class Hiera::Backend::Eyaml_backend. Lookup recursion will not be detected
    2016-12-01 18:05:05,471 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): [eyaml_backend]: Set option: datadir = /etc/puppetlabs/code/environments/production/hieradata
    2016-12-01 18:05:05,471 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): [eyaml_backend]: Set option: pkcs7_private_key = /etc/puppetlabs/puppet/keys/private_key.pkcs7.pem
    2016-12-01 18:05:05,471 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): [eyaml_backend]: Set option: pkcs7_public_key = /etc/puppetlabs/puppet/keys/public_key.pkcs7.pem
    2016-12-01 18:05:05,471 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): [eyaml_backend]: Looking up ntp::servers in eYAML backend
    2016-12-01 18:05:05,471 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): [eyaml_backend]: Looking for data source nodes/master.inf.puppet.vm
    2016-12-01 18:05:05,472 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): Cannot find datafile /etc/puppetlabs/code/environments/production/hieradata/nodes/master.inf.puppet.vm.eyaml, skipping
    2016-12-01 18:05:05,472 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): [eyaml_backend]: Looking for data source environment/production
    2016-12-01 18:05:05,472 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): Cannot find datafile /etc/puppetlabs/code/environments/production/hieradata/environment/production.eyaml, skipping
    2016-12-01 18:05:05,472 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): [eyaml_backend]: Looking for data source datacenter/infrastructure
    2016-12-01 18:05:05,472 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): Cannot find datafile /etc/puppetlabs/code/environments/production/hieradata/datacenter/infrastructure.eyaml, skipping
    2016-12-01 18:05:05,473 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): [eyaml_backend]: Looking for data source virtual/virtualbox
    2016-12-01 18:05:05,473 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): Cannot find datafile /etc/puppetlabs/code/environments/production/hieradata/virtual/virtualbox.eyaml, skipping
    2016-12-01 18:05:05,473 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): [eyaml_backend]: Looking for data source common
    2016-12-01 18:05:05,473 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): Cannot find datafile /etc/puppetlabs/code/environments/production/hieradata/common.eyaml, skipping
    2016-12-01 18:05:05,473 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): Looking up ntp::servers in YAML backend
    2016-12-01 18:05:05,474 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): Looking for data source nodes/master.inf.puppet.vm
    2016-12-01 18:05:05,475 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): Cannot find datafile /etc/puppetlabs/code/environments/production/hieradata/nodes/master.inf.puppet.vm.yaml, skipping
    2016-12-01 18:05:05,476 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): Looking for data source environment/production
    2016-12-01 18:05:05,477 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): Looking for data source datacenter/infrastructure
    2016-12-01 18:05:05,477 DEBUG [qtp385824407-32] [puppetserver] Puppet hiera(): Found ntp::servers in datacenter/infrastructure

Conclusion

The goal of this post was to help peel back the layers of complexity surrounding how Hiera works with Puppet and Puppet Server. The tools and processes represented here have made the difference for me between confused frustration and harnessing the power of Hiera. I hope these tools and techniques will help you just as much.

If you have any questions or suggestions of your own, please share them in the comments section below.

Chris Matteson is a senior technical solutions engineer at Puppet.

Learn more

Puppet sites use proprietary and third-party cookies. By using our sites, you agree to our cookie policy.