Debugging Hiera redux

See more posts about: Tips & How To

This post updates Lee Lowder's popular post, Debugging Hiera.

Hiera is a key-value lookup system that operates using a deceptively simple algorithm:

  1. When asked to look up a key, give back a value. There are multiple possible values that could be returned, so figure out first which one is the right value.
  2. Use a hierarchy to do that.

Sounds easy, right? It’s pretty simple on the surface.

Hiera might be simple, but it’s also powerful, and can be subject to emergent complexity. Like an ant: An ant seems simple enough if you’re watching one move around on a tabletop, but find a whole bunch of them all together in a real-world environment – think anthill – and you pretty quickly stop using “simple” as your go-to descriptor.

Every Hiera lookup is performed in the context of a particular Puppet node being configured, and a large scope of data pertaining specifically to that node is used to inform Hiera’s decision-making process: which data sets to select, how to order them, and how to interpolate key values. That’s not watching an ant wander around on the tabletop: that’s trying to follow one through its inscrutable inside-the-anthill routine.

Debugging Hiera is the act of trying to understand why, for any given lookup key in the context of a particular Puppet node, Hiera a) returned the value it did, b) didn’t return any value, or c) returned an error.

Two years ago, Lee Lowder wrote a blog post on the problems commonly encountered when working with Hiera, and how to debug them. In the intervening years, the Puppet ecosystem has evolved and new tools have emerged. One of those tools in particular, Puppet Lookup, greatly simplifies figuring out Hiera lookups, so it’s time for a redux on how to debug Hiera.

Hiera pain points

The kinds of problems you might run into with Hiera can be broken down into two main categories.

  • Problems with configuration or syntax of the data files.
  • Logic or evaluation problems with the actual data in the files.

The common pain point of debugging either type of error is that in both cases Hiera can be opaque with regard to what exactly is going on. Because Hiera is all about trying a bunch of different ways to look up a value, it doesn’t expect all of them to work. It fails some things silently that we might want to know more about. When it fails loudly, it can be difficult to trace the error raised back to the specific configuration or data file with a problem.

Generalized debugging

Debugging usually starts after Puppet emits some kind of error related to Hiera—either an explicit error message, or by returning unexpected configuration data. For example:

[root@master ~]# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on     SERVER: Evaluation Error: Error while evaluating a Resource Statement,     Lookup of key 'pe_console_prune::ensure_prune_cron' failed:     DataBinding 'hiera': (<unknown>): mapping values are not allowed here     at line 2 column 68 on node master.inf.puppetlabs.demo
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

When Puppet raises this kind of error, or others like it, it's a cue to look into what Hiera is doing. The first objective is to get more information. In the case of the error message shown above, there are references to lines and columns with syntax errors, but no reference to what file those lines and columns came from. If the problem is that the data returned is not the data expected, no message will be printed at all. (Hiera doesn’t know the data it retrieved isn’t the data you thought it should retrieve.) Therefore, in general, Step No.1 in debugging Hiera is always going to be asking, “tell me more.”

Hiera will happily tell you step-by-step what hierarchy it has created and what file(s) it is opening as it attempts to open them, provided you first raise the logging verbosity level to debug.

Old debugging techniques

It’s always been possible to manually run Hiera independent of Puppet using the hiera command-line executable.

hiera -c /etc/puppetlabs/puppet/hiera.yaml \
  --debug \

This runs Hiera in the desired debug verbosity, and seems to allow a pinpoint analysis of a specific lookup problem. In practice, it’s insufficient as shown. The invocation fails to collect all the context Hiera needs to perform an accurate key lookup for a given node — namely, the node’s full set of facts and other relevant information (such as environment and clientcert). It’s possible to manually supply this information, but it quickly becomes burdensome and potentially error-prone to do so.

hiera -c /etc/puppetlabs/puppet/hiera.yaml \
  --debug \
  --json facts.json \
  pe_console_prune::ensure_prune_cron \
  environment=production \

Furthermore, by manually supplying versions of what we believe the correct facts and values should be, we blind ourselves to discovering real-world problems such as, “Puppet actually returned environment=dev, and that’s why everything is backwards.” For this reason, another old recommended debugging technique was to write one line of Puppet code that includes a Hiera lookup for the key to test, and run it with puppet apply. This doesn’t seem as targeted — lots of extra work involved in a Puppet run beyond just the Hiera call — but it negates the need to manually reconstruct facts and other contextual values.

puppet apply --debug -e '$somevar = hiera(foo) notify { $somevar: }'

There’s a major limitation to this technique, in that it is effectively limited to running on the Puppet master because Hiera data is typically not available on client systems. This makes it impractical for debugging emergent or non-generic issues.

Note that both commands are being passed the --debug flag, which will make Hiera talk. In the case of hiera there will be a short and to-the-point amount of information returned. In the case of puppet apply, the information you need will be there, but you’ll want to grep or otherwise filter the output because there’ll be a LOT of text, and the stuff you care about will be buried somewhere in the middle.

New debugging technique

Puppet 4 introduces the puppet lookup command. The puppet lookup command is part of a system that goes far beyond just Hiera debugging. More information on lookup and all of its component pieces can be found here. In the context of Hiera debugging, puppet lookup can help identify exactly what Hiera was doing when it raised an error, or how it decided to look up a key and where it got its value(s) from.

Debugging a syntax error

Consider the error message:

SERVER: Evaluation Error: Error while evaluating a Resource Statement, Lookup of key 'pe_console_prune::ensure_prune_cron' failed: DataBinding 'hiera': (<unknown>): mapping values are not allowed here at line 2 column 68 on node master.inf.puppetlabs.demo

To debug this, start by running the following command on the master.

puppet lookup pe_console_prune::ensure_prune_cron

This will not be hugely useful, but you’ll note that all the rest of the errors and logging around the error are cleared. This command performs only the work relevant to the Hiera error. To expose what’s going on under the hood add the --debug flag.

puppet lookup --debug pe_console_prune::ensure_prune_cron

Just like when running puppet apply --debug, a LOT of output will be produced. The difference is there will be less, and the last lines of output will be exactly the ones we care about.

Debug: hiera(): [eyaml_backend]: Hiera eYAML backend starting
Debug: hiera(): Hiera YAML backend starting
Debug: hiera(): Looking up lookup_options in YAML backend
Debug: hiera(): Looking for data source     nodes/master.inf.puppetlabs.demo
Debug: hiera(): Cannot find datafile /etc/puppetlabs/code/environments/production/hieradata/nodes/master.inf.puppetlabs.demo.yaml, skipping
Debug: hiera(): Looking for data source environment/production
Debug: hiera(): Looking for data source datacenter/infrastructure
Debug: hiera(): Looking for data source virtual/virtualbox
Debug: hiera(): Looking for data source common
Error: Could not run: Evaluation Error: Error while evaluating a Resource Statement, Lookup of key 'pe_console_prune::ensure_prune_cron' failed: DataBinding 'hiera': (<unknown>): mapping values are not allowed in this context at line 2 column 68

In this example, we can now see that the last thing Hiera was trying to do when it threw the error was to look for the common datasource in the YAML backend. That is, it was trying to read common.yaml. Armed with this more complete information about what Hiera was trying to do when the error was thrown, we now know that the syntax error alluded to is in common.yaml.

Debugging why Hiera returned a value

Just because no error was thrown doesn’t mean there isn’t still a use case for cranking up the visibility to see why Hiera did something it did. If you think your node should be configured to use but it’s actually being configured with, no error message is thrown — but there’s definitely something wrong that needs debugging.

The lookup command accepts a --node flag to set the node context for performing the lookup. If --node isn’t passed, it defaults to the context of the node the command is being run on (usually the master).

When I’m trying to determine why one of my non-master nodes is getting a particular value for the ntp::servers parameter, I can use lookup and the --node flag to introspect that process.

[root@master ~]# puppet lookup --node centos7a.pdx.puppetlabs.local --debug ntp::servers
Debug: hiera(): Looking up ntp::servers in YAML backend
Debug: hiera(): Looking for data source nodes/centos7a.pdx.puppetlabs.demo
Debug: hiera(): Cannot find datafile /etc/puppetlabs/code/environments/production/hieradata/nodes/centos7a.pdx.puppetlabs.demo.yaml, skipping
Debug: hiera(): Looking for data source environment/production
Debug: hiera(): Looking for data source datacenter/portland
Debug: hiera(): Found ntp::servers in datacenter/portland

What’s interesting in this example is that I’m seeing, step-by-step, how Hiera walks through the fully evaluated version of my hierarchy as it was constructed for the node I’m targeting. The original hierarchy was full of variables for interpolation.

  - nodes/%{clientcert}
  - environment/%{environment}
  - datacenter/%{datacenter}
  - virtual/%{virtual}
  - common

In the lookup debug output, you can see that for this node, environment/%{environment} evaluated to environment/production, but that Hiera didn’t find any data there for ntp::servers. And so on, down the hierarchy. This visibility gives you what you need to think through the lookup process step by step, and confirm with each iteration whether or not everything is working correctly. If a node is returning an unexpected value for datacenter or not returning a value for datacenter at all, that would be visible here. If Hiera is looking in the correct data file but not returning any data from it, that will also be visible here. You’ll know to focus your efforts on why Hiera isn’t returning data from that file. Debugging is all about continuously narrowing the problem down, and that’s exactly what using lookup to watch Hiera in action lets you do.

Specific errors

Getting directions to the specific place that Hiera encountered an error goes a long way towards eliminating frustration in debugging the system. Even armed with that kind of homing-beacon algorithm, it can be useful to get an enumerated idea of what problems can be found when you get there. Towards that end, and to fulfil this post’s role as a redux, I’ll include here some of Lee’s original pointers about what common error messages are encountered and what causes them.


Error: Could not run: (<unknown>): mapping values are not allowed in this context at line 2 column 8
Error: Could not run: (<unknown>): did not find expected '-' indicator while parsing a block collection at line 1 column 1


The opening --- could be malformed. If it gets converted into a unicode character, such as , or if you have a space at the start of the line ---, or in between the three dashes - --, you might get an error like this.


Error: Could not run: (<unknown>): mapping values are not allowed in this context at line 3 column 10


This can be caused by using tabs for indentation instead of spaces. This can be particularly tricky to spot visually. Yaml files should generally never contain tab characters.


Error: Could not run: Hiera type mismatch: expected Hash and got String
Error: Could not run: Hiera type mismatch: expected Hash and got Array
Error: Could not run: Hiera type mismatch: expected Array and got Hash


These kinds of errors can happen when you try to use hiera_array() or hiera_hash(), but one or more of the found values are of a data type incompatible with that lookup.

One final tip: remember that the Puppet server does not run as the root user. Sometimes problems occur when the Puppet server is running, but are not reproducible when running manually as root with puppet lookup. These seemingly inscrutable issues can often boil down to things as mundane as incorrect file permissions.

Reid Vandewiele is a principal technical solutions engineer at Puppet.

Learn more

To fully emulate any Hiera call, you’ll need to learn a few more flags for lookup, beyond what was shown in these starting examples. Check out the full command reference here. Take specific note of the --merge flag, which lets you perform hash, array, and deep merge lookups in addition to the default priority-first.

Most of the information from Lee’s original Debugging Hiera post is redux’d here, but if you’re curious to go back and see the original it can be found here.

Finally, if lookup piqued your interest and you’re curious to explore it beyond just the Hiera debugging use case and docs, check out R.I. Pienaar’s impressively deep dive into all aspects of the tool.

Puppet sites use proprietary and third-party cookies. By using our sites, you agree to our cookie policy.