homeblogdebugging hiera redux

Debugging Hiera redux

This post updates Lee Lowder's popular post, Debugging Hiera.

Hiera is a key-value lookup system that operates using a deceptively simple algorithm:

  1. When asked to look up a key, give back a value. There are multiple possible values that could be returned, so figure out first which one is the right value.
  2. Use a hierarchy to do that.

Sounds easy, right? It’s pretty simple on the surface.

Hiera might be simple, but it’s also powerful, and can be subject to emergent complexity. Like an ant: An ant seems simple enough if you’re watching one move around on a tabletop, but find a whole bunch of them all together in a real-world environment – think anthill – and you pretty quickly stop using “simple” as your go-to descriptor.

Every Hiera lookup is performed in the context of a particular Puppet node being configured, and a large scope of data pertaining specifically to that node is used to inform Hiera’s decision-making process: which data sets to select, how to order them, and how to interpolate key values. That’s not watching an ant wander around on the tabletop: that’s trying to follow one through its inscrutable inside-the-anthill routine.

Debugging Hiera is the act of trying to understand why, for any given lookup key in the context of a particular Puppet node, Hiera a) returned the value it did, b) didn’t return any value, or c) returned an error.

Two years ago, Lee Lowder wrote a blog post on the problems commonly encountered when working with Hiera, and how to debug them. In the intervening years, the Puppet ecosystem has evolved and new tools have emerged. One of those tools in particular, Puppet Lookup, greatly simplifies figuring out Hiera lookups, so it’s time for a redux on how to debug Hiera.

Hiera pain points

The kinds of problems you might run into with Hiera can be broken down into two main categories.

  • Problems with configuration or syntax of the data files.
  • Logic or evaluation problems with the actual data in the files.

The common pain point of debugging either type of error is that in both cases Hiera can be opaque with regard to what exactly is going on. Because Hiera is all about trying a bunch of different ways to look up a value, it doesn’t expect all of them to work. It fails some things silently that we might want to know more about. When it fails loudly, it can be difficult to trace the error raised back to the specific configuration or data file with a problem.

Generalized debugging

Debugging usually starts after Puppet emits some kind of error related to Hiera—either an explicit error message, or by returning unexpected configuration data. For example:

When Puppet raises this kind of error, or others like it, it's a cue to look into what Hiera is doing. The first objective is to get more information. In the case of the error message shown above, there are references to lines and columns with syntax errors, but no reference to what file those lines and columns came from. If the problem is that the data returned is not the data expected, no message will be printed at all. (Hiera doesn’t know the data it retrieved isn’t the data you thought it should retrieve.) Therefore, in general, Step No.1 in debugging Hiera is always going to be asking, “tell me more.”

Hiera will happily tell you step-by-step what hierarchy it has created and what file(s) it is opening as it attempts to open them, provided you first raise the logging verbosity level to debug.

Old debugging techniques

It’s always been possible to manually run Hiera independent of Puppet using the hiera command-line executable.

This runs Hiera in the desired debug verbosity, and seems to allow a pinpoint analysis of a specific lookup problem. In practice, it’s insufficient as shown. The invocation fails to collect all the context Hiera needs to perform an accurate key lookup for a given node — namely, the node’s full set of facts and other relevant information (such as environment and clientcert). It’s possible to manually supply this information, but it quickly becomes burdensome and potentially error-prone to do so.

Furthermore, by manually supplying versions of what we believe the correct facts and values should be, we blind ourselves to discovering real-world problems such as, “Puppet actually returned environment=dev, and that’s why everything is backwards.” For this reason, another old recommended debugging technique was to write one line of Puppet code that includes a Hiera lookup for the key to test, and run it with puppet apply. This doesn’t seem as targeted — lots of extra work involved in a Puppet run beyond just the Hiera call — but it negates the need to manually reconstruct facts and other contextual values.

There’s a major limitation to this technique, in that it is effectively limited to running on the Puppet master because Hiera data is typically not available on client systems. This makes it impractical for debugging emergent or non-generic issues.

Note that both commands are being passed the --debug flag, which will make Hiera talk. In the case of hiera there will be a short and to-the-point amount of information returned. In the case of puppet apply, the information you need will be there, but you’ll want to grep or otherwise filter the output because there’ll be a LOT of text, and the stuff you care about will be buried somewhere in the middle.

New debugging technique

Puppet 4 introduces the puppet lookup command. The puppet lookup command is part of a system that goes far beyond just Hiera debugging. More information on lookup and all of its component pieces can be found here. In the context of Hiera debugging, puppet lookup can help identify exactly what Hiera was doing when it raised an error, or how it decided to look up a key and where it got its value(s) from.

Debugging a syntax error

Consider the error message:

To debug this, start by running the following command on the master.

This will not be hugely useful, but you’ll note that all the rest of the errors and logging around the error are cleared. This command performs only the work relevant to the Hiera error. To expose what’s going on under the hood add the --debug flag.

Just like when running puppet apply --debug, a LOT of output will be produced. The difference is there will be less, and the last lines of output will be exactly the ones we care about.

In this example, we can now see that the last thing Hiera was trying to do when it threw the error was to look for the common datasource in the YAML backend. That is, it was trying to read common.yaml. Armed with this more complete information about what Hiera was trying to do when the error was thrown, we now know that the syntax error alluded to is in common.yaml.

Debugging why Hiera returned a value

Just because no error was thrown doesn’t mean there isn’t still a use case for cranking up the visibility to see why Hiera did something it did. If you think your node should be configured to use au.pool.ntp.org but it’s actually being configured with us.pool.ntp.org, no error message is thrown — but there’s definitely something wrong that needs debugging.

The lookup command accepts a --node flag to set the node context for performing the lookup. If --node isn’t passed, it defaults to the context of the node the command is being run on (usually the master).

When I’m trying to determine why one of my non-master nodes is getting a particular value for the ntp::servers parameter, I can use lookup and the --node flag to introspect that process.

What’s interesting in this example is that I’m seeing, step-by-step, how Hiera walks through the fully evaluated version of my hierarchy as it was constructed for the node I’m targeting. The original hierarchy was full of variables for interpolation.

In the lookup debug output, you can see that for this node, environment/%{environment} evaluated to environment/production, but that Hiera didn’t find any data there for ntp::servers. And so on, down the hierarchy. This visibility gives you what you need to think through the lookup process step by step, and confirm with each iteration whether or not everything is working correctly. If a node is returning an unexpected value for datacenter or not returning a value for datacenter at all, that would be visible here. If Hiera is looking in the correct data file but not returning any data from it, that will also be visible here. You’ll know to focus your efforts on why Hiera isn’t returning data from that file. Debugging is all about continuously narrowing the problem down, and that’s exactly what using lookup to watch Hiera in action lets you do.

Specific errors

Getting directions to the specific place that Hiera encountered an error goes a long way towards eliminating frustration in debugging the system. Even armed with that kind of homing-beacon algorithm, it can be useful to get an enumerated idea of what problems can be found when you get there. Towards that end, and to fulfil this post’s role as a redux, I’ll include here some of Lee’s original pointers about what common error messages are encountered and what causes them.



The opening --- could be malformed. If it gets converted into a unicode character, such as , or if you have a space at the start of the line ---, or in between the three dashes - --, you might get an error like this.



This can be caused by using tabs for indentation instead of spaces. This can be particularly tricky to spot visually. Yaml files should generally never contain tab characters.



These kinds of errors can happen when you try to use hiera_array() or hiera_hash(), but one or more of the found values are of a data type incompatible with that lookup.

One final tip: remember that the Puppet server does not run as the root user. Sometimes problems occur when the Puppet server is running, but are not reproducible when running manually as root with puppet lookup. These seemingly inscrutable issues can often boil down to things as mundane as incorrect file permissions.

Reid Vandewiele is a principal technical solutions engineer at Puppet.

Learn more

To fully emulate any Hiera call, you’ll need to learn a few more flags for lookup, beyond what was shown in these starting examples. Check out the full command reference here. Take specific note of the --merge flag, which lets you perform hash, array, and deep merge lookups in addition to the default priority-first.

Most of the information from Lee’s original Debugging Hiera post is redux’d here, but if you’re curious to go back and see the original it can be found here.

Finally, if lookup piqued your interest and you’re curious to explore it beyond just the Hiera debugging use case and docs, check out R.I. Pienaar’s impressively deep dive into all aspects of the tool.