published on 20 February 2014

For updated instructions on using Hiera, read Debugging Hiera redux.

Hiera is a powerful tool for separating data from code. But once Hiera has been installed and you start to use it with Puppet, you may feel like Cthulhu has invaded your code because you aren’t getting the data you expected. So how do you go about debugging it without waking the sleeping Old Great Ones or losing your sanity?

Glad you asked. There are several common issues that come up with Hiera, and I’m here to show you how to track them down and fix them.

The first thing to remember is that any time you edit the hiera.yaml file you will need to restart your Puppet master. The hiera.yaml file is located by default at $confdir/hiera.yaml, which translates to:

  • /etc/puppetlabs/puppet/hiera.yaml for Puppet Enterprise
  • /etc/puppet/hiera.yaml for Open Source Puppet

To restart your puppet master:

  • For Puppet Enterprise, use sudo service pe-httpd restart
  • For Open Source Puppet:
    • restart Apache httpd (or Nginx) if using Passenger
    • use sudo service puppetmaster restart if using WEBrick

So, the first step in debugging Hiera is to restart your master, especially if you have edited your hiera.yaml. The next step is to see what results you get from the command line. One thing to make note of: Command line hiera uses a default config file of /etc/hiera.yaml instead of the default path of $confdir/hiera.yaml that Puppet uses for Hiera. There are two options for dealing with this. You can specify -c /path/to/hiera.yaml each time you call hiera, or — what I prefer to do — create a symlink at /etc/hiera.yaml that points to $confdir/hiera.yaml.

Now that we have Hiera using the same config file, whether it is invoked via Puppet or via the command line, we are now ready to do some actual testing. Well, almost. Before you can accurately invoke Hiera from the command line, you will need to know some information about your hierarchy and the node you want to see the results from. If your hierarchy makes use of any facts or other variables, you will have to supply those to be able to do a successful lookup.

There are a couple of different ways you can supply that information.

  • As command line variables

    • osfamily=Debian timezone=CST
  • Using YAML

    • --yaml facts.yaml (or -y)
      ---
      osfamily: Debian
      timezone: CST
  • Using JSON

    • --json facts.json (or -j)
      {
      “osfamily” : “Debian”,
      “timezone” : “CST”
      }
  • From MCollective

    • --mcollective <identity>
    • identity will be the certname of the node you want to do the lookup from.
    • Note: If you are using Puppet Enterprise this will have to be done as the peadmin user. On Open Source Puppet, it will have to be run by a user who has been set up to run MCollective commands.

You might feel inclined to do this from Puppet, using --inventory_service <identity> — but don’t. At the time of writing, this feature does not work and will generate an error.

Which method you use doesn’t really matter too much, and will largely depend on how complex your hierarchy is. For the moment, let’s assume we have a simple case.

`hiera somefact osfamily=Debian`    

This will return “somefact,” and is comparable to using the hiera() function in a manifest. This means it will stop looking after it finds one match. If we need to find all matches, we can add the -a or --array flag, which will make it act like hiera_array(), or for hiera_hash(), -h or --hash. Both of these will find all matching keys and then merge them into a single array or hash.

This output is useful, but if we have multiple data sources in our hierarchy, we may not be getting the results we expect, or perhaps we want to make sure that we are using the right hierarchy. If we add the --debug (or -d) flag, we will be able to see what data sources Hiera looks in.

Another useful construct for doing lookups is the following:

puppet apply -e '$somevar = hiera(foo) notify { $somevar: }'

This will allow you to do a quick lookup of a variable within Puppet, but not have to worry about all the extra stuff. Keep in mind, if you need to do the lookup from the point of view of a node other than the master, this will not work. Depending on your setup, running this on the agent in question may work. Otherwise, you’ll need to use one of the other methods discussed.

Now that we are able to do Hiera lookups from the command line, let’s look at some common problems and how to fix them.

Perhaps the most common problem has to do with syntax errors in the yaml files. These are fairly easy to identify in Puppet, as they will include a line number, a column number but no file name. These types of errors are both the simplest to fix, and some of the most annoying to find.

Some of the more common errors are when the opening --- is malformed. If it gets converted into a unicode character, such as , or if you have a space at the start of the line (‘---’), you will see an error like:

Error: Could not run: (<unknown>): mapping values are not allowed in this context at line 2 column 8

If you accidentally put a space in the opening set of dashes, - -- , you will get an error like this:

Error: Could not run: (<unknown>): did not find expected '-' indicator while parsing a block collection at line 1 column 1

When you have a normal key value pair, you can enter is as key: value or key : value. However, if you enter it as key:value then you see an error like:

Error: Could not run: (<unknown>): mapping values are not allowed in this context at line 3 column 10

If you should happen to put a tab instead of spaces for the indentation, you will see an error like this:

Error: Could not run: (<unknown>): found character that cannot start any token while scanning for the next token at line 4 column 1

If you try to use hiera_array() or hiera_hash() with the wrong data types, you will get errors such as:

Error: Could not run: Hiera type mismatch: expected Hash and got String

Error: Could not run: Hiera type mismatch: expected Hash and got Array

Error: Could not run: Hiera type mismatch: expected Array and got Hash

If you notice, there is no error listed about expecting an array and getting a string. This is because the string is silently converted into an array with a single element in this case.

One common problem is permissions. Running Hiera with debug can help identify this, but you will have to make sure that you run the command as the same user that the Puppet master runs as. This user will typically be pe-puppet or puppet. If you run Hiera as a different user, the permissions may vary, and you may see more or less than what will be available to Puppet.

You can find more information about setting up Hiera and how to use it in its documentation.

Learn More

Share via:
Posted in:
Tagged:

Testing hiera lookups using `puppet apply -e` is a great idea! You can simplify it a bit by using a `notice` statement like so: `puppet apply -e "notice hiera('foo')"`.

Also, you may need to use sudo for the lookup to work properly.

Thanks for the tip on 'notice'. I am also using hiera_array() instead hiera(), as hiera() only returns the first match.

```
[root@puppet3 puppet]# puppet apply -e "notice hiera_array(classes)"
Notice: Scope(Class[main]): vmwaretools stefan-yum ntp motd stefan-puppet epel users
```

Using `hiera --array` also works:

```
[root@puppet3 puppet]# hiera --array classes ::virtual=vmware
["vmwaretools", "ntp", "motd", "stefan-puppet", "epel", "users"]
```

Check your code examples. In versions of hiera after 1.1.2, it needs to use ::osfamily.

This was just fixed in the docs.

(Note, I'm doing this from memory, so slight chance I'm wrong.)

Mathew,

For the hiera.yaml, you can do it one of two ways.

You can do - "%{osfamily}" or - %{::osfamily}.

Basically, you have to use double quotes around the string OR use the :: top scope indicator, but not both.

FriedBob, are you sure about that? It seems to have no bearing on my tests. In the following example, 'virtual/${::virtual}' is not enclosed with quotes, but '"osfamily/%{::osfamily}"' is. It doesn't seem to make a difference:

```
[root@puppet3 puppet]# cat hiera.yaml
# /etc/puppet/hiera.yaml
# borrowed from http://docs.puppetlabs.com/hiera/1/complete_example.html

---
:backends:
- yaml
:yaml:
:datadir: /etc/puppet/hieradata
:hierarchy:
#- "nodes/%{::fqdn}"
- "nodes/%{::clientcert}"
- virtual/%{::virtual}
- "osfamily/%{::osfamily}"
- common
[root@puppet3 puppet]# hiera classes ::virtual=vmware
vmwaretools
[root@puppet3 puppet]# hiera vmwaretools::working_dir ::osfamily=Debian
/usr/local/vmware
[root@puppet3 puppet]#
```

Note that you may need to prepend your facts using `::`, depending on how the variables are specified in hiera.yaml. For example, given the following example which I borrowed from the Puppet Labs hiera docs (see [1] below):

```
[root@puppet3 puppet]# cat /etc/puppet/hiera.yaml
# /etc/puppet/hiera.yaml
# borrowed from http://docs.puppetlabs.com/hiera/1/complete_example.html

---
:backends:
- yaml
:yaml:
:datadir: /etc/puppet/hieradata
:hierarchy:
- "nodes/%{::clientcert}"
- "virtual/%{::virtual}"
- "osfamily/%{::osfamily}"
- common
```

I must specify facts using `::`. If I don't the facts won't display which is very confusing:

```
[root@puppet3 puppet]# hiera vmwaretools::working_dir osfamily=Debian
nil
[root@puppet3 puppet]# hiera vmwaretools::working_dir ::osfamily=Debian
/usr/local/vmware
[root@puppet3 puppet]#
```

[1] http://docs.puppetlabs.com/hiera/1/complete_example.html

`puppet apply -e '$somevar = hiera(foo) notify { $somevar: }'`

Could you elaborate on this some more using real facts, or with a real like 'osfamily' hiera.yaml manifest? All I get is errors, even 'classes' and 'osfamily' are listed ion my Hierarchy. I don't understand why these aren't getting called, and I'm looking for some troubleshooting to help me out.

```
[root@clone1 ~]# puppet apply -e '$somevar = hiera("classes") notify { $somevar: }'
Error: Could not find data item classes in any Hiera data file and no default supplied at line 1 on node clone1.example.org
Error: Could not find data item classes in any Hiera data file and no default supplied at line 1 on node clone1.example.org

[root@clone1 ~]# puppet apply -e '$somevar = hiera("osfamily") notify { $somevar: }'
Error: Could not find data item osfamily in any Hiera data file and no default supplied at line 1 on node clone1.example.org
Error: Could not find data item osfamily in any Hiera data file and no default supplied at line 1 on node clone1.example.org
[root@clone1 ~]#
```

FWIW - the error "Could not find data item classes in any Hiera data file and no default supplied at line 1 on node clone1.example.org" comes about from asking hiera for the classes defined in the hieradata, but not having any defined - and also not providing a default.

So if I come to read about

> Debugging Hiera
> February 20, 2014 by Lee Lowder in Product, Tips & How To, Hiera

The last thing I want to hear is

> Hiera is a POWERFUL tool for separating data from code.
> But once Hiera has been installed and you start to use it with Puppet,
> you MAY FEEL LIKE CTHULEU HAS INVADED your code because you
> AREN'T GETTING the data you expected.
> So how do you go about debugging it without WAKING THE SLEEPING
> OLD GREAT ONES OR LOSING YOUR SANITY ?

> GLAD YOU ASKED. There are several common issues that come up with Hiera,
> and I’m HERE TO SHOW YOU how to track them down and fix them.

seriously, you know what makes me loose my sanity ?

Add new comment

The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.