Upgrading to Hiera 5
Upgrading to Hiera 5 offers some major advantages. A real environment data layer means changes to your hierarchy are now routine and testable, using multiple backends in your hierarchy is easier and you can make a custom backend.
Hiera 5 uses the same built-in data formats as Hiera 3. You don't need to do mass edits of any data files.
Updating your code to take advantage of Hiera 5 features involves the following tasks:
Task | Benefit |
---|---|
Enable the environment layer,
by giving each environment its own hiera.yaml file.
Note: Enabling the environment layer takes the most work,
but yields the biggest benefits. Focus on that first, then do the
rest at your own pace.
|
Future hierarchy changes are
cheap and testable. The legacy Hiera functions (hiera , hiera_array ,
hiera_hash , and hiera_include ) gain full Hiera 5 powers in any migrated
environment, only if there is a hiera.yaml in the environment root. |
Convert your global hiera.yaml
file to the version 5 format. |
You can use new Hiera 5 backends at the global layer. |
Convert any experimental
(version 4) hiera.yaml files to version 5. |
Future-proof any environments or modules where you used the experimental version of Puppet lookup. |
In Puppet code, replace legacy Hiera
functions (hiera , hiera_array , hiera_hash , and hiera_include ) with lookup() . |
Future-proof your Puppet code. |
Use Hiera for default data in modules. | Simplify your modules with an
elegant alternative to the "params.pp " pattern. |
Considerations for hiera-eyaml users
Upgrade now. In Puppet 4.9.3, we added a built-in hiera-eyaml
backend for Hiera 5. (It still requires that the hiera-eyaml
gem be installed.) See the
usage instructions in the hiera.yaml
(v5) syntax reference. This means you can move your
existing encrypted YAML data into the environment layer at the same time you move
your other data.
Considerations for custom backend users
Wait for updated backends. You can keep using custom Hiera 3 backends with Hiera 5, but they'll make upgrading more complex, because you can't move legacy data to the environment layer until there's a Hiera 5 backend for it. If an updated version of the backend is coming out soon, wait.
If you're using an off-the-shelf custom backend, check its website or contact its developer. If you developed your backend in-house, read the documentation about writing Hiera 5 backends.
Considerations for custom
data_binding_terminus
users
Upgrade now, and replace it with a Hiera 5 backend as soon as possible.
There's a deprecated data_binding_terminus
setting in the puppet.conf
file, which changes the behavior of automatic
class parameter lookup. It can be set to hiera
(normal), none
(deprecated; disables auto-lookup), or the name of a custom plug-in.
With a custom data_binding_terminus
, automatic lookup results are different from
function-based lookups for the same keys. If you're one of the few who use this
feature, you've already had to design your Puppet code to avoid that problem, so it's safe to
upgrade your configuration to Hiera 5. But because we've deprecated that extension point, you have to replace your
custom terminus with a Hiera 5
backend.
If you're using an off-the-shelf plug-in, such as Jerakia, check its website or contact its developer. If you developed your plug-in in-house, read the documentation about writing Hiera 5 backends.
After you
have a Hiera 5 backend, integrate
it into your hierarchies and delete the data_binding_terminus
setting.
Enable the environment layer for existing Hiera data
A key feature in Hiera 5 is per-environment hierarchy configuration. Because you
probably store data in each environment, local hiera.yaml
files are more logical and convenient than a single global
hierarchy.
You can enable the environment layer gradually. In migrated environments, the legacy Hiera functions switch to Hiera 5 mode — they can access environment and module data without requiring any code changes.
hiera-eyaml
, continue upgrading — Puppet 4.9.3 and higher include a Hiera 5 eyaml backend. See the usage instructions in the hiera.yaml
(v5) syntax
reference.In each environment:
For more information on mapping environments to branches, see control repo.
Convert a version 3 hiera.yaml
to version 5
Hiera 5
supports three versions of the hiera.yaml
file:
version 3, version 4, and version 5. If you've been using Hiera 3, your existing configuration is a version 3 hiera.yaml
file at the global layer.
-
Creating new v5
hiera.yaml
files for environments. -
Updating your global configuration to support Hiera 5 backends.
These are essentially the same process, although the global hierarchy has a few special capabilities.
hiera.yaml
version 3 file:
:backends:
- mongodb
- eyaml
- yaml
:yaml:
:datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"
:mongodb:
:connections:
:dbname: hdata
:collection: config
:host: localhost
:eyaml:
:datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"
:pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
:pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem
:hierarchy:
- "nodes/%{trusted.certname}"
- "location/%{facts.whereami}/%{facts.group}"
- "groups/%{facts.group}"
- "os/%{facts.os.family}"
- "common"
:logger: console
:merge_behavior: native
:deep_merge_options: {}
To
convert this version 3 file to version 5:version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: "Per-node data (yaml version)"
path: "nodes/%{trusted.certname}.yaml" # Add file extension
# Omitting datadir and data_hash to use defaults.
- name: "Per-node data (MongoDB version)"
path: "nodes/%{trusted.certname}" # No file extension
hiera3_backend: mongodb
options: # Use old backend-specific options, changing keys to plain strings
connections:
dbname: hdata
collection: config
host: localhost
- name: "Per-group secrets"
path: "groups/%{facts.group}.eyaml"
lookup_key: eyaml_lookup_key
options:
pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem
- name: "Other YAML hierarchy levels"
paths: # Can specify an array of paths instead of a single one.
- "location/%{facts.whereami}/%{facts.group}.yaml"
- "groups/%{facts.group}.yaml"
- "os/%{facts.os.family}.yaml"
- "common.yaml"
Convert an experimental (version 4) hiera.yaml
to version 5
If you used the experimental version of Puppet lookup (Hiera 5's predecessor), you might have some version 4 hiera.yaml
files in your environments and modules.
Hiera 5 can use these, but you will need to
convert them, especially if you want to use any backends other than YAML or JSON. Version 4 and
version 5 formats are similar.
hiera.yaml
file:# /etc/puppetlabs/code/environments/production/hiera.yaml
---
version: 4
datadir: data
hierarchy:
- name: "Nodes"
backend: yaml
path: "nodes/%{trusted.certname}"
- name: "Exported JSON nodes"
backend: json
paths:
- "nodes/%{trusted.certname}"
- "insecure_nodes/%{facts.networking.fqdn}"
- name: "virtual/%{facts.virtual}"
backend: yaml
- name: "common"
backend: yaml
To convert to version 5, make the following changes:
After being converted to version 5, the example looks like this:
# /etc/puppetlabs/code/environments/production/hiera.yaml
---
version: 5
defaults:
datadir: data # Datadir has moved into `defaults`.
data_hash: yaml_data # Default backend: New feature in v5.
hierarchy:
- name: "Nodes" # Can omit `backend` if using the default.
path: "nodes/%{trusted.certname}.yaml" # Add file extension!
- name: "Exported JSON nodes"
data_hash: json_data # Specifying a non-default backend.
paths:
- "nodes/%{trusted.certname}.json"
- "insecure_nodes/%{facts.networking.fqdn}.json"
- name: "Virtualization platform"
path: "virtual/%{facts.virtual}.yaml" # Name and path are now separated.
- name: "common"
path: "common.yaml"
For full syntax details, see the hiera.yaml
version 5 reference.
Convert experimental data provider functions to a Hiera 5 data_hash
backend
Puppet
lookup had experimental custom backend support, where you could set data_provider = function
and create a function with a name
that returned a hash. If you used that, you can convert your function to a Hiera 5 data_hash
backend.
Updated classic Hiera function calls
The hiera
, hiera_array
, hiera_hash
, and hiera_include
functions are all deprecated. The lookup
function is a complete replacement for all of
these.
Hiera function | Equivalent lookup call |
---|---|
hiera('secure_server')
|
lookup('secure_server') |
hiera_array('ntp::servers')
|
lookup('ntp::servers', {merge =>
unique}) |
hiera_hash('users')
|
lookup('users', {merge => hash}) or
lookup('users', {merge =>
deep})
|
hiera_include('classes')
|
lookup('classes', {merge =>
unique}).include |
To prepare for deprecations in
future Puppet versions, it's best
to revise your Puppet modules to
replace the hiera_*
functions with
lookup
. However, you can adopt
all of Hiera 5's new features
without updating these function calls. While you're revising, consider refactoring
code to use automatic class parameter lookup instead of manual lookup calls. Because
automatic lookups can now do unique and hash merges, the use of manual lookup in the
form of hiera_array
and hiera_hash
are not as important as they
used to be. Instead of changing those manual Hiera calls to be calls to the lookup
function, use Automatic Parameter Lookup
(API).
Adding Hiera data to a module
Modules need default values for their class parameters.
Before, the preferred way to do this was the “params.pp” pattern. With Hiera 5, you can use the “data in modules” approach
instead. The following example shows how to replace params.pp
with the new approach.
params.pp
pattern is still valid, and the features it relies on
remain in Puppet. But if you
want to use Hiera data
instead, you now have that option.Module data with the params.pp
pattern
The params.pp
pattern takes advantage of the Puppet class inheritance behavior.
One class in your module does nothing but set variables for the other
classes. This class is called <MODULE>::params
. This class uses Puppet code to construct values; it uses conditional logic based
on the target operating system. The rest of the classes in the module inherit from the params
class. In their parameter lists, you can use the params class's variables as default
values.
When using params.pp
pattern, the values set in the params.pp
defined class cannot be used in lookup merges and
Automatic Parameter Lookup (APL) - when using this pattern these are only used for defaults
when there are no values found in Hiera.
# ntp/manifests/params.pp
class ntp::params {
$autoupdate = false,
$default_service_name = 'ntpd',
case $facts['os']['family'] {
'AIX': {
$service_name = 'xntpd'
}
'Debian': {
$service_name = 'ntp'
}
'RedHat': {
$service_name = $default_service_name
}
}
}
class ntp (
$autoupdate = $ntp::params::autoupdate,
$service_name = $ntp::params::service_name,
) inherits ntp::params {
...
}
Module data with a one-off custom Hiera backend
With Hiera 5's custom backend system, you can convert and existing params class to a hash-based Hiera backend.
To create a Hiera backend, create a function written in the Puppet language that returns a hash.
params
class as a starting
point:# ntp/functions/params.pp
function ntp::params(
Hash $options, # We ignore both of these arguments, but
Puppet::LookupContext $context, # the function still needs to accept them.
) {
$base_params = {
'ntp::autoupdate' => false,
# Keys have to start with the module's namespace, which in this case is `ntp::`.
'ntp::service_name' => 'ntpd',
# Use key names that work with automatic class parameter lookup. This
# key corresponds to the `ntp` class's `$service_name` parameter.
}
$os_params = case $facts['os']['family'] {
'AIX': {
{ 'ntp::service_name' => 'xntpd' }
}
'Debian': {
{ 'ntp::service_name' => 'ntp' }
}
default: {
{}
}
}
# Merge the hashes, overriding the service name if this platform uses a non-standard one:
$base_params + $os_params
}
hiera.yaml
. A simple backend like this one
doesn’t require path
, datadir
, or options
keys. You have a choice of adding it
to the default_hierarch
if you
want the exact same behaviour as with the earlier params.pp
pattern, and use the regular hierarchy
if you want the values to be merged with
values of higher priority when a merging lookup is specified. You may want to split up the
key-values so that some are in the hierarchy
, and some in the default_hierarchy
, depending on whether it makes sense to merge a value of
not.# ntp/hiera.yaml
---
version: 5
hierarchy:
- name: "NTP class parameter defaults"
data_hash: "ntp::params"
# We only need one hierarchy level, since one function provides all the data.
They do not need to inherit from any other class.
You do not need to explicitly set a default value with the
=
operator.-
Instead APL comes into effect for each parameter without a given value. In the example, the function
ntp::params
will be called to get the default params, and those can then be either overridden or merged, just as with all values in Hiera.# ntp/manifests/init.pp class ntp ( # default values are in ntp/functions/params.pp $autoupdate, $service_name, ) { ... }
Module data with YAML data files
You can also manage your module's default data with basic Hiera YAML files,
hiera.yaml
file:
# ntp/hiera.yaml
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: "OS family"
path: "os/%{facts.os.family}.yaml"
- name: "common"
path: "common.yaml"
# ntp/data/common.yaml
---
ntp::autoupdate: false
ntp::service_name: ntpd
# ntp/data/os/AIX.yaml
---
ntp::service_name: xntpd
# ntp/data/os/Debian.yaml
ntp::service_name: ntp
You can also use any other Hiera backend to provide your module’s data. If you want to use a custom backend that is distributed as a separate module, you can mark that module as a dependency.
For more information, see class inheritance, conditional logic, write functions in the Puppet language, hash merge operator.