Upgrading to Hiera 5
Upgrading to Hiera 5 offers some major advantages. A real environment data layer means changes to your hierarchy are now routine and testable, using multiple backends in your hierarchy is easier and you can make a custom backend.
Hiera 5 uses the same built-in data formats as Hiera 3. You don't need to do mass edits of any data files.
Updating your code to take advantage of Hiera 5 features involves the following tasks:
Task | Benefit |
---|---|
Enable the environment layer,
by giving each environment its own hiera.yaml file.
Note: Enabling the environment layer takes the most work,
but yields the biggest benefits. Focus on that first, then do the
rest at your own pace.
|
Future hierarchy changes are
cheap and testable. The legacy Hiera functions (hiera , hiera_array ,
hiera_hash , and hiera_include ) gain full Hiera 5 powers in any migrated
environment, only if there is a hiera.yaml in the environment root. |
Convert your global hiera.yaml
file to the version 5 format. |
You can use new Hiera 5 backends at the global layer. |
Convert any experimental
(version 4) hiera.yaml files to version 5. |
Future-proof any environments or modules where you used the experimental version of Puppet lookup. |
In Puppet code, replace legacy Hiera
functions (hiera , hiera_array , hiera_hash , and hiera_include ) with lookup() . |
Future-proof your Puppet code. |
Use Hiera for default data in modules. | Simplify your modules with an
elegant alternative to the "params.pp " pattern. |
Considerations for hiera-eyaml users
Upgrade now. In Puppet 4.9.3, we added a built-in
hiera-eyaml backend for Hiera 5. (It still requires that the hiera-eyaml
gem be installed.) See the usage instructions in the
hiera.yaml
(v5) syntax reference. This means you
can move your existing encrypted YAML data into the environment layer at the same
time you move your other data.
Considerations for custom backend users
Wait for updated backends. You can keep using custom Hiera 3 backends with Hiera 5, but they'll make upgrading more complex, because you can't move legacy data to the environment layer until there's a Hiera 5 backend for it. If an updated version of the backend is coming out soon, wait.
If you're using an off-the-shelf custom backend, check its website or contact its developer. If you developed your backend in-house, read the documentation about writing Hiera 5 backends.
Considerations for custom data_binding_terminus
users
Upgrade now, and replace it with a Hiera 5 backend as
soon as possible. There's a deprecated data_binding_terminus
setting in the puppet.conf
file, which changes the behavior of
automatic class parameter lookup. It can be set to hiera
(normal),
none
(deprecated; disables auto-lookup), or the name of a
custom plug-in.
With a custom data_binding_terminus
, automatic lookup results are
different from function-based lookups for the same keys. If you're one of the few
who use this feature, you've already had to design your Puppet code to avoid that problem, so it's safe to
upgrade your configuration to Hiera 5. But because
we've deprecated that extension point, you have to replace your custom terminus with
a Hiera 5 backend.
If you're using an off-the-shelf plug-in, such as Jerakia, check its website or contact its developer. If you developed your plug-in in-house, read the documentation about writing Hiera 5 backends.
After you have a Hiera 5 backend, integrate it into
your hierarchies and delete the data_binding_terminus
setting.
Enable the environment layer for existing Hiera data
A key feature in Hiera 5 is
per-environment hierarchy configuration. Because you probably store data in each environment,
local hiera.yaml
files are more logical and
convenient than a single global hierarchy.
You can enable the environment layer gradually. In migrated environments, the legacy Hiera functions switch to Hiera 5 mode — they can access environment and module data without requiring any code changes.
hiera-eyaml
, continue upgrading — Puppet 4.9.3 and
higher include a Hiera 5 eyaml backend. See the usage
instructions in the hiera.yaml
(v5) syntax reference.In each environment:
For more information on mapping environments to branches, see control repo.
Convert a version 3 hiera.yaml
to version 5
Hiera 5 supports three versions of the hiera.yaml
file: version 3, version 4, and version 5. If
you've been using Hiera 3, your existing configuration is a
version 3 hiera.yaml
file at the global
layer.
-
Creating new v5
hiera.yaml
files for environments. -
Updating your global configuration to support Hiera 5 backends.
These are essentially the same process, although the global hierarchy has a few special capabilities.
hiera.yaml
version 3 file:
:backends:
- mongodb
- eyaml
- yaml
:yaml:
:datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"
:mongodb:
:connections:
:dbname: hdata
:collection: config
:host: localhost
:eyaml:
:datadir: "/etc/puppetlabs/code/environments/%{environment}/hieradata"
:pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
:pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem
:hierarchy:
- "nodes/%{trusted.certname}"
- "location/%{facts.whereami}/%{facts.group}"
- "groups/%{facts.group}"
- "os/%{facts.os.family}"
- "common"
:logger: console
:merge_behavior: native
:deep_merge_options: {}
To
convert this version 3 file to version 5:version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: "Per-node data (yaml version)"
path: "nodes/%{trusted.certname}.yaml" # Add file extension
# Omitting datadir and data_hash to use defaults.
- name: "Per-node data (MongoDB version)"
path: "nodes/%{trusted.certname}" # No file extension
hiera3_backend: mongodb
options: # Use old backend-specific options, changing keys to plain strings
connections:
dbname: hdata
collection: config
host: localhost
- name: "Per-group secrets"
path: "groups/%{facts.group}.eyaml"
lookup_key: eyaml_lookup_key
options:
pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem
- name: "Other YAML hierarchy levels"
paths: # Can specify an array of paths instead of a single one.
- "location/%{facts.whereami}/%{facts.group}.yaml"
- "groups/%{facts.group}.yaml"
- "os/%{facts.os.family}.yaml"
- "common.yaml"
Convert an experimental (version 4) hiera.yaml
to version 5
If you used the experimental version of Puppet lookup ( Hiera 5's
predecessor), you might have some version 4 hiera.yaml
files in your environments and modules. Hiera 5 can use these, but you need to convert them, especially if
you want to use any backends other than YAML or JSON. Version 4 and version 5 formats are
similar.
hiera.yaml
file:# /etc/puppetlabs/code/environments/production/hiera.yaml
---
version: 4
datadir: data
hierarchy:
- name: "Nodes"
backend: yaml
path: "nodes/%{trusted.certname}"
- name: "Exported JSON nodes"
backend: json
paths:
- "nodes/%{trusted.certname}"
- "insecure_nodes/%{facts.networking.fqdn}"
- name: "virtual/%{facts.virtual}"
backend: yaml
- name: "common"
backend: yaml
To convert to version 5, make the following changes:
After being converted to version 5, the example looks like this:
# /etc/puppetlabs/code/environments/production/hiera.yaml
---
version: 5
defaults:
datadir: data # Datadir has moved into `defaults`.
data_hash: yaml_data # Default backend: New feature in v5.
hierarchy:
- name: "Nodes" # Can omit `backend` if using the default.
path: "nodes/%{trusted.certname}.yaml" # Add file extension!
- name: "Exported JSON nodes"
data_hash: json_data # Specifying a non-default backend.
paths:
- "nodes/%{trusted.certname}.json"
- "insecure_nodes/%{facts.networking.fqdn}.json"
- name: "Virtualization platform"
path: "virtual/%{facts.virtual}.yaml" # Name and path are now separated.
- name: "common"
path: "common.yaml"
For full syntax details, see the hiera.yaml
version 5 reference.
Convert experimental data provider functions to a Hiera 5 data_hash
backend
Puppet lookup had experimental
custom backend support, where you could set data_provider = function
and create a function with a name that returned a hash. If you
used that, you can convert your function to a Hiera 5 data_hash
backend.
Updated classic Hiera function calls
The hiera
, hiera_array
, hiera_hash
, and hiera_include
functions are all deprecated. The lookup
function is a complete replacement for all of
these.
Hiera function | Equivalent lookup call |
---|---|
hiera('secure_server')
|
lookup('secure_server') |
hiera_array('ntp::servers')
|
lookup('ntp::servers', {merge =>
unique}) |
hiera_hash('users')
|
lookup('users', {merge => hash}) or
lookup('users', {merge =>
deep})
|
hiera_include('classes')
|
lookup('classes', {merge =>
unique}).include |
To prepare for deprecations in
future Puppet versions, it's best to revise your Puppet modules to replace the hiera_*
functions with lookup
. However, you can adopt all of Hiera 5's new features without updating these
function calls. While you're revising, consider refactoring code to use automatic
class parameter lookup instead of manual lookup calls. Because automatic lookups can
now do unique and hash merges, the use of manual lookup in the form of hiera_array
and hiera_hash
are not as important as they used to
be. Instead of changing those manual Hiera calls to
be calls to the lookup
function,
use Automatic Parameter Lookup (API).
Adding Hiera data to a module
Modules need default values for their class parameters.
Before, the preferred way to do this was the “params.pp” pattern. With Hiera 5, you can use the “data in modules” approach instead.
The following example shows how to replace params.pp
with the new approach.
params.pp
pattern is still valid, and the
features it relies on remain in Puppet. But if
you want to use Hiera data instead, you now have
that option.Module data with the params.pp
pattern
The params.pp
pattern takes advantage of the Puppet class
inheritance behavior.
One class in your module does nothing but set variables for the other
classes. This class is called <MODULE>::params
. This class uses Puppet code
to construct values; it uses conditional logic based on the target operating system. The rest
of the classes in the module inherit from the params class. In their parameter lists, you can
use the params class's variables as default values.
When using params.pp
pattern, the values set in the params.pp
defined class cannot be used in lookup merges and
Automatic Parameter Lookup (APL) - when using this pattern these are only used for defaults
when there are no values found in Hiera.
# ntp/manifests/params.pp
class ntp::params {
$autoupdate = false,
$default_service_name = 'ntpd',
case $facts['os']['family'] {
'AIX': {
$service_name = 'xntpd'
}
'Debian': {
$service_name = 'ntp'
}
'RedHat': {
$service_name = $default_service_name
}
}
}
class ntp (
$autoupdate = $ntp::params::autoupdate,
$service_name = $ntp::params::service_name,
) inherits ntp::params {
...
}
Module data with a one-off custom Hiera backend
With Hiera 5's custom backend system, you can convert and existing params class to a hash-based Hiera backend.
To create a Hiera backend, create a function written in the Puppet language that returns a hash.
params
class as a starting point:# ntp/functions/params.pp
function ntp::params(
Hash $options, # We ignore both of these arguments, but
Puppet::LookupContext $context, # the function still needs to accept them.
) {
$base_params = {
'ntp::autoupdate' => false,
# Keys have to start with the module's namespace, which in this case is `ntp::`.
'ntp::service_name' => 'ntpd',
# Use key names that work with automatic class parameter lookup. This
# key corresponds to the `ntp` class's `$service_name` parameter.
}
$os_params = case $facts['os']['family'] {
'AIX': {
{ 'ntp::service_name' => 'xntpd' }
}
'Debian': {
{ 'ntp::service_name' => 'ntp' }
}
default: {
{}
}
}
# Merge the hashes, overriding the service name if this platform uses a non-standard one:
$base_params + $os_params
}
hiera.yaml
. A simple backend like this one doesn’t require path
, datadir
, or options
keys. You have a choice of adding it to the default_hierarch
if you want the exact same behaviour
as with the earlier params.pp
pattern, and use the regular hierarchy
if you want the values to be merged with values of higher priority when
a merging lookup is specified. You can split up the key-values so that some are in the hierarchy
, and some in the default_hierarchy
, depending on
whether it makes sense to merge a value or not.# ntp/hiera.yaml
---
version: 5
hierarchy:
- name: "NTP class parameter defaults"
data_hash: "ntp::params"
# We only need one hierarchy level, because one function provides all the data.
They do not need to inherit from any other class.
You do not need to explicitly set a default value with the
=
operator.-
Instead APL comes into effect for each parameter without a given value. In the example, the function
ntp::params
is called to get the default params, and those can then be either overridden or merged, just as with all values in Hiera.# ntp/manifests/init.pp class ntp ( # default values are in ntp/functions/params.pp $autoupdate, $service_name, ) { ... }
Module data with YAML data files
You can also manage your module's default data with basic Hiera YAML files,
hiera.yaml
file:
# ntp/hiera.yaml
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: "OS family"
path: "os/%{facts.os.family}.yaml"
- name: "common"
path: "common.yaml"
# ntp/data/common.yaml
---
ntp::autoupdate: false
ntp::service_name: ntpd
# ntp/data/os/AIX.yaml
---
ntp::service_name: xntpd
# ntp/data/os/Debian.yaml
ntp::service_name: ntp
You can also use any other Hiera backend to provide your module’s data. If you want to use a custom backend that is distributed as a separate module, you can mark that module as a dependency.
For more information, see class inheritance, conditional logic, write functions in the Puppet language, hash merge operator.