Published on 12 January 2016 by

Configuring Puppet is hard. Not only are there a lot of different configuration files, but they use a lot of different formats, making these files hard to manage with Puppet itself. In an effort to make everyone’s life easier, we decided to standardize on the HOCON format.

HOCON, or “Human-Optimized Config Object Notation,” is a JSON superset configuration format that is currently used by all of Puppet Labs’ Clojure apps, including Puppet Server and trapperkeeper-authorization. We considered a lot of configuration formats, but decided on HOCON as it provides some compelling advantages. As a superset of JSON, HOCON supports simple data structures such as maps and arrays. It is also a lot less finicky and strict than JSON, meaning that it’s easier to read and write by hand. Furthermore, it fully supports inline comments, allowing us to add documentation to the config files themselves.

Previously, despite standardizing on the format, there was no tooling for managing these files as native Puppet resources. HOCON configuration files either had to be managed by hand, or managed by Puppet using templates. But now there’s a better way: the puppetlabs-hocon module!

This module provides a simple resource type, hocon_setting, that allows you to manage individual settings within a HOCON file while preserving that file’s whitespace, formatting, and comments. No more templates, no more management by hand – just a simple, easy way to model and enforce your HOCON configuration. This provides a lot of value, as it not only makes managing configuration settings much easier, it also allows you to test your config changes, since settings are now managed individually through Puppet. It also provides forward compatibility with new versions of Puppet and the HOCON gem.

The HOCON module is powerful, and has a lot of cool features. These include:

  • The ability to manage individual settings in your HOCON configuration file without having to explicitly manage the entire file. The module can manage top-level settings, but it can also manage settings nested within HOCON maps that are multiple levels deep.
  • The ability to add non-existent settings into your configuration file. HOCON can add non-existent top-level settings, but it can also add non-existent nested settings. And if the setting you want is within a map that does not yet exist, the module will create the non-existent parent map, as well as the setting you specify.
  • The ability to add settings to non-existent configuration files. If you are adding a setting to a configuration file that does not exist on disk, that file will be created.
  • Full preservation of whitespace, formatting, and comments.
  • The ability to manage values of all types, whether simple values like strings and integers or complex data structures like hashes and arrays.
  • The ability to manage individual elements within an array.

Installation

Installation of the HOCON module takes only two steps. First, you must install the hocon gem, which powers the HOCON module. After that, you can install the module using the Puppet Module Tool:

$ gem install hocon
Fetching: hocon-0.9.3.gem (100%)
Successfully installed hocon-0.9.3
Parsing documentation for hocon-0.9.3
Installing ri documentation for hocon-0.9.3
1 gem installed

$ puppet module install puppetlabs-hocon
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/etc/puppetlabs/code/environments/production/modules
└── puppetlabs-hocon (v0.9.3)

The HOCON gem can also be installed using Puppet like so (though the module will still need to be installed with the PMT tool):

package { ‘hocon’:
  ensure   => ‘present’,
  provider => ‘gem’,
}

The HOCON module fully supports Puppet 4, which is included in Puppet Enterprise 2015.2 and above. The module can be used with Puppet 3, included in the PE 3.x series, but some values may not be inserted properly into configuration files, and may end up being inserted as strings.

The HOCON module is not included with Puppet Enterprise, and will need to be installed separately.

Resource overview

The HOCON module provides one resource type: hocon_setting. With this type, you can specify a file, setting, value, and (optionally) a value type, and Puppet will ensure that that setting is correctly set inside your configuration file with the value you specified.

Example use cases

The HOCON module is pretty powerful, and can do some really cool things.

Let’s say you start with a config file at /tmp/foo.conf with the following contents:

foo : 123 
baz : {
  qux : 12
}

In the most basic case, you could add a top-level setting to your configuration file like so:

hocon_setting { “sample setting”:
  ensure => present,
  path     => ‘/tmp/foo.conf’,
  setting => ‘foosetting’,
  value   => ‘FOO!’,
}

After the resource is applied, your resulting config will look like this:

foo : 123
baz : {
  qux : 12
}
foosetting : “FOO!”

But let’s say you’re feeling a little bit adventurous. Instead of setting a top-level setting, you want to set the qux setting that lives inside the map at baz. To do that, you could do the following:

hocon_setting { “sample setting”:
  ensure => present,
  path     => ‘/tmp/foo.conf’,
  setting => ‘baz.qux’,
  value   => “BAZQUX!”,
}

The resulting configuration would look like this:

foo : 123
baz : {
  qux : “BAZQUX!”
}

Let’s say you want to go even further. What if you want a top level setting, bar, with its value being a map? That map contains a single setting, bar2, which itself contains a map with a single setting, bar3. You want to set the value of bar3, but bar and bar2 don’t exist. How do you set bar3?

Easy! You just do the following:

hocon_setting { “sample setting”:
  ensure => present,
  path     => ‘/tmp/foo.conf’,
  setting => ‘bar.bar2.bar3’,
  value   => “This is bar3”
}

You would then end up with the following configuration:

foo : 123
baz : {
  qux : 12
}
bar : {
  bar2 : {
    bar3 : “This is bar3”
  }
}

As you can see, even though bar and bar2 are maps that don’t exist in the initial config, the HOCON module will create them and nicely format them.

Up to this point, we’ve only been inserting strings. However, the HOCON module and HOCON itself also support numbers and booleans. In addition to these types, HOCON also supports insertion of complex data structures — specifically, arrays and hashes.

If you wanted to insert an array, you could do the following:

hocon_setting { “sample setting”:
  ensure => present,
  path     => ‘/tmp/foo.conf’,
  setting => ‘bar’,
  value   => [1, 2, 3, 4, 5]
}

You’d end up with a configuration like the following:

foo : 123
baz : {
  qux : 12
}
bar : [1, 2, 3, 4, 5]

The HOCON module also supports inserting hashes. Say you had the following resource:

hocon_setting { “sample setting”:
  ensure => present,
  path     => ‘/tmp/foo.conf’,
  setting => ‘bar’,
  value   => { ‘a’ => ‘b’ }
}

You’d end up with something like the following:

foo : 123
baz : {
  qux : 12
}
bar : {
  ‘a’ : ‘b’
}

We’ve covered a lot of the use cases up to this point, but there’s one parameter we haven’t touched on: the type parameter. In the HOCON module, you can optionally use the type parameter to specify the type of the value you’re setting. The type of this parameter can be number, boolean, string, hash, array, array_element, or text, and has to match the type of the value parameter.

In most cases, the module is smart enough to figure out this information on its own, but this parameter is necessary in three cases. The first is when inserting a single-element array. All values are passed to the HOCON module as arrays, so it needs a way to distinguish when it needs to insert a single-element array or a non-array value. You could do the following:

hocon_setting { “sample setting”:
  ensure => present,
  path     => ‘/tmp/foo.conf’,
  setting => ‘bar’,
  value   => [1],
  type     => ‘array’,
}

And the value [1] would be inserted in your configuration under the bar setting. If array was not specified for the type, 1 would be inserted instead of [1].

The second use case is using the type value to directly insert text into a configuration file using the text type, which is useful if you want to insert a map with comments and specific formatting. To do this, simply set the value to the string you want to insert, and set the type to text.

The final use case is managing individual elements in an array. To do this, ensure that the value of setting points to a setting with an array value, and set the type to array_value. This will then insert the value of the value parameter into the array at the specified setting.

Tests

The HOCON module comes with two types of tests. The first are the smoke tests, which live under the tests directory and can be run with puppet apply. These provide a good example of the module’s basic usage, and can be used to experiment with the module.

The other provided tests are the spec tests, which live under the spec directory. These can be run by running rake spec in the root of the module directory. These tests are a good indicator of the module’s expected behavior.

Conclusion

While it only provides a single resource type, the HOCON module provides you with a lot of functionality and flexibility in the way you handle HOCON config files, so you’ll never need to use a template or make changes by hand again. It’s already been immensely useful here at Puppet Labs, where we use it to manage Puppet Enterprise configuration. We also use it in the authorization module to manage your auth.conf files. It gives you a safer, more consistent, and easier way to manage your configurations, one that can be tested and provides you forward compatibility with any future changes made to HOCON.

Module issues can be filed at the HOCON Jira Project. For questions, use the puppet-dev mailing list or the #puppet-dev IRC channel.

Preben Ingvaldsen is a software engineer at Puppet Labs.

Learn More:

Share via:
Posted in:
Tagged:
The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.