Hiera data generation at Swisscom
Hiera data creation for Puppet modules is an onerous task, especially if a module offers a lot of configuration possibilities, and you will use a bunch of configurations.
Swisscom (Switzerland) Ltd., a telecommunication and IT service provider, offers its enterprise customers different cloud solutions, and builds a custom cloud for each client. To build custom clouds on a competitive basis requires a high degree of automation of all components.
Swisscom relies on Puppet, along with Hiera data, as a configuration and automation platform. As you can imagine, a lot of different servers, components and therefore Puppet modules are needed to build a cloud. That means putting a lot of time and effort into creating the Hiera data. To build a whole new cloud stack, an engineer at Swisscom needed two working days to prepare all the configurations, and without any guarantee that the created Hiera data would be free of mistakes.
To get more stability in the Hiera data creation process, and to avoid manually-created mistakes in the configuration files, Swisscom automation architect Thomas von Steiger pursued the idea of automatically creating Hiera data with a generator, based on a predefined source. A prototype of Hiera data generation was created within the scope of a university project, with the focus on creating Hiera data for Arista switches. In January this year, the real implementation of Hiera data generation started within a new project.
The project was a success. Automated generation of Hiera data with Hieragen and Puppet now takes about one minute, and it takes about 10 minutes to create the source file. We've regained 99 percent of the time it used to take when we generated Hiera data manually. Read on for how it works.
The Hiera data generator prototype, called Hieragen, is written in Ruby and consists a few classes and embedded Ruby (ERB) templates. The predefined ERB templates are used to create the final YAML file for Hiera data. Every Puppet module has its own ERB template.
The generator collects all the specific configuration data from a predefined data source, buffers it to a hash, and creates valid YAML files from the ERB templates. Finally, all the generated Hiera data files are uploaded to the internal GitLab instance, from which they can be cloned and used by the Puppet master.
A configuration option allows the user to do small configurations on the generator over command line — for example, to specify the data source or to add the Git repository.
Hieragen is packaged as an RPM and can be installed on every version of Red Hat Enterprise Linux. The code sequences below will show you how Hieragen works on a server.
The installation is pretty simple: It's done by installing the two RPMs, one for the Hieragen source and the other for the different Hiera data templates.
bash [root@puppetmaster]# yum install ruby193-hieragen hieragen-templates
Before the first run, the configuration is needed to specify different locations.
bash [root@puppetmaster]# hieragen configure Configuration already exists, would you like to reconfigure? (Y,N) Y Please enter the source location: /etc/hieragen/input/testdata.json Please enter the log file path: /var/log/ Please enter the output location: /etc/hieragen/output/ Please enter the gitrepository: https://git.testserver.com/hieradata/puppetmaster.git Your source location is /etc/hieragen/input/testdata.json log file path is /var/log/ output location is /etc/hieragen/output/ gitrepository is https://git.testserver.com/hieradata/puppetmaster.git ################################ successfully saved to file. see /etc/hieragen/hieragen.conf ################################
And Hieragen starts with the run command.
bash [root@puppetmaster]# hieragen run Starting Data Collection … Data Collection successfully finished … Starting YAML File Creation … Starting save Data for /etc/hieragen/output/nodes/torswitch-1.testdomain.com.yaml Starting render and load YAML … Starting save Data for /etc/hieragen/output/nodes/torswitch-2.testdomain.com.yaml Starting render and load YAML ... Starting save Data for /etc/hieragen/output/nodes/spineswitch-1.testdomain.com.yaml Starting render and load YAML ... Starting save Data for /etc/hieragen/output/nodes/spineswitch-2.testdomain.com.yaml Starting render and load YAML ... Starting save Data for /etc/hieragen/output/nodes/mgmtswitch-1.testdomain.com.yaml Starting render and load YAML ... Starting save Data for /etc/hieragen/output/nodes/mgmtswitch-2.testdomain.com.yaml Starting render and load YAML ... Process finished ... Successfully Uploaded to git.
Without Hiera data automation, scaling stack deployments would simply not be conceivable. We learned a lot, and got the idea for how to implement the real Hiera data generator.
The final developed product, called Stackgear, is for professional use, and completely meets Swisscom's needs. At the end of the day, as soon as Stackgear is fully implemented, Swisscom will save a lot of time on preparation work, both during the initial stack deployment and during every upgrade or change on a cloud stack. Just as a reminder, we went from two days to 11 minutes to build the configurations for a new cloud stack.
Within the Stackgear generator and its templates packaged as an RPM, it's now possible to do continuous integration and continuous delivery of the Hiera data, as well as Puppet modules and software packages. With this streamlined workflow, Swisscom can scale to more customers efficiently, and with a minimum of errors.
Raffaela Wuest is an application engineer at Swisscom.
- If you are interested in more details, the whole project documentation about Hieragen is available in German : PDF
- Not familiar with Hiera? Read the blog post, Separation of Concerns: How Hiera Helps.
- Learn how to build more generic Hiera hierarchies plus the custom facts your organization needs.
- Learn about debugging Hiera.
- Get into Hiera by skipping the hierarchy.