The Benefits of Using Hiera Data Manager

00:00:18Demetrius MalbroughHey, everyone, thanks for joining this episode of Pulling the Strings podcast powered by Puppet, and I'm delighted to be your host. My name is Demetrius Malbrough and I'm on the product marketing team here at Puppet. I am really excited today to talk with Martin Alfke. Martin has been using Puppet since 2007. Also since 2011, Martin conducts the official training in Germany, so he is the co-founder and CEO at example42. Martin supports companies and individuals and having success with IT automation based on Puppet. He also collaborates within the Vox Pupuli community and co-maintains the approved example42 modules. He also holds the Puppet certified professional and the Puppet certified consultant certification so he knows his Puppet. All right. So, Martin, welcome to Pulling the Strings. How are you today?

00:01:27Martin AlfkeThanks. I'm really doing great.

00:01:28Demetrius MalbroughWell, let's go ahead and get started. Let's start off really slow here just by explain to us what is Hiera Data Manager.

00:01:40Martin AlfkeYeah, HDM or Hiera Data Manager, who we call it, is one of the example42 products that we build around Puppet. So at the moment we maintain, as you already mentioned or approve, Puppet modules on the Forge. We have super generic control repository that we make available to the public and we have a specific module called Tiny Puppet. So our newest product is our Hiera Data Manager. In short, we call it HDM. HDM is a web application which helps people dealing with Hiera data. It heavily reduces time needed for data analysis and setting, HDM provides features which are not available either for Puppet open source or even Puppet enterprise. But to get a better understanding on what HDM does, I first need to explain what Hiera itself is. Hiera is a data store which is built into Puppet. Within Hiera one does provide data to Puppet code, which reflects differences among an IT platform. So when you usually have IT platforms, you have differences in configuration. That's not something weird or special. This is something we see in every IT infrastructure. To give you some examples, you might have differences in server configuration. For example, the servers in the first datacenter need another backup or DNS server compared to servers in another data center. Or when you had an application level, you might have an application and integration stage and you might have an integration and application and production stage and each need different database servers configured or they are deployed in different versions or in different settings. So changes and differences in IT infrastructures. This is something that is very common, that is not a problem, but it's a requirement that any configuration management tool has to deal with and Puppet deals with these differences by using or offering the possibilities of Hiera. So the question is not where you put all these information about differences in configuration and Puppet offers you just saying we have a hierarchical data structure where you can place in these differences. So Hiera allows us to separate code from data that has a huge impact also on CI platforms. We no longer need to add data processing into Puppet code, but we can place the data outside of Puppet code, Puppet is automagically -- I really love that term automagically -- it's not matching and it's just happening in the background automatically. And Puppet carries Hiera for data whenever it needs data. So I already mentioned this built in into Puppet. So if we want to change a setting, we no longer must change code as we change data only. So this has an impact. For example, for CI/CD, we don't need to test the whole code base anymore. We only must test that the data are still valid even after the change. Within Hiera you try to build hierarchical data structure for Puppet. So you built something like Hiera keys. These Hiera keys are usually based on Puppet facts and these Puppet facts are identifiers for different configuration. So usually you're the first level. You start with something like a common level. So there you add data where you say: most of the systems have these data, and on top of this common layer, you have the possibility to add additional layers where you can override or add data on a specific level. What kind of facts is it that you can use for Hiera? Well, this might be something like in which stage is the server running and is it a development, testing or production system? You can make differences based, for example, on which data center is the system running on or in which networks. And you can go up to the level where you say there is a certain application running on a system and maybe you need a combination of application in development state is different to an application and production state.  On top, you usually say there's always one system that is slightly different and this gives you a node specific Hiera data layer. All of these information is stored either in YAML or JSON files, and it's just key value pairs. So the key on the left hand side is always a string. On the right hand side you have the data, the value. That sounds super easy to do. It is easy to do. So what is the problem, though? Why do we need HDM? The biggest problem that we see is when people try to identify which data are taken for a specific node within specific application from which Hiera level and which data are active and what value is used. And are there other layers of Hiera where we have data inside and this is where HDM comes into place. It's an open source web application that is capable of showing data available for a node. It visualizes all of your Hiera layers. It checks whether a layer exists. It checks whether there is contents or whether the data are in this layer. And it shows you visualizes which data are finally used by Puppet for that specific node. Just by having this Web interface as a visualization, the web interface eliminates the time needed for Hiera data analysis dramatically. So, we saw people digging five to 10 minutes, for example, to find out which data are taken from a node by using other things, by digging through the code base, digging through all the files, which in Hiera it's three clicks it's one minute and you see all the data. But it's not only about visualization. We also want with HDM to allow people to do changes directly via a web interface.

00:07:29Demetrius MalbroughOK, well thank you for that very thorough summary of what HDM is. I am definitely more clear now on HDM and thank you for explaining that. So I guess what were some of the difficulties that people were having before HDM even existed, Martin?

00:07:46Martin AlfkeFor some companies, they find Puppet difficult. Others find it not difficult. So many people see difficulties in many places. So the first thing we were saying, we need to make things as simple as possible. So one, for example, a topic that we see uncovered by Hiera is the responsibility and access layer. So who's allowed to view data? Who's allowed to change data? This is all something that is not built into Hiera because Hiera itself is just plain files. There is no access control for Hiera.  Puppet reads data, and somebody supplies data. So the question now is who needs to change or view which kind of data for which nodes, in which data center? So it's more like a responsibility. A pattern that we're trying to implement. What is a difficulty? So we don't want to make all the data public to everywhere and everyone in the company. We want to limit access to the data. So when it comes to Puppet developments, you usually have some kind like Puppet infrastructure developers. When you have to manage an application state with Puppet, you're no longer dealing just with these infrastructure developers. You have people being responsible for an application and these people want to remain responsible for the application and these people want to change their applications. And maybe it's even a hard requirement that only an application admin is allowed to change any data for an application. So this is more like a separation of concerns and maybe from a security perspective. This separation of concerns that allows developers, infrastructure developers, to concentrate on their work doing Puppet code. And on the other hand, you have application administrators that should have the possibility to independently on the work of the Puppet infrastructure developers, to change data. So what we see as several customers or even within the training courses, people who come to Puppet trainings, mostly from an admin perspective, even from the customer side, the people we collaborate with are system administrators. Some of them have heard already about Git, some of them have never used it before. So GitHub is a version control system heavily used by developers, application developers. It's their day to day work and application admins knows his application in detail, but maybe they have never learned Git before. They have never used it before and they lack the time of learning it. So the application administrators understand how to maintain and change the application with the least impact. And now you're coming into the company and saying, hey, we're doing DevOps principal roles is fairly simple. Everybody must just use the same developer tools as developers use and this includes Git, and someone has never worked before with Git. This is a pain. People have to learn about how to use it, how it is working. They have to configure specific settings. This takes time and it needs people's time and motivation. What we see at customers, the work that application administrators are doing is that they are fully busy maintaining their application. Maybe they have several applications, maybe they have other projects they have to take care of. And on top of that, they have to react to something like monitoring incidents, some alerts or stuff like that. So these people are definitely the people that say, can't we have a more simple or more simple way to visualize and even modify Hiera data? And that's the point where HDM kicks in. It's a web interface. You don't need knowledge on Git. You just click by choosing the Puppet environment. You select a node, within a second you see all the data keys that are available in Hiera. By clicking on a data key. You just see which Hiera keys are there, which Hiera keys have data and which is the data that are taken into account for that selected node. Let's say it's a one, two, three process. First, select the Puppet environment. Second, select the node where you want to visualize the data for. And third, select the key and now you have all the information that you need.

00:12:07Demetrius MalbroughOK, yeah, thanks for walking us through that process. It seems like there are a few options that you have there. You mentioned developers. And one thing that that I've learned just in my six months here at Puppet, is that developers like multiple options and multiple ways of doing things, especially like when they can go to their keyboard and start pounding out commands and things like that. So is there another way to do it, Martin? Like the command line?

00:12:35Martin AlfkeYeah, there is a command line tool. It's the Puppet look up command. This already gives you some support in that direction. But you must consider that this command must be run by a privileged user, the route user, directly on a Puppet [server]. So what we usually see at customers is that from a security perspective, they want to limit access to the Puppet [server], the heart of their automation, to as few people as possible. So this means now, either that the Puppet infrastructure developer or Puppet admins must run the command on behalf of an application admin, sending him the email. See, this is the output. So this causes additional workload on the Puppet admins. And we have the same problem here. It's not only that the take care of Puppet, they might have different things like storage, like logging, so they have additional work to do. Since you mentioned your six months with Puppet, you missed last year's Puppetize event in Portland, Oregon. It was a really great event. And within the keynote, your CEO said a great thing over there, which is what Puppet would like to take care that people that work on IT get rid of soul crushing work. So doing the same things over and over again and not getting any benefit out of this. So the mentioned process like now an Puppet application admin, sends an email to a Puppet admin: please give me the information for that node, on that data. The Puppet admin runs the command. This is not elimination. This increases soul crushing work. And besides this, to be honest, reading the output from the Puppet look up command is super hard because it's multiple lines and you understand the content of all the lines and you just don't run the Puppet “look up” command to get an understanding which Hiera keys are there in which data are taken, you must use for getting these information, you must use the same parameter to that command.

00:14:37Demetrius MalbroughYeah, I do remember the 2019 Puppetize keynote from Yvonne where she did mention soul crushing work. It definitely resonates with me because you know, I've been in the IT industry for over 20 years and I've definitely had my share of soul crushing work tasks back in the day when I had to wear a pager and I was on call when a system went down and I had to actually I was responsible of recovering that system, so it was definitely so crushing and I had to work on Christmas one day, which was even more soul crushing. So do you have any, I just mentioned some of my war stories, right. So, Martin, do you have any war stories of your own that HDM helped overcome some challenges around?

00:15:26Martin AlfkeWell, let's say it this way. When someone works for 20 years in IT and does not have a war story, that's definitely something wrong. So I especially have a war story specific for this use case. It was one of the most severe impacts on IT infrastructure I've ever seen before. This all happened prior we had the idea for developing HDM. Just to give you a first understanding, it was caused by incorrect YAML Hiera data. So we had a customer and the customer was also managing the IBM management solution by Puppet. So one day they changed, they introduced a small unintended change to the YAML data structure. They had a review process and no one took care of the single missing whitespace character in the review process in the YAML files.

00:16:20Demetrius MalbroughHold on. You got to stop right there, Martin. So you said like one missing white space character brought this on?

00:16:28Martin AlfkeYes, one missing white space character that changed the Hiera data. So the Hiera data in YAML and it's like Python. So a white space has a specific meaning to it. And you must apply these changes properly when you deal with YAML. That's not the problem where you deal with JSON data. JSON data are fully described, but YAML uses white spaces to identify whether something is a string or an array or whether it becomes a hash. So they had a review. They did the merge process. They merged everything in production and Puppet did what it was supposed to do based on the data. So in this case, the data meant that the virtualization machine should not have any running VM. It should not have any network. It should not have any kind of virtualization systems in the background where it can deploy virtual machines. So Puppet removed all 1,200 running VMs, which were specifically for a build process of the company within seconds. This included the Puppet [server], the GIT server, the VICI, the ticket system. Luckily the backup server was still barebones or bare hardware, so they were able to re spin up the system by using their backup system. But it took three days to restore everything. They had 100 developers, 50 permanent employees, 50 freelancers, and just doing some rough calculation, that's almost 60,000 Euros per day, let's say $50,000 US dollars per day where people were just sitting there and they couldn't do anything, nothing, no ticket system. They could just do some reduced springform, like to do a review, and planning was not possible, and maybe they just did some reviews.

00:18:24Demetrius MalbroughSo it sounds like it wasn't Puppet fault. It was the white space character.

00:18:30Martin AlfkeIt was a white space character fault. No one took care of it, nobody was interested in it. And this case showed it.

00:18:37Demetrius MalbroughYeah, I think we have the name of this podcast episode: "Blame it on the white space character."

00:18:45Martin AlfkeIt was not a DNS problem, it was a white space problem. Yes. So this case showed us that analysis of data structure is an essential topic when it comes to Hiera. And this is also one of the core features that we would like to see to have an HDM. So HDM can then just refuse storing false data structure or incompatible data structure.

00:19:08Demetrius MalbroughYeah, that was definitely a war story that I haven't heard one quite at that scale before.

00:19:18Martin AlfkeI've never seen another at that scale.

00:19:18Demetrius MalbroughI'm sure that someone may have gotten a pink slip that day or may have had a slap on the wrist.

00:19:29Martin AlfkeLet's say it this way: nobody was very happy, but everybody said we're dealing with people and we're dealing with innovative people and innovative people can do errors. So I was very happy that nobody was blamed for doing this. Everybody took it seriously and they tried to to provide actions to not have this happen again.

00:19:46Demetrius MalbroughOK, yeah, that that's fair enough. We are humans and we make mistakes and computers make a mistake if you leave a white space character in there, too. So it just goes to show everyone is prone to a mistake. The computer was doing what it was supposed to do. So we talked a little bit about, you know, what HDM is from the beginning, all the way up to some of the more advanced types of things that you can do with HDM. So let's go a little deeper or maybe a little higher with the level of maturity that you think HDM is at right now at the moment.

00:20:22Martin AlfkeSo we made HDM an open source product. We decided to put it under an Apache to license. And to be honest, it's still under heavy development. We decided to head for an open source model because we believe that this model brings value to anybody, whether an individual or a company. It makes no difference. Open source is a success model, it's the same like Puppet did with Puppet open source and then adding their enterprise product. The biggest problem for us is all the people within example42, we are not developers. So naming us a developer would be something like an insult to every developer in the world because we say no developer can code as bad as we. I know that especially, Stefan is already mentioned by saying this, you already gain some level of expertise others will never reach, but we are still not developers. So that's the reason why I was super excited to have Stefan and this team. He is a very experienced developer and they are now taking over as of today, by the way, we just signed the contract. So they are taking over the lead development. So we have the idea and they do the implementation.

00:21:37Demetrius MalbroughCongratulations.

00:21:38Martin AlfkeWe still want this to be open source because we want anybody to be able to contribute, whether it's just reporting an issue, whether it is by improving the documentation or writing an article, anything of this is still possible. And of course, we still need some additional support and fixing and proving HDM. So we believe it makes more sense, maybe even engage the Puppet community within this product so we get more open source support. So everybody who is interested in HDM just get in contact with us. Open issues for us. What functionality do you need inside? What is working for you? What is not working for you? So the next two months we will spend the time and the budget on getting HDM stable and ready for production. We want to solve all of the existing issues we have at the moment and we want to add some of the core required missing features. Like I mentioned, with the data analysis. We also have ideas of bringing HDM into enterprises. Enterprises have some different feature requests. For example, they don't want to have a I can see/ I can't see login. They want to have some kind of rule based access control. So which user is allowed to modify or even see data for which node or even down to another node? What kind of data is he able to see or not to see? Within the initial product, we just saw everything back onto file systems, maybe a good repository, but some companies have the requirement to say we need to have some for eyes review principles. So not the person that changes the data is allowed to bring these data into production. There must be somehow an approval process and we consider doing something like it, merge reviews on data changes and build them into Hiera, into HDM, into the Hiera Data Manager so that people don't have to deal with GitHub at all. This allows us to grant access, for example, or Hiera key names or node names or node groups for a single user or for groups of users. We get something like and for IAS approval of changes and we are looking forward to having a stable release with maybe even a container within the next two months. So basically, we plan to do another Christmas present for the Puppet community by example42 by saying on Christmas we have the release ready.

00:24:26Martin AlfkeYes, for sure. So personally, I'm available by my nickname, @tuxmea. There's also example42, too. We have a GitHub account, so everything is placed on GitHub. So just open your browser and go to https://Github.com/example42/HDM. This will bring you directly to our HDM repository. Open a ticket over there, ping us on Twitter, notify us on Slack. You will find me @tuxmea.

00:25:08Demetrius MalbroughWell, any, any final, I guess, closing words. You want to take us out of the Pulling Strings episode podcast please.

00:25:18Martin AlfkeEverybody stay safe. We are living in difficult times. Take care of people who need help, try to help local businesses, go local, buy local and whatever you do, do it with an open mind, build up communities and share. I believe sharing is caring and caring is something everybody needs nowadays.

00:25:37Demetrius MalbroughAll right. We can all use some more sharing and caring, especially during these trying times. So, Martin, I would like to thank you for sharing with us on Pulling the Strings podcast powered by Puppet.

00:25:48Martin AlfkeIt was a pleasure being with you. Thank you.

00:25:50Demetrius MalbroughHave a great week.