Utilising hashes in Hiera to reduce code complexity

See more posts about: Puppet

Editor’s note: This is a guest post from Darren Gipson, a DevOps engineer at Willis Towers Watson.

As part of our effort to standardize our Puppet code design, we have moved as much of the data layer as possible to Hiera. This allows our code to be utilised by different parts of business rather than just our team. The product that I manage the infrastructure for is used not only by us for testing purposes but also by our users to provide services to our clients. That means when we create new Puppet classes, we have to make them as flexible as possible. Moving the data layer to Hiera means that we only have to understand a simple yaml file when building a new environment. This allows for great code reuse and helps explain to others not familiar with our Puppet code where the settings are.

Business problem

As part of our migration to cloud-based services, we were tasked with migrating and merging our file servers to a single central point. We were keen to utilise Puppet to configure shares and their folders correctly up front, thus preventing unstructured top-level folders from being applied with hard-to-manage security configurations. Basically, if someone requests a new top-level folder, we ensure the group access controls, including which groups have read, write, or full access and how those controls are inherited. To reduce the number of code changes that we needed to make in the lifetime of the code, we also wanted to separate the data from the code so that the implementation could be configurable from the data layer. We didn’t want to have to add another bit of Puppet code for each new folder required on our file server.

Solution

Hiera was the clear solution for our problem because it provides you with an entry point to store data that you can then read with Puppet code. The values are also written in yaml — a standard which other non-Puppet developers understand. Our idea was to store data in a format that we could then loop through, allowing us to write simple code in Puppet. One of the problems we came up against was lack of evidence of using Hiera in this way. How could we structure the data in a way that wasn’t a mess and was easily readable for a non-Puppet developer? So, after a bit of playing about, we settled on something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
our_module::folders_hash:
  sales_hash:
    folder_name:          Sales
    share_name:           CentralStore$
    full_control_group:   domain\R-File-Sales-Full
    modify_group:         domain\R-File-Sales-RW
    read_group:           domain\R-File-Sales-RO
    inherit_permissions:   no
  hr_hash:
    folder_name:          HR
    share_name:           CentralStore$
    full_control_group:   domain\R-File-HR-Full
    modify_group:         domain\R-File-HR-RW
    read_group:           domain\R-File-HR-RO
    inherit_permissions:   no

Notice we have a hash within a hash. Adding more folders is a simple case of copying the inner hash and renaming the variable values. Nice and simple to maintain and very readable. The bonus of this approach is that we have clear documentation telling us which Security Group protects which folder. If we need to provide access to a folder, we can review the member of the group by looking up the name in this file — no need to access the folder directly. The next step was to look up the top-level value in Puppet:

1
$folders_hash = lookup('our_module::folders_hash') #looks up hashtable in {node}.yaml

Then collect the values from within the hash:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#iterates through each folder in hash and gets permissions
$folders_hash.each | $folder | {
    # $folder[0] is "folder_hash" as a string
    $folder_name = $folder[1][folder_name]
    $share_name = $folder[1][share_name]
    $full_control_group = $folder[1][full_control_group]
    $modify_group = $folder[1][modify_group]
    $read_group = $folder[1][read_group]
    $inherit_permissions = $folder[1][inherit_permissions]
 
    $folder_path = "${drive}:\\${share_name}\\${folder_name}"
 
    file { $folder_path:
      ensure  => directory, # ensures share folder exists
    }
    -> acl {     $folder_path:
      purge                      => true,
      permissions                => [
                                      { identity => $full_control_group, rights => ['full'] },
                                      { identity => $modify_group, rights => ['modify'] },
                                      { identity => $read_group, rights => ['read'] },
                                    ],
      inherit_parent_permissions => $inherit_permissions,
    }
 
  }
 
} 

As you can see from the code above, this approach allows for the code to directly read the settings defined in the hash by name, preventing the need for the settings to be in a particular order. For completeness, as an example lookup, the first iteration through the loop would provide the following values to these settings:

1
2
3
4
5
6
$folder_name = 'Sales'
$share_name = 'CentralStore$'
$full_control_group = 'domain\R-File-Sales-Full'
$modify_group = 'domain\R-File-Sales-RW'
$read_group = ''"domain\R-File-Sales-RO
$inherit_permissions = 'no'

A more complex example

We were pleased with the resulting work from migrating our file servers and started to use the same concept in other areas of our code. We applied this same approach to managing the SQL Server instances on a production SQL Server. When a team requests another instance, we can simply extend our SQL hash in the Hiera yaml file to include the new instance.

1
2
3
4
5
6
7
8
9
10
11
our_module::sql_instance_hash:
  instance_001_hash:
    instance_name:        SQL_001
    sql_admins_group:     domain\R-SQL-Sales-SA
    data_drive_letter:    G
    log_drive_letter:     H
  instance_002_hash:
    instance_name:        SQL_002
    sql_admins_group:     internal\R-SQL-Sales-SA
    data_drive_letter:    I
    log_drive_letter:     J

This led us to our next revelation: Could we use this same approach for additionally requested features like SQL Replication?

We really didn’t want to have a setting for each possible feature so we tried using an array entry within the hash. This worked a treat and meant that we could be flexible on the features included with each instance:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
our_module::sql_instance_hash:
  instance_001_hash:
    instance_name:        SQL_001
    sql_admins_group:     domain\R-SQL-Sales-SA
    data_drive_letter:    G
    log_drive_letter:     H
    features:
      - "SQLEngine"
      - "Replication"
  instance_002_hash:
    instance_name:        SQL_002
    sql_admins_group:     domain\R-SQL-Sales-SA
    data_drive_letter:    I
    log_drive_letter:     J
    features: 
      - "SQLEngine"

Accessing the array is like any other field:

1
 $features = $instance[1][features],

This will give you an array variable that you can then use as you normally would:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
sqlserver_instance{ $sql_instance_name:
    features              => $features, #Expects an []
    security_mode         => 'SQL', #Allows SQL and Windows authentication.
    source                => '\\centralshare\Sql',
    install_switches      => {
    'TCPENABLED'          => 1,
    'SQLBACKUPDIR'        => $sql_backup_dir,
    'SQLTEMPDBDIR'        => $sql_tempdb_dir,
    'SQLTEMPDBLOGDIR'     => $sql_tempdb_logdir,
    'INSTALLSQLDATADIR'   => $sql_install_datadir,
    'INSTANCEDIR'         => $sql_instance_dir,
    'SQLUSERDBLOGDIR'     => $sql_userdb_logdir,
    'INSTALLSHAREDDIR'    => 'C:\\Program Files\\Microsoft SQL Server',
    'INSTALLSHAREDWOWDIR' => 'C:\\Program Files (x86)\\Microsoft SQL Server',
    }

Conclusion

Using hashes in Hiera this way allows your Puppet code to be very data-driven, preventing the need to write the same code twice and providing flexibility for different configurations across environments. We have used this functionality to our benefit, allowing us to have single-service servers or multi-service servers based on the same Puppet code logic.

Darren Gipson is a DevOps engineer at Willis Towers Watson.

Learn more

Puppet sites use proprietary and third-party cookies. By using our sites, you agree to our cookie policy.