Published on 1 February 2016 by

In addition to regulations organizations must adopt, companies often have their own security policies and processes that all of IT must comply with. While the term "security" is a wide topic including intrusion detection, log analysis, and penetration testing, it also includes policy management that outlines secure software and configuration implementations such as authentication and encryption models, system hardening, and review processes.

Puppet is an exceptional tool for defining and enforcing configuration policies over time. Since a subset of configuration policies are related to security and regulatory compliance, Puppet is by definition exceptional at managing those security and compliance policies that overlap with configuration policy. This article will focus on the processes of defining, deploying, testing, and enforcing security and compliance policies as they pertain to configuration management.

where puppet fits into security and compliance

The anti-pattern: Teams working against each other

Information assurance and information security (InfoSec) teams often find themselves in conflict with systems operations teams. This isn't due to malice on either side; it's just that InfoSec sometimes deploys compliance tools without reference to the current configurations and why systems administrators configured things the way they did. That causes sysadmins to circumvent policy tests. Over time, both sides develop negative perceptions of each other, and working together is seen as a burden or worse, rather than as an opportunity to learn from each other while protecting the organization's assets.

Combining infrastructure as code, policy-driven development and DevSecOps practices can enable effective collaboration between InfoSec, development and SysOps. These teams can also get a much more accurate view into the compliance state of their organization's infrastructure, and become more efficient, too.

DevSecOps

DevOps promotes the goal of thinking about how an application and its services will operate in production through every phase of the product delivery process: planning, development, and deployment. Similarly, DevSecOps promotes thinking about the security of what’s being delivered at every stage of the development process: planning, development, deployment, and continuing operation of services.

DevSecOps begins by including InfoSec and SysOps in the planning of software development from the very beginning. InfoSec can help guide security considerations developers should be thinking about — for example, data access control, encryption, API design, and secrets management. Part of the outcome of the planning phase should be policies written in prose that capture the intent of the decision. Note that the policy should rarely include low-level implementation details, since the implementation of a feature or deployment mechanism hasn't been built yet.

The second step of DevSecOps should be to implement the decisions made during the planning phase, such as designing encryption models and deployment mechanisms. Policy and penetration tests should be implemented and run early as part of the continuous integration process, to ensure security and policy compliance are considered for every line of code as it's written.

Finally, as features and services are deployed, SysOps should continuously scan for and report on security violations. Vulnerability remediation retrospectives are a great way to continuously improve the process of inter-team security remediations. SysOps should collect vulnerability and retrospective assessment data to feed back into the next planning phase.

DevSecOps focuses on the cultural aspect of how teams work together. Like DevOps, culture is critical, but is just one component of working successfully. A well defined and practiced process combined with the right tooling is essential to making the cultural shifts successful. Policy driven development focuses more on the process and artifacts from each step of the process.

Policy-driven development

If this term is unfamiliar, you probably are more familiar with the concept test-driven development: creating development processes and workflows that take account of the need to test at frequent junctures and proceed based on the results of tests. Similarly, policy-driven development makes it possible for InfoSec, SysOps, and Devs to continuously validate policy compliance of new product features and system configurations by testing each change against a set of shared and agreed upon security policies.

Policy-driven development defines a set of steps and artifacts to be used throughout product delivery. The steps flow from policy definition, policy tests, policy implementation, and continuous policy monitoring. Below we’ll walk through each step and discuss the goals and artifacts to be produced.

policy process

The Process

The key to successful cross-team commitment to maintaining security policies is collaboration, process, and tooling. By following four high level steps, teams can leverage each other’s expertise, ensure each infrastructure configuration change and product feature complies with security policies early in the development process, and continuously monitor the infrastructure for compliance.

Step 1: Define policies in prose

The decisions made during the DevSecOps based development planning should be documented as policies Development and Operations both should consider when building features and system configurations.

Writing a good policy

Policies should be written in prose and not take potential low-level implementations into consideration unless unavoidable. These policies are for humans, not computers. They define considerations for those producing code, whether application code or infrastructure code.

For example, a policy might describe how to restrict services' access to a particular database. Another policy might specify that user accounts should never contain a blank password, and that authentication systems should never allow blank password authentications. Note that neither of these examples dictate implementations; there are numerous ways to enforce these policies. The best implementation should be determined by the team who owns that piece of the application or infrastructure, as long the implementation satisfies the defined policy.

Example of well-written policy

“Any system of authentication that makes use of passwords should not allow blank passwords to successfully authenticate.”

Example of poorly written policy

“PAM should be configured with the nullok module disabled.”

Step 2: Write automated tests to test the policy’s intent

Many teams skip over this step, but it’s incredibly important for validating true policy compliance, rather than implementation compliance. Policy tests should test the policy's intent, not its implementation. To expand on the blank password example from above, a proper test will not simply grep for the “nullok” string in the PAM configurations. That’s grossly insufficient. Instead, the policy should be covered by multiple tests. One test should look at every local user account and ensure none have a blank password. Another should create a user (local and directory if applicable) with a blank password, and attempt to authenticate as that user. The point is to test the actions the policy hopes to prevent, rather than testing the system's configuration.

Policy Engine

Puppet has written a proof-of-concept module named Policy Engine to enable writing policy tests as Facter facts. Using facts as policy tests has several advantages:

  • Tests are run prior to each Puppet run, and the results can be used as Facter variables in Puppet code to perform reactive measures if a test fails.
  • Test results are stored in PuppetDB, and thus can be queried against the current state of the infrastructure. See this PuppetDB query for an example ofgetting all failed tests in production.
  • Tests can be written in any language, and can even adopt your existing tests, while the report format for test results is standardized.

To get started with Policy Engine, create a class in Puppet that will contain your policy tests.

class rhel_6_stig::no_blank_passwords {

  policy_engine::test { 'no_blank_passwords':
    source             => 'puppet:///modules/rhel_6_stig/no_blank_passwords.rb',
    interpreter        => ‘/opt/puppetlabs/puppet/bin/ruby’,
    expected_output    => [],
    output_format      => ‘json’,
    tags               => [‘stig’,’V-38497’],
  }
}

We can see that the module where the class is defined — in this case rhel_6_stig — contains a file called no_blank_passwords.rb and that the module has set the test’s interpreter to /opt/puppetlabs/puppet/bin/ruby to run the file as a script. Note we use the ruby interpreter that ships with the Puppet agent since we’ll be using Puppet’s Resource Abstraction Layer (RAL) to introspect resources on the system. Let’s take a look at the script and what it does:

require 'puppet/face'

Puppet::Type.loadall

results = Array.new

# Get all users on the system from Puppet’s 
# Resource Abstraction Layer (RAL)
# and test them for blank passwords
Puppet::Face[:resource, :current].search('user').each do |user|
  if user[:password] == ''
    results << {'user' => user.title , 'blank_password' => true}
  end
end

#Output results in JSON
puts results.to_json

The script above asks Puppet to retrieve all local users on the system, regardless of the current operating system, and iterates through each one to see if their passwords are blank. If any of the passwords are blank, they are put into an array. Finally, the array is serialized into JSON and printed to standard out.

If we look back at the policy_engine::test resource above, we can see the expected output of the script to be an empty array. So if any user accounts with blank passwords were discovered and added to the array, the expectation for the test would fail. The output would look something like this:

{'result' => 'fail', 'tags' => ['policy_engine','stig','V-38497'], 'expected_output' => [], 'is' => [{'user' => 'foo', 'blank_password' => true}]}

Declare the class that contains your Policy Engine tests to the relevant systems. In this case, the rhel_6_stig::no_blank_passwords class should be declared on all Red Hat 6 systems. Once Puppet sets up the policy test, the test results will begin showing up in the inventory data including PuppetDB.

Step 3: Use Puppet to continuously enforce policy implementation

Once you write tests for a policy’s intent, you can consider the implementation of the policy. For example, if the no_blank_password test above fails, you can consider an implementation that makes the test pass. In this case, the affected user would either be removed or have their password managed with Puppet.

Here's another example. Let's say there was a test to validate that an account with a blank password could not be logged into with a blank password. If that test failed, we’d likely configure PAM to ensure the nullok module is disabled using the ghoneycutt/pam Forge module.

Note that it's unnecessary to write a test to validate that the nullok module is disabled. Puppet will ensure that the desired configuration you’ve stated will stay in place over time, and the policy test will validate that blank passwords cannot be used to authenticate user accounts. If you wrote a test to validate the nullok module was disabled, there could be other ways a blank password vulnerability would be exposed while the PAM test would pass. This is why it’s important not to test low-level implementations, and instead test a policy’s intent.

Step 4: Continuously report on configuration policy compliance

Event Inspection in Puppet Enterprise shows change as it occurs across the infrastructure. Event Inspection shows change in the different contexts where a change can occur: on nodes, as part of a class, or to individual resources that span multiple systems. Seeing the context enables you to quickly understand, for example, if the same change occurred to the SSH configuration across multiple hosts, rather than having to dig through change reports one at a time to understand if all the changes were actually the same change.

Event Inspection in Puppet Enterprise

Policy Engine reports

Policy Engine stores tests results in PuppetDB, enabling complex queries to be constructed through the PuppetDB query API.

Example PuppetDB queries

The query below can be run against the PuppetDB API. From the Puppet Enterprise master, you can use the curl utility to issue the query. Alternatively, custom utilities can be built to query the PuppetDB API. Below are a few examples of PuppetDB queries that can be run with the curl command. The example curl command below can be run from the Puppet Enterprise master. See the PuppetDB curl tips documentation.

# curl -X GET http://localhost:8080/pdb/query/v4/facts --data-urlencode [email protected]<query_file>.json

all_failed_tests.json

["and",
  ["in", "name",
    ["extract", "name",
      ["select_fact_contents",
       ["and",
         ["~>", "path", [".*", "tags", ".*"]],
         ["=", "value", "policy_engine"]
       ]
      ]
    ]
  ],
  ["in", "name",
    ["extract", "name",
      ["select_fact_contents",
        ["and",
           ["~>", "path", [".*", "result"]],
           ["=", "value", "fail"]
        ]
      ]
    ]
  ]
]

failed_in_production.json

["and",
  ["in", "name",
    ["extract", "name",
      ["select_fact_contents",
       ["and",
         ["~>", "path", [".*", "tags", ".*"]],
         ["=", "value", "policy_engine"],
         ["=", "environment", "production"]
       ]
      ]
    ]
  ],
  ["in", "name",
    ["extract", "name",
      ["select_fact_contents",
        ["and",
           ["~>", "path", [".*", "result"]],
           ["=", "value", "fail"]
        ]
      ]
    ]
  ]
]

Example output

If we have a user foo on a local system with a blank password, the example test above will produce a Facter fact called no_blank_password with the following output.

# facter -p no_blank_passwords
{
  result => "fail",
  is => [
    {
      user => "foo",
      blank_password => true
    }
  ],
  expected_output => [],
  tags => [
    "stig",
    "V-38497",
    "policy_engine"
  ]
}

The following is example output from the failed_in_production PuppetDB query example above.

# curl -X GET  http://localhost:8080/pdb/query/v4/facts --data-urlencode [email protected]_in_production.json

[ {
  "certname" : "centos7",
  "environment" : "production",
  "name" : "no_blank_passwords",
  "value" : {
    "expected_output" : [ ],
    "is" : [ {
      "blank_password" : true,
      "user" : "foo"
    } ],
    "result" : "fail",
    "tags" : [ "stig", "V-38497", "policy_engine" ]
  }
} ]

Conclusion

Through process, communication, shared tooling, and shared policies and responsibilities, teams can keep up with the constant pace of change and ensure that security and compliance policies that fall under configuration policies can be met at all times. By pushing policy testing to the left of the product delivery pipeline, teams can prove compliance of infrastructure and application code long before it touches production.

Learn more

Share via:
Posted in:
The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.