Cloud provisioning with Terraform and Bolt
*Editor’s notes: This post was originally published on lucywyman.me. We are republishing it with Lucy’s permission.*
*SEE ALSO: Since writing this, we've added inventory plugins to Bolt, which allow you to dynamically load inventory from sources like Terraform, PuppetDB, and Azure. I recommend checking out Tony Green's blog post about using the Terraform plugin.*
Terraform is a cloud provisioning tool that's great at managing low-level infrastructure components such as compute instances, storage, and networking. While Terraform is great at creating the infrastructure you need, it's not great at managing the state of your resources over time or enforcing certain states. Nathan Handler described it in a talk at OSCON 2018 as a way to get boxes that you can then go fill with the users, files, applications, and tools you need.
Image from 'Terraform All The Things' by Nathan Handler
Bolt is an open source remote task runner that can run commands, scripts, and puppet code across your infrastructure with a few keystrokes. It's available with RBAC and more enterprise features in Puppet Enterprise. Bolt combines the declarative Puppet language model with familiar and convenient imperative code, making it easy to learn and effective for both one-off tasks and long-term configuration management.
I want to demonstrate how powerful using these tools together is, and how they each enable you quickly get the cloud resources you need and provision them with minimal setup and code. We'll first create an AWS EC2 instance with Terraform, then use Bolt to get the IP of the instance and manage it using Puppet code (with zero Puppet knowledge required1). Let's get started!
Note: If you want to follow along or see a more complete example all my code is available on github.
Create Cloud Resources with Terraform
This step was simple: I followed the Terraform Getting Started Guide to set up a t1.micro EC2 instance, then added a few bits and bobs mostly around ensuring we can SSH into the machine. Here's some key notes and the code:
- SSH Key: We need to make sure there's a way to SSH into the boxes we create. I chose to do this with SSH key pairs, but you could also just have username + password set.
- Outputs: To make it easier to get the IP addresses for the instances we create I added an output to produce an array of the IPs of the instances. Parsing the default terraform json output in Bolt is equivalent.
- Ubuntu Xenial AMI: I'm totally new to AWS and wasn't sure how to create a user on my new instance, or more importantly whether Puppet would work on it. So I just used an ubuntu image instead of the usual Amazon Linux one.
- Security Group: This adds a security group to allow traffic into and out of the node so that we can, y'know, make use of it.
Configure Instances with Bolt Plans
Now that we've created a box with Terraform, we'll:
- Collect the public IP in a Bolt Plan
- Add it to the Bolt inventory file so it picks up configuration
- And do anything we want with it - for example, deploy a small webpage
First let's create a Bolt inventory file with configuration that Bolt will need to connect to the EC2 instance. This inventory includes 1 group called 'terraform', which defaults to using the SSH transport. It then configures the ssh private key, user, and host key check for this group.
Next we'll write the Bolt plan to run terraform apply, collect the IPs of the instances it creates, and provision those instances.
In less than 30 lines of code we've got an apache server up and running!
A few other files we'll need to support running Bolt:
A bolt configuration file, to tell it where to find modules
A Puppetfile with dependencies:
And lastly, an HTML page to serve:
Again, all these files are available in this git repo, with a bit more verbosity and structure!
Phew! Now that all our files are in place, here's how easy it is to deploy our server:
And that's it! The plan should output something like:
Visit the IP in your browser and check out your new site!
Terraform and Bolt are both great tools with different strengths. Together they make automating your infrastructure so much easier, and enable you to easily get resources, then configure and manage them over time, without too much overhead or learning. This example, while simple, is just the beginning - so what are you going to build?
1 If you don't believe me, 6 months ago I took the Puppet Certification test and got 48%, and I can do this. Really, I mean no Puppet knowledge!