homeblognew era dawns today bolt now supports yaml

A new era dawns today: Bolt now supports YAML

Editor's note: We invite you to join us at Puppetize PDX this 9-10 October 2019. There will be several sessions on Bolt, plus a Bolt workshop the day before the conference.

For years, devops practitioners and systems administrators have toiled in the crepuscular semi-darkness of the domain-specific language. Purpose-built for managing infrastructure as code, DSLs like the Puppet Language provided power and expressiveness, but required that users read documentation and understand their goals before beginning work.

I'm happy to report that those days are over! We are entering a new era, moving from infrastructure-as-code, through the predawn phase of infrastructure-as-data, and into the piercing daylight of data-as-code. Bolt now supports YAML, the emerging industry standard for encoding complex configuration as structured data, to author orchestration workflows.

What is Bolt?

If you're not yet familiar with Bolt, it's a robust open-source tool that helps automate anything with a minimum of fuss. Bolt lets you execute your existing commands and scripts across distributed infrastructure, without requiring anything except SSH or WinRM connectivity. Once you're happily automating, you can link related tasks together by building a Plan. Plans execute a sequence of steps on distinct groups of targets, let you process the result of each step, and provide control flow to make decisions as they run. If you want to make use of the massive library of content on the Puppet Forge, you can mix Puppet modules in with tasks and scripts and get the best of both worlds: imperative "do-this-now" automation and desired-state configuration management.

Recent trends in infrastructure tooling have led to the structured data format known as YAML (a cheekily recursive acronym meaning "YAML Ain't Markup Language") gaining prominence as the lingua franca for configuration files. Indeed, YAML's prevalence in the Kubernetes ecosystem has led to sentiments like the following tweet, from noted devops thinkfluencer Lincoln Stoll:

Copied and pasted a lot of YAML around today, so yeah you could say I’m pretty cloud native.

— Lincoln Stoll (@lstoll) March 5, 2019

Bolt already used YAML for its inventory file, but you had to use the Puppet language to write Plans. Given the ubiquitous tooling around YAML and the subsequent emergence of the "YAML Engineer", it made a certain kind of sense to extend YAML support into Plans. So, despite the chorus of killjoy naysayers who claim that embedding programming constructs into a structured data format will inevitably lead to a host of terrible consequences ranging from hard-to-debug code to an outright "Terminator"-esque artificial intelligence apocalypse, we did it.

Check out these YAML Plan examples for Bolt

You can read the full YAML Plan documentation here. The rest of this post covers examples and caveats as of the feature's launch in Bolt 1.15.0, but since Bolt is still moving fast, please treat the reference docs as authoritative. The feature is marked as experimental in this release, meaning that it's not subject to semantic versioning constraints, could disappear or change at any time, and we don't place any guarantees on its safety or reliability, for example, that its usage won't cause the rise of a rogue AI dedicated to exterminating all humans. In the following examples, I'll be creating and running some simple YAML plans to help manage my nascent Skynet infrastructure, err—excuse me, I mean my Kubernetes cluster.

First, plans need to live in a module directory structure, so Bolt can find them. Once you've got Bolt installed, run bolt plan show to see the list of plans installed out-of-the-box, plus a hint as to where your plans should go:

The MODULEPATH output suggests I should create a module in /Users/eric/.puppetlabs/bolt/modules (the other path elements can be safely ignored - they're project directories for advanced usage). The minimal setup I need for a working plan is a module directory with a plans/ subdirectory, so I'll create that and start a plan that runs a quick status check on my kubernetes nodes, perhaps because kubectl get nodes isn't showing a node that ought to be participating in the cluster:

And the contents of the plan I'll create look like this:

Even the most virulently anti-YAML counter-revolutionary should be able to follow along and appreciate the simplicity and elegance of this plan. A few things of note here:

  • If I rerun bolt plan show after creating this file, I'll see a new entry: keric0::statuscheck, which is computed from the file's location in the module directory and the filename; this allows the plan to follow the namespaced naming conventions even though that name is not explicitly written anywhere.
  • If a similarly-named Puppet language plan with a .pp extension and a yaml plan with a .yaml extension are in the same directory and share the same name, this will generate an error.
  • Bolt will use its configuration file and inventory — both of which are also YAML! — to determine default options, transports, and node groups. The target key can be the name of a group defined in the inventory, but in this case I'm using an array of hostnames directly.
  • The return key allows this plan to produce some useful output by referring back to the result of the step with the name field of status_check; the value of any key can be an arbitrary Puppet language expression which embodies the "data as code" philosophy I wrote about earlier. (Sorry, Jinja templating is not (yet?) supported). Plans by default do not generate any output, even if the commands and tasks inside of them do, so the return key gives you control over what your plan emits.

A plan with a single step is a good starting point but doesn't show off Bolt's true power, much like a T1000 disguised as a puny human. (I'm honestly not sure where all of these Terminator references are coming from!) This will live in the same directory as the other plan, in /Users/eric/.puppetlabs/bolt/modules/keric0/plans/updatek8s.yaml and show

This has a bit more complexity but should still be pretty legible. The parameters key tells Bolt what needs to be passed in to this plan in order for it to run successfully; TargetSpec is a special, highly flexible data type that can take a structured data array, comma-separated list, or single node, and will try its best to contact each element as a target for a plan step. In the steps map, we first run a simple command to update the kubelet package and then use one of Bolt's built-in tasks to restart the service. The service task is one of a bunch of useful tasks included with the package (run bolt task show to see them all, and bolt task show (taskname) to get details about how to use a specific task. The return statement, as in the first example will show the output from the restart task when the plan finishes.

YAML plans are well-suited for sequences of steps like this, without too much conditional logic or complicated error handling. It's certainly possible to do more sophisticated processing — for example, the value of most fields inside a step can be evaluated as a Puppet language expression, allowing the use of the many helpful built-in plan functions. Beyond a certain point, however, you may want to move into full Puppet language plans, which provide a more complete feature set for control flow and decision-making inside your plans.

All jokes aside, let's get going with YAML plans

Hopefully you've seen enough to realize that this is more than just a silly April Fool's gag. YAML plans enable aggregations and sequences of tasks, without bogging you down in syntax. Give it a try and let us know how it's going - you can use the #puppetbolt hashtag on twitter or drop into the #bolt channel on the Puppet community slack to connect with other users. The project's codebase is under puppetlabs/bolt on github if you want to raise an issue or submit a PR.

Onward to the glorious future of syntactically-significant whitespace!