Use Boltwash to interact with your Bolt targets and their filesystems

See more posts about: Tips & How To and Tools & Integrations

If you're not using Bolt, check it out.

If you are using Bolt, we’ve created the Boltwash plugin that lets you navigate your inventory. Let's check it out:

1
2
3
4
5
wash . > stree bolt
bolt
└── [group]
    └── [target]
        └── fs

Wash is a UNIX-like shell that lets you manage your infrastructure as files and directories. It ships with support for AWS, Docker, Kubernetes, and GCP resources, but can easily be extended to talk to other things via its external plugin interface. Boltwash is one such external plugin.

Exploring Boltwash

As you can see, the Bolt plugin organizes Bolt targets as groups. It also mounts something called fs (a view of the filesystem on that target). Let's explore it a bit more.

1
2
3
4
5
6
wash . > cd bolt
wash bolt > ls
all/
my-docker-app/
my-linux-nodes/
my-windows-nodes/

I have an inventory setup organizing some machines according to where they run. Your groups can be organized however you like. The all group will appear for everyone; it contains all targets in your inventory (it's an implicit group that can be used in Bolt). Let's explore a particular target.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
wash bolt > tree -L 2 my-linux-nodes
my-linux-nodes
├── node1
│   └── fs
└── node2
    └── fs
wash bolt > cd my-linux-nodes/node1/fs
wash bolt/my-linux-nodes/node1/fs > ls
bin/
boot/
dev/
etc/
home/
...

Great — I can see the files on that system, and I didn't have to do any additional setup. I can go look at config or watch logs.

1
2
3
4
5
6
7
wash bolt/my-linux-nodes/node1/fs > cat etc/hosts
127.0.0.1 localhost
...
wash bolt/my-linux-nodes/node1/fs > tail -f var/log/syslog
Feb  7 18:39:51 pe-agent01 systemd[18116]: Startup finished in 23ms.
Feb  7 18:39:51 pe-agent01 systemd[1]: Started User Manager for UID 1000.
...

We can also explore Windows targets using WinRM:

1
2
3
wash bolt/my-windows-nodes/node5/fs > ls Users
Administrator/
Public/

Boltwash includes target configuration as metadata, so you can also view that.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
wash bolt/my-linux-nodes > meta node1
alias: []
config:
  ssh:
    connect-timeout: 10
    disconnect-timeout: 5
    load-config: true
    run-as: root
    tty: false
    user: ubuntu
  transport: ssh
facts: {}
features: []
name: node1
plugin_hooks: {}
uri: node1
vars:
  datacenter: west coast

And you can use find to search for nodes based on that metadata.

1
2
3
4
5
wash bolt > find . -meta .vars.datacenter 'west coast'
all/node1
all/node5
my-linux-nodes/node1
my-windows-nodes/node5

Debugging with Bolt + Wash

Wash can be helpful debugging all sorts of failures. As an example, suppose you have a cluster of machines (in the webservers group) that are having problems:

1
2
3
4
wash bolt/webservers > ls
server1/
server2/
server3/

You've noticed an issue in one of their logs, and you want to see if it's present everywhere:

1
2
3
4
wash bolt/webservers > grep OutOfMemory */fs/var/log/syslog
server1/fs/var/log/syslog:Jan 28 12:09:34 server1 my-server[435]: ...java.lang.OutOfMemoryError...
server2/fs/var/log/syslog:Jan 28 12:09:43 server2 my-server[435]: ...java.lang.OutOfMemoryError...
server3/fs/var/log/syslog:Jan 28 12:09:01 server3 my-server[435]: ...java.lang.OutOfMemoryError...

You can see that they're all experiencing these issues. Let's see where the server process is still running:

1
2
3
wash bolt/webservers > wps * | grep bin/java
server1     9027   12:10.02   /usr/bin/java -Xmx2048m -cp /opt/my-server/server.jar ...
server2    21204   12:13.02   /usr/bin/java -Xmx2048m -cp /opt/my-server/server.jar ...

It looks like they've restarted recently, and the server's not running on server3. Let's reconfigure the servers to allow more memory (using Bolt or by directly modifying config/service files) and restart them.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
wash bolt/webservers > bolt task run service -t webservers action=restart name=my-server
Started on server1...
Started on server2...
Started on server3...
Finished on server1:
  {
    "status": "MainPID=751,LoadState=loaded,ActiveState=active"
  }
Finished on server2:
  {
    "status": "MainPID=2927,LoadState=loaded,ActiveState=active"
  }
Finished on server3:
  {
    "status": "MainPID=21936,LoadState=loaded,ActiveState=active"
  }
Successful on 3 targets: server1,server2,server3
Ran on 3 targets in 1.91 sec

You can imagine other scenarios as well. Suppose you ran a Bolt plan across several targets and want to get some files back from them. You can use Wash to directly copy them with something like:

1
wash bolt > for f in group/*/fs/path/to/file; do cp -v "$f" ~/dest/"${f//\//_}"; done

The Bash string manipulation (${f//\//_}) replaces all slashes in the original path to the file with underscores to make it unique.

Configuring Boltwash

Boltwash loads the default Boltdir at ~/.puppetlabs/bolt, but you can configure it to load any Boltdir in Wash's config. To use Boltwash, you add it to the external-plugins key in ~/.puppetlabs/wash/wash.yaml; adding a custom Boltdir location looks like:

1
2
3
4
external-plugins:
- script: '/path/to/boltwash/bolt.rb'
bolt:
  dir: /path/to/boltdir

The Bolt plugin for Wash provides an accessible, interactive means of investigating multiple systems. Combined with Bolt inventory, it's natural to then take action with Bolt. Try it out!

Puppet sites use proprietary and third-party cookies. By using our sites, you agree to our cookie policy.