Published on 4 April 2016 by

In this blog post, I'll show in detail how to run Docker Swarm in a production-ready state, all orchestrated with Puppet. Docker Swarm has a few moving parts, so the first thing we should look at is this: Docker Swarm needs a key/value store in the backend to know where all its member nodes are, who the Swarm master is, etc. For the examples in this blog post, we are going to use Consul, but you could very easily swap Consul with etcd or Zookeeper, depending on what you already have in your environment.

The great thing is, there is already an awesome module for Consul on the Puppet Forge, written by Kyle Anderson. We will be using this module to install and configure Consul. The bootstrap node gets set up like this:

class { 'consul':
    config_hash => {
      'datacenter'       => 'dev',
      'data_dir'         => '/opt/consul',
      'ui_dir'           => '/opt/consul/ui',
      'bind_addr'        => $::ipaddress_enp0s8,
      'client_addr'      => '0.0.0.0',
      'node_name'        => "$::hostname",
      'advertise_addr'   => '172.17.8.101',
      'bootstrap_expect' => '1',
      'server'           => true
       }
     }
  }  

And members of the cluster get set up like this:

class { 'consul':
  config_hash => {
      'bootstrap'      => false,
      'datacenter'     => 'dev',
      'data_dir'       => '/opt/consul',
      'ui_dir'         => '/opt/consul/ui',
      'bind_addr'      => $::ipaddress_enp0s8,
      'client_addr'    => '0.0.0.0',
      'node_name'      => "$::hostname",
      'advertise_addr' => $::ipaddress_enp0s8,
      'start_join'     => ['172.17.8.101','172.17.8.103','172.17.8.103'],
      'server'         => false
      }
    }
  }

Now that we have a Swarm backend, we can install Docker Swarm. You can get the Docker Swarm module from the Puppet Forge here: https://forge.puppetlabs.com/scottyc/docker_swarm.

Again, there are two blocks of code we will need: one to set up the master or manager, and another to join members to the cluster. For the initial cluster, we will create a Swarm cluster called cluster 1:

swarm_cluster {'cluster 1':
   ensure       => present,
   backend      => 'consul',
   cluster_type => 'join',
   port         => '8500',
   address      => '172.17.8.101',
   path         => 'swarm'
   }

We will then deploy the Swarm managers. It is a best practice to have more than one manager, as the manager provides the vital function of controlling the state of the cluster. If you have more than one manager, the module will set up replication for you. We set up the managers with this code:

swarm_cluster {'cluster 1':
    ensure       => present,
    backend      => 'consul',
    cluster_type => 'manage',
    port         => '8500',
    address      => '172.17.8.101',
    advertise    => $::ipaddress_enp0s8,
    path         => 'swarm',
    } 

The next thing to configure is a private Docker network, in case we need to link our containers:

  docker_network { 'swarm-private':
    ensure => present,
    create => true, 
    driver => 'overlay',
    }

Now we have our Swarm cluster up and running, we will want to deploy applications to the nodes in the cluster. In this example, we will deploy an ELK stack. We will need to set up service discovery to allow external services from the Swarm cluster to find our containers. To do this, we will install the bind package and point our DNS resolver to Consul:

 package { 'bind':
    ensure => present
  } ->

  file { '/etc/named.conf':
    ensure  => present,
    content => template("config/named.conf.erb"),
    mode    => '0644',
    owner   => 'root',
    group   => 'root',
    require => Package['bind'],
  } ~>

  file { '/etc/named/consul.conf':
    ensure  => present,
    content => template("config/consul.conf.erb"),
    mode    => '0644',
    owner   => 'root',
    group   => 'root',
    require => Package['bind'],
  } ~>

  service { 'named':
    ensure  => running,
    enable  => true,
    require => File['/etc/named.conf'],
  }

Here's the code for our two template files:

Consul.conf.erb

zone "consul" IN {
  type forward;
  forward only;
  forwarders { 127.0.0.1 port 8600; };
};

Named.conf.erb

options {
  listen-on port 53 { 127.0.0.1; };
  listen-on-v6 port 53 { ::1; };
  directory       "/var/named";
  dump-file       "/var/named/data/cache_dump.db";
  statistics-file "/var/named/data/named_stats.txt";
  memstatistics-file "/var/named/data/named_mem_stats.txt";
  allow-query     { localhost; };
  recursion yes;

  dnssec-enable no;
  dnssec-validation no;

  /* Path to ISC DLV key */
  bindkeys-file "/etc/named.iscdlv.key";

  managed-keys-directory "/var/named/dynamic";
};

include "/etc/named/consul.conf";

The last piece of the puzzle for service discovery is an application that will register our containers as they spawn. For this we will use registrator, and we will install it using Docker Compose.

file { '/root/docker-compose.yml':
  ensure  => file,
  content => template("config/registrator.yml.erb"), 
  } ->

docker_compose {'swarm app':
  ensure => present, 
  source => '/root',  
  scale => ['1']
    }
  }

Registrator.yml.erb

registrator:
  image: gliderlabs/registrator
  net: "host"
  volumes: 
   - "/var/run/docker.sock:/tmp/docker.sock"
  command: consul://<%= @consul_ip %>:8500 

Then we can deploy our ELK stack:

swarm_run {'logstash':
    ensure     => present,
    image      => 'scottyc/logstash',
    network    => 'swarm-private',
    ports      => ['9998:9998', '9999:9999/udp', '5000:5000', '5000:5000/udp'],
    env        => ['ES_HOST=elasticsearch', 'ES_PORT=9200'],
    command    => 'logstash -f /opt/logstash/conf.d/logstash.conf --debug',
    }

   swarm_run {'elasticsearch':
     ensure     => present,
     image      => 'elasticsearch:2.1.0',
     network    => 'swarm-private',
     volumes    => ['/etc/esdata:/usr/share/elasticsearch/data'],
     command    => 'elasticsearch -Des.network.host=0.0.0.0',
     log_driver => 'syslog',
     log_opt    => 'syslog-address=tcp://logstash-5000.service.consul:5000',
     depends    => 'logstash', 
     }

   swarm_run {'kibana':
     ensure     => present,
     image      => 'kibana:4.3.0',
     network    => 'swarm-private',
     ports      => ['80:5601'],
     env        => ['ELASTICSEARCH_URL=http://elasticsearch:9200', 'reschedule:on-node-failure'],
     log_driver => 'syslog',
     log_opt    => 'syslog-address=tcp://logstash-5000.service.consul:5000',
     depends    => 'logstash',
     }

From the code you can see that we are using our service discovery address to connect to Logstash: tcp://logstash-5000.service.consul:5000.

One of the cool features released in Swarm v1.1 and above is rescheduling containers on node failure. You can see that in our Kibana container, we are declaring reschedule:on-node-failure. If the node fails, Kibana will spawn on a healthy node. The Swarm module supports this, as the provider checks the cluster for the container, not the individual node. Thus, the module gives you the resiliency Swarm offers, while keeping the advantage of Puppet's idempotency. For people who would like to play with the code in this blog post, I have created a fully functional demo lab that can be found on my GitHub page.

Scott Coulton is a solution architect at HealthDirect Australia.

Share via:
The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.