GNS Science

Industry

Technology

Background

GNS Science is New Zealand’s leading provider of Earth, geoscience and isotope research
and consultancy services. GeoNet is a GNS Science project funded by the New Zealand Earthquake commission, to build and operate a modern geological hazard monitoring system in New Zealand. It comprises of a network of geophysical instruments, automated software applications and skilled staff to detect, analyze and respond to earthquakes, volcanic activity, large landslides, tsunami and the slow deformation that precedes large earthquakes.

Challenge

  • Build an exceptionally resilient infrastructure that can withstand natural disasters.

Solution

  • Puppet Enterprise

Results

  • Puppet Support got employees quickly trained; confidence in the resiliency of its infrastructure.

Automation is the best documentation…Gone are the complex and out-of-date documents explaining how to create and install systems. They have been replaced by code which is used to automatically and repeatably create our critical IT systems.

Geoff Clitheroe, GeoNet systems development team lead, GNS Science

Keeping an Island Nation Apprised of Geological Events

New Zealand is a geologically hazardous country with a relatively sparse population, making it a challenge to deliver timely and accurate information about geological events that could have an impact on people, animals and property. GeoNet was designed and built to meet that challenge.

GeoNet relies on a network of geophysical instruments, automated software applications and skilled staff to detect, analyze and respond to earthquakes, volcanic activity, large landslides, tsunami and the slow deformation that precedes large earthquakes.

Where most traditional IT shops might consider a large damaging earthquake a risk of doing business in the region, GeoNet is expected to continue operating during such circumstances, so its IT has to be unusually resilient for an organization of its size. GeoNet has had multiple domestic data centres since its inception, with the majority of products and services replicated across several locations.

Goal: Achieve Consistency, Reliability and Visibility in a Heterogeneous, Distributed Infrastructure

The GeoNet IT infrastructure consisted of around 100 or so completely heterogeneous nodes. The majority of these nodes were a mix of physical and virtual machines running Red Hat

Enterprise Linux (RHEL), CentOS or Scientific Linux in 5.x and 6.x, on both 32-bit and 64-bit architectures. There was also a smattering of Windows hosts for interfacing with specialized hardware. Most nodes were run in a geographically redundant pair, no with little or no staging.

While many parts of the GeoNet infrastructure were well maintained, with versioned configurations, the team wanted to centralize configurations. It was difficult to keep such heterogeneous systems maintained and up to date; much configuration was being done manually, resulting in some divergence in configurations between hosts.

The team also needed to be able to replicate its configurations in a sandbox for testing, to have confidence that code coming through testing environments would run equally successfully in production.

Because GeoNet has to be kept highly redundant in case of a natural event interrupting service at one of the system’s data centers, the team needed to plan for fast recovery, portability and scaling, moving data centers and rebuilding machines as quickly as possible.

Puppet Enterprise makes it possible to fulfill all of these requirements, and its console also provides visibility into the current state of infrastructure at all times. It’s also much easier to report on the state of infrastructure and demonstrate compliance with operational requirements.

The GeoNet team got up and running with Puppet very quickly, due in part to having training available in Australia. The team was also able to scale from initial installation quickly, and orchestrate complex changes that would previously have required a great deal of manual intervention.

GeoNet now has a well defined base server profile for all nodes, services, repositories and packages, minimizing divergence and making it possible to quickly build out infrastructure as needed. GeoNet began developing a mobile application for notifying people about events such as earthquakes in 2010.

The team moved its mobile alert system to Amazon Web Services in 2012, saving on operating costs while gaining the ability to scale notification services quickly during an event. GeoNet uses Puppet Enterprise to manage its hybrid cloud infrastructure hosted by AWS and other cloud providers. “The big win here was just being able to treat AWS EC2 instances as regular old VMs, and let Puppet Enterprise do its magic,” said Richard Guest, GeoNet platform manager at GNS Science.

Starting with Open Source Puppet

GeoNet staff initially evaluated different configuration management tools (including both Open Source Puppet and Chef) on a limited set of production machines. After deciding Puppet was a better choice for GeoNet’s requirements, a few team members went through Puppet training, and decided that Puppet Enterprise would fill more of the organization’s needs.

DevOps at GeoNet

GeoNet wanted to streamline the process of getting code from development to production — a common reason for organizations to look into DevOps. Adopting Puppet Enterprise was an important step in getting DevOps practices established, as described in a blog post by Geoff Clitheroe, GeoNet’s systems development manager:

“Puppet describes server infrastructure as code. Once a server, and the installed software, are described in code, then making a new server that is exactly the same becomes as simple as running a computer program — something that the Puppet master is very good at.”

Of course, DevOps isn’t just about improving operations. “For the development side there has been a lot of work to improve the way we package, configure, and monitor code for production. ‘It worked on my laptop’ is not acceptable in our office anymore,” Geoff writes.

GeoNet now has streamlined, repeatable techniques for developing and operating its infrastructure,
and can confidently move code quickly from development to production — often in just minutes, knowing that code was tested on systems that precisely replicate production. This capability is especially important in case of disaster recovery.

Geoff notes the change of pace when he describes building servers by hand in March 2012 at the launch of GeoNet Rapid, a website that gives details of an earthquake’s strength and location within two minutes of its occurrance.

“It took a couple of days, tediously logging in to each server and installing and configuring the required packages,” he writes. “Even with the best intentions it’s an error-prone approach that leaves us with fragile computer systems that are very hard to maintain. In the event of a disaster, recovery would be very difficult.”

By 2014, with Puppet Enterprise and DevOps processes in place, “in a few minutes’ work, we swapped in four new servers running SeisComP3, three new web servers, and completely replaced the messaging system between them,” Geoff writes. “This is so repeatable that later in the week we did it again on additional cloud servers to give us a complete duplicate system for testing and development.”

The GeoNet team has also discovered what Geoff calls “an unexpected benefit” — the fact that Puppet is actually executable documentation. As Geoff notes, “automation is the best documentation.”

“Gone are the complex and out-of-date documents explaining how to create and install systems,” Geoff writes. “They have been replaced by code which is used to automatically and repeatably create our critical IT systems.”

Top Outcomes of Using Puppet Enterprise

  • Fast ROI – The integration of the various components in Puppet Enterprise greatly sped up installation, allowing GeoNet to begin making real advances quickly.
  • Scalability and Orchestration – The scalability and orchestration of MCollective was a quick and powerful way to achieve a task which previously would have involved much manual intervention.
  • Defined Infrastructure – GeoNet now has a well defined minimal base server profile for all nodes, services, repositories and packages. This lowers the divergence in the infrastructure and shows the “power of Puppet.”

Starting Environment

  • The GeoNet IT infrastructure consisted of around 100 or so completely heterogeneous nodes.
  • The majority of these nodes were Red Hat based, but were a mix of physical and virtual running RHEL,
  • CentOS or Scientific Linux in 5.x and 6.x on both 32bit and 64bit architectures, with a smattering of
  • Windows hosts for interfacing with specialized hardware.
  • Most nodes were run in a geographically redundant pair, no with little or no staging.

Key Objectives

  • Centralized Versioned Configuration – While many parts of GeoNet’s software stack configuration was well versioned and maintained, they wanted everything in one place.
  • Consistency – Its highly heterogeneous systems were difficult to keep maintained and up-to-date. The process of doing each configuration step both repeatedly and manually had resulted in a slight divergence in configurations between hosts.
  • Repeatability – This would allow GeoNet to duplicate its infrastructure in a sandbox environment, safe in the knowledge that its testing machines are as close to production as possible.
  • Speed of recovery, scalability and portability – They wanted to be able to move data centers and rebuild nodes as quickly as possible.

Why Puppet Enterprise?

  • Support and Training – Having training and support available in Australia would allow GeoNet to get their people the skills they needed to get off the ground quickly.
  • Reporting and Compliance – The Puppet Enterprise console would provide GNS Science the confidence that its infrastructure is up and running, and configured correctly.
  • Event Inspector – The features of Event Inspector would make visualizing incoming changes and failures a much easier task

 

See for yourself what Puppet Enterprise can do for you.

TRY PUPPET ENTERPRISE