Published on 30 November 2012 by

One of the things I’ve always found amusing about Operations is that people think we spend all of our time fixing broken things. It’s true to some extent: something breaks, we fix it. But the most challenging issues in Operations aren’t that simple binary break/fix. The really interesting issues are performance issues, intermittent bugs, and transitory problems: “it’s a little slow” or “something is off” or the “it’s not quite right” issues. Not only are those problems really challenging, but they can also really interesting (and often fun) to solve.

Diagnosing and solving those problems, however, is very different from dealing with break/fix issues. You still need good diagnosis skills to deal with break/fix issues, but you are usually able to rely on the change-test-validate cycle: make a change, test it fixes the issue, and then validate you haven’t broken anything else. With less binary problems, you sometimes don’t even know where to start or are working from unsubstantive qualitative feedback: “Something doesn’t feel right.”

Operations Loves Data

In those cases, Operations people love one thing: data. Internally at Puppet Labs, we’ve always collected a lot of data. We’re big users of Graphite, as well as producers and consumers of metrics, both in Operations and across the organisation. So when we deployed the new Puppet Forge (our online marketplace of 600+ freely downloadable configurable modules), the aptly code-named “Vulcan”, one of the things Engineering and Operations agreed on was “we should instrument … a lot.” If we have lots of data up-front, then we should be able to not only diagnose issues, but also proactively identify issues before they become customer impacting.

To give us a head-start on this instrumentation, we decided to use New Relic to monitor the new Puppet Forge. New Relic is a SaaS-based application performance management tool. It allows you to hook in New Relic’s probes to your applications and servers and then send the resulting data, right down to transaction level events, back to New Relic for analysis. You can then use their dashboard to get snapshot, trending, and event-based views of your applications and servers and how they are performing.

We chose New Relic for three major reasons:

  • They’ve already made the investment in building an awesome application performance monitoring suite and we didn’t want to reinvent the wheel.
  • New Relic integrates deeply with the technology (Sinatra and PostgreSQL) we’re using, and provides huge amounts of data that allows us to determine not only whether the application IS working, but HOW it’s working.
  • It "just works," and we’re big fans of tools that just work.

Puppet Module for New Relic

To make our use of New Relic and Puppet a bit more elegant, we developed some additional integration between Puppet and New Relic, and we’re releasing this integration as a module— a bundle of reusable Puppet code. The integration provides support for notifying New Relic about application deployments performed by Puppet. New Relic comes with a Deployment API that allows you to notify your application when deployment events occur, for example when a new version of the application is deployed. This allows you to track and identify the impact of changes made through new deployments, for example see if a new deploy of the application impacts the performance of the application.

The puppetlabs/newrelic module provides two ways to send these notifications: 1) a report processor that sends deployment events post a Puppet run, and 2) a new type called newrelic_notify, that notifies from your manifests, for example:

        newrelic_notify { '123456':
          ensure      => present,
          api_key     => 'abc123',
          description => 'Notify New Relic that Application Bob was deployed.',
          revision    => 'v1.0.1', 
          user        => 'Puppet',

Altogether this provides some powerful and tight integration between New Relic and Puppet. When application deployments and changes occur, you can now track the impact of those changes in New Relic by taking advantage of their deployment analysis dashboard.

We’re hoping that this new module will help Operations and Development folks using Puppet and New Relic together to make better use of them both. We’d love to have feedback and input into the module too. If there’s new functionality or features you’d like to see or you can see a way of making the integration better, please log a ticket or even better send through a patch!

Learn More:

Share via:
Posted in:
Community  — Company News  — Company News  — Product  — Product

Add new comment

The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.