homeblogcontinuous delivery puppet controlled chaos to well oiled machine

Continuous delivery at Puppet: from controlled chaos to a well-oiled machine


Everybody talks about how continuous delivery is desirable to deliver products faster, so you can get feedback faster, get to revenue faster, and compete effectively. That said, continuous delivery is not easy. Achieving continuous delivery requires you to focus on your internal processes.

Being a skeptical person, I asked myself, "Will focusing on our internal processes really enable us to deliver value to customers, as and when needed?"

In fact, we did significantly cut our release cycles. We also found that shortening our release cycles helped us solicit feedback along the way, and build product that really answers our customers’ needs.

When I first came to Puppet almost four years ago, we released our enterprise product with new features once a year or sometimes in eight months depending on feature set. Now we have made great strides towards continuous delivery by releasing a shippable increment every two weeks to our internal customers. This post will tell you how we made this huge change in our release cycles.

Continuous delivery isn't easy

Current thinking in the software industry demands that we be ready to release a product whenever our customer is ready — not when our company is ready. It’s also important to build and deliver software that's designed from the customer’s viewpoint. It's no longer about, "Here's a cool feature that makes us look like awesome coders." It's about releasing increments that allow us to deliver features or capabilities that customers really need.

To design for the customer, we need validation and feedback from end users even before releasing the product. That's why it is so compelling to move to continuous delivery: it helps us deliver features to customers quickly, get continuous feedback, and iterate. Once you're able to do continuous delivery, you are in control. We can release in response to market conditions and customer demand.

In reality, it is difficult to sequence all the activities that lead to a release and actually deliver value-added product on time. In my experience, there is always tension in the room when we discuss timeline and milestones. I am sure most of you have experienced this. Even with a solid plan, last-minute critical bug fixes can increase the uncertainty around whether it's possible to to deliver product on time.

Earlier this year at Puppet, we were given the goal of being shippable at end of every sprint using continuous delivery practices. We were working on shortening our cycles, and then a couple of months ago, the Puppet Enterprise team was challenged to begin releasing every two weeks (that is, releasing at the end of every sprint.)

How were we going to approach this, without disrupting what we had in flight? It's the big obstacle for every company, not just ours … in fact, we have heard this concern repeatedly from our customers when they're thinking about how they're going to speed up their own delivery cycles.

So the quest for the right approach began. We consulted many of the references we have created for our customers. You'll find these listed below in the section headed "Learn more."

How we practice continuous delivery

After some interviews around our engineering staff and brainstorming meetings with teams, we quickly understood that shortening release cycles is not as simple as it sounds. People raised a number of objections:

"Improve engineering efficiency with iterative development" sounds great, but iterative development doesn’t always mean improved efficiency. Sometimes there can be rework that decreases engineering efficiency. This depends on how we slice the feature. With practice, though, your efficiency does improve, we've found. "Reducing release hardening to zero" sounds like a myth. Incremental delivery of value will lead to not providing enough value for customers in each release.
Business always has concrete deadlines. How can we release incrementally with fixed deadlines and fixed scope? Releasing continuously requires that we execute some release steps that we would normally put off to the end - making us less efficient.

In our interviews and meetings, we discovered that shifting to faster release cycles is about more than just tools and processes; it's a mindset shift towards always thinking about the value we are delivering to customers. Keeping value-to-customers in mind, we approached our continuous delivery project via three different tracks:

Tooling & process. Support continuous promotion, integration & testing. Alter our processes, allowing us to always be ready to ship. Iterative development. Deliver the minimum workflow needed. Adjust as we deliver increments of value. Incorporating feedback loops. Build the product that customer wants. Embrace "build, measure, learn."

Tooling and process: automate and eliminate waste

First we identified the steps and milestones needed to do a release, and where we could streamline, automate and trim waste. Within our engineering processes, we reduced waste by introducing continuous promotion. This resulted in a single touchpoint: keeping continuous integration pipeline green all the time. If you don't know that term already, "green" means the code passes acceptance and system integration level tests.

When CI is red, we revert the promotion causing test failure.Then you fix the code, test and promote when ready. Shifting focus back to keeping continuous integration pipeline always green. Before this optimization, our approach was to fix broken tests causing CI to be red for extended periods. Now keeping it green takes precedence over other coding efforts. Our tooling additions support teams to test their changes in integrated puppet enterprise environment before promoting any code to mainline. This focus has also allowed us to reduce our transient CI failures by over 50%.

Making sure our pipeline was always green, triggered good practices in our development lifecycle. Including the practice of promoting code early and often. Always have a back-out strategy planned for a feature.This could be using configuration settings to enable the feature when needed.

Now we're at the point where integration is the responsibility of every engineering team at Puppet — not just a single team or a group of maintainers. We have minimized our release hardening to almost zero. Getting the foundation ready for this was a huge effort, and definitely took more than couple of months. We are still finding ways to expose the problem points with better dashboards and engineering toolkits to get visibility into failures, helping us reduce feedback cycles still further.

Iterative development and being agile

This track required us to incorporate agile practices and an agile framework into our product development lifecycle, and has been the most challenging track. Every team has unique challenges while decomposing a feature into increments of value. For instance, doing research on what functionality is most important to our users.You might also need to build infrastructure before you can develop value that is visible to users.

It helped us to keep our primary objective for the feature in view while slicing user stories. Your objectives can vary quite a bit, from getting feedback with the minimum deliverable increment all the way to delivering a product with the best possible experience. Only practice can make it better, and sustaining good practices along the way is the hardest part.

After slicing our user stories, we broke down the big rock features into incremental pieces or small batches, and iterated on these. We started agile coaching with a few teams. We set expectations with standard success criteria using scrum principles, and began measuring every four sprints (every eight weeks).

As we worked, we shared our lessons and insights with other teams, including useful metrics. Being able to demo value at end of every sprint was the shared goal. Having this goal helped us inspect small batches of feature work each iteration, and adapt as we aimed for fully iterative development.

Business deadlines are important, and can’t be missed. We usually want to deliver the entire feature with perfection, and it's hard to adopt and adapt to the idea that delivering incrementally is ultimately better. Providing intermediate checkpoints for delivering incremental value does help you gain confidence, however: You can show and share the feature or capability as it is being developed. This reassures people they're making progress, and brings the focus back to being agile. This is also crucial to validate with customers that you are headed in the right direction.

Incorporating feedback loops: build, measure and learn

The most important benefit of iterative development is incorporating feedback into product development. It is essential to tie this back to our big picture of embracing early customer feedback. We are using this sprint-based release schedule to demo more often and give our users early access to changes. We are continuously evolving our framework to insert customer feedback loops into our system.

To test out our new framework, we started with our internal customers, designating our Puppet site reliability engineering group and our customer success team as early adopters. We verified that the feedback loops were working, with tracking mechanisms in place. With our internal customer validation, we quickly learned that every release can provide value to customers.

You don't really need the entire feature, perfectly finished, to get customer validation. Sharing increments of value generated curiosity in our customers, and prompted them to provide helpful feedback, which then made it easier for us to develop software that was most useful to our customers. Feedback could be bugs, improvements or suggestions for next iteration or a new feature. Now all of us, especially our product and engineering teams, appreciate the importance of building software with ongoing customer validation and feedback.

We are now releasing Puppet Enterprise every two weeks (per sprint) and our internal customers are deploying builds on the day of release. This frequency is helping us get our customer’s viewpoint before we release the product to our external customers. We are building on this foundation and moving towards getting builds accessible to our customers as and when they need them.

Going through this transformation really told us that getting to shorter cycles isn't just about streamlining processes — it’s actually a mindset shift towards focusing on the value we are delivering to customers. Thinking about value assists at every step along the way, decomposing a feature or a capability from a vague opportunity to a small chunk of functionality.

It’s hard to constantly remind ourselves to focus on increments and be iterative. Slice a feature for incremental delivery providing user value with each increment. Iterate on each increment supporting feedback. Work on subsequent iterations applying learnings from previous iterations. Only practice can make this transition smoother. Now that I've lived through this transformation, I am confident that the new process works, even if you fail for the first few iterations.

In a nutshell, here are the key areas to focus on as you move to continuous delivery:

Define, design and develop your features by breaking them down to user stories, each one a shippable increment that provides some value to your customer. Promote code and test frequently, and document early and often. Always keep continuous integration testing pipelines green. Automate the release process, and make builds available when the customer needs them.

When I look back, I realize my skepticism wasn't about the benefits of continuous delivery. It was about whether we could actually accomplish the shift in internal processes needed to support the shift to a mindset of continuously delivering customer value. Now we have proved it: we are successfully delivering customer value with each incremental release. We are a continuously improving, well-oiled machine.

Join us at PuppetConf 2017 for insightful sessions on “The Path to Continuous Delivery”

Rajasree Talla is a senior program manager for Puppet Enterprise at Puppet.

Learn more