We often get asked what the ideal organizational structure is for implementing DevOps practices. The answer to this question depends on a range of variables, such as the flexibility of your organization's current structure, available skills and resources, relationships between teams and team leaders, and corporate culture and politics. In this post, we’ll examine the three patterns we see most commonly.
Conflicting incentive structures created by misaligned goals and siloed teams do not produce the best outcomes. At Heartland Payment Systems, for example, these misaligned incentives once turned software releases into painful, months-long cycles. “Ops was meant to keep things secure and stable, but Dev wanted to move very quickly,” says Jez Miller, Heartland’s director of operations. That's why Heartland and many other companies have realigned their teams around common goals, processes and tools as a necessary step in their successful DevOps transformations.
Type 1: Close-Knit Collaboration Between Dev & Ops
This structure takes once-siloed development and operations functions and redeploys them as tightly woven, highly collaborative teams working side by side. Consider this advice from Sam Eaton, director of engineering operations at Yelp: Begin with a willingness to experience someone else’s pain points first-hand as a means for building trust and incremental change. “It helps if you’ve done what they do, so you should take the on-call rotation, take the alerts, and walk a mile in the other person’s shoes before you speak up about what needs to be changed,” says Sam.
For Jez’s team at Heartland, aligning development and operations on a common toolchain boosted collaboration while dramatically speeding up deployments and reducing errors. In fact, the company no longer tracks downtime because it occurs so rarely.
“Conversations with development went from, ‘oh that’s tough, we don’t know that, we aren’t experts’...to ‘we can change that easily because we know we can run those changes through QA, we can validate them, and treat the infra[structure] as a code base as well,’” Jez says. “And that fundamentally changed what we were able to do as a company, and the speed with which we were able to deliver.”
This approach also maximizes the blend of software engineering skills and deep systems knowledge most organizations need. That combination can be tough to find in a single person, which poses problems for the "NoOps" approach. “NoOps” is typically seen in startups or smaller companies where the devs have enough systems and infrastructure knowledge to get the app up and running, but lack the deep knowledge required to scale or to succeed in a large enterprise setting. Similarly, the completely embedded ops team, in which systems people can code and devs have extensive operational expertise, is relatively rare, and often only found in technically mature, highly innovative web companies like Netflix or Facebook.
Other organizations, then, are better served building upon the individual skills and expertise they already have in house. Make sure dev and ops teams are collaborating and sharing responsibilities throughout the software delivery lifecycle — from initial planning all the way to managing the production environment. Many organizations employ a “You build it, you run it approach” meaning that developers are responsible for operating their applications in production. This builds empathy and weans everyone off the habit of thinking: “That’s someone else’s job.” Ultimately, no matter what team you’re on, you should be working towards well-defined business outcomes.
CenturyLink's vice president of product, Richard Seroter, writes on his blog about the tight partnership between development and operations in his own cloud organization: "There’s no 'us versus them' tolerated, and issues that come up between the teams — and of course they do — are resolved quickly and decisively."
Type 2: Dedicated DevOps Team
Dedicated DevOps teams are often made up of experienced operations people with a mix of skills including using version control, writing infrastructure as code, and continuous delivery. These teams typically start by addressing the things that are most painful, such as deployment automation, and if they’re successful, can evolve to providing shared services for the rest of the organization. In our 2015 State of DevOps Survey, we wanted to better understand what services these DevOps teams were providing. Here’s what we found.
As Matthew Skelton notes in his great blog post on team typologies, you do need to watch out for a pitfall of the dedicated team: Even if it's temporary, it can become just as much of a silo as the functional teams that preceded it.
But if you avoid that pitfall, there's evidence for both the rapid growth of dedicated DevOps teams and their efficacy. One surprise in our 2014 State of DevOps Report was that 16 percent of respondents identified themselves as working specifically in a DevOps department, and 90 percent of those DevOps team members were working in high- or medium-performing IT organizations. Preliminary analysis of survey data for our 2015 report shows that 19 percent of respondents now work on a DevOps department. Watch out for the the 2015 State of DevOps Report coming soon for more details!
Type 3: Cross-functional teams
Cross-functional teams consist of representatives from all disciplines responsible for developing and deploying a service (business analysts, developers, quality engineers, ops, security, etc.). These teams are fully empowered and self-sufficient — they write it, test it, and deploy it.
Gareth Rushgrove, senior software engineer here at Puppet Labs, helped build that kind of team inside the Government Digital Service in the UK. Here’s what Gareth had to say about it: "Rather than a traditional [structure] — the operations people over there, the developers are over there, designers are over there, everyone in separate teams by discipline — our structure was very much teams around a product, around a thing.”
In some cases, these teams are temporary, in the sense that they’re created by drawing people with the right mix of skills and experience from different areas of the company — and external resources as needed — to work on fast-paced, time-boxed projects.
GE Capital's DevOps journey, for instance, has relied on interdisciplinary teams pulled together for six-month projects, with shorter-duration sprints baked into that timeframe. There are a variety of names for these kinds of teams: "agile" appears quite a bit; "tiger team" is another.
The combination of being self-sufficient and also having shared incentives is powerful: Cross-functional teams work more efficiently, with fewer hand-offs and more opportunities for knowledge sharing.
There is No “I” in Team, or Is There?
DevOps is definitely a team sport, but it's still important to clearly define individual roles and responsibilities for success within the team. Here are some recommendations:
- IT manager: Build trust with counterparts on other teams; create a climate of learning and continuous improvement; delegate authority to team members
- Dev manager: Build trust with Ops counterpart; bring Ops into the planning process early.
- Systems engineer: Automate the things that are painful.
- Quality engineer: Provide input into scale and performance; provide feedback on staging environments.
- Devs: Plan for deployment as you're planning new features; get feedback from Ops and work with them on deployment process.
Regardless of your organizational structure, DevOps requires collaboration across multiple functions and shared responsibilities for ensuring the best possible quality.
And the rewards can be tremendous for everyone, as Jez at Heartland Payment Systems discovered: “Stop the firefighting, keep consistent platforms, let people sleep at night, give people their lives back, and really make their development lives and their operational lives easier, and make them successful within their businesses.”
Isn't that what it's all about?
Alanna Brown is senior product marketing manager at Puppet.
- DevOps can boost overall organizational performance, and depends strongly on IT leadership for success. Read all about the state of DevOps today in the 2017 report.
- Are you running into skepticism about DevOps before you've begun? Here's how to build the business case for DevOps.