Fighting the Good Fight: DevOps at Raytheon
What does it take to change the culture at a decades-old company in a highly conservative and risk-averse industry? Terri Potts, Engineering Fellow at Raytheon, a major American defense contractor, explains all in her upcoming DevOps Enterprise Summit talk, “Leading the horse to water, and enticing him drink: A DevOps Success Story.”
Love the title? So did I and I wanted to hear more. I had the pleasure of chatting with Terri about how Raytheon has made inroads with DevOps, enabling them to achieve higher operational efficiency and lower tax payer costs.
What’s your role at Raytheon?
I’m a Software Technical Director. We have approximately 1500 software developers across roughly 100 programs all over the US. I set technical direction and try to move the organization towards more modern techniques and methodologies.
When and why did you embark on your DevOps journey?
We started this journey about 5 years ago. It’s been a twisty, windy trail getting to this point. The reason DevOps is important to us is we build programs for the government. It’s imperative that we get more efficient at what we’re doing to lower tax payer cost and deliver more capabilities for the same cost. We need to get the right product out to the war-fighter faster.
What were some of the business problems you were trying to solve?
There’s a tendency in defense and aerospace companies to organize functionally. We felt that if we could bring the different disciplines together, including Ops, we would be able to deliver a better product that more matches what the end user needs. We were also looking to deliver faster because there are heavy testing requirements for any DOD product. The traditional way of manually integrating and testing puts a large burden to our programs and really slows things down. We wanted to show the operators more quickly what we were working towards so we could quickly make course corrections, while also making it possible to test more thoroughly and quickly.
What were some of the major hurdles along the way?
The number one hurdle is “That’s not how we’ve always done it.” It’s hard to change the culture. We’ve also faced hurdles with program management within the company and with our customers that have concerns that these changes will cause risk and might cost the program in terms of time and dollars in labor.
What were some of the major successes?
As much as I’d like to see a major revolution, the successes have been the small evolutionary changes. For example, getting a team to go from executing two integration procedures a month to running 27 procedures overnight. Being able to automate the builds, run all those procedures and know that the build is still healthy is huge. Also, we’ve seen success in connecting developers and operators, who are often light years apart. We set up a team that demonstrates to the operators every 3 weeks. The operators provide feedback and ultimately help build a product that they love.
How are dev and ops teams structured to facilitate DevOps?
We’re still working on that. We’re making inroads, but it’s not a wholesale change. We still don’t have the operators, for example, writing stories, or giving us direction, but in some cases they are participating in sprint reviews. They can give us feedback; they can tell us what they like and don’t like. It’s tough in our environment to integrate them, but we have had some successes with a couple programs in getting both groups interacting more regularly.
Can you describe the processes you’ve put in place to facilitate communication across teams?
We set up web meetings for sprint reviews. We try to keep demos simple so developers don’t have to spend a ton of time preparing. We send out a list of stories that have been implemented in the current sprint and the developers walk through the stories and demonstrate the capabilities. Both the operators and the acquisition agency are there and can provide feedback.
The feedback on the process has been phenomenal. We’ve seen increased quality and happier end users. Internally, the feedback is not always positive because of the separation between software and integrators or testers. The work that the software team does to automate testing, to make sure they’re building the right product, doesn’t pay off in their costing, but it pays off down the road in the integration and test cycle.
What’s been your biggest surprise?
My biggest surprise has been the resistance by the customer to change. They have a fear of risk. On some programs they have to approve our processes and in some cases, we haven’t succeeded in convincing them to move in this direction.
What steps would you recommend to an organization trying to move towards DevOps?
It’s worth eating the elephant one bite at a time. You can’t change the whole program, but you can start by getting a few of your sub teams going in the right direction. Sometimes people are so busy trying to hit their next milestone that they can’t take the time to automate their programs. It can be helpful to bring in someone from the outside to automate a few tests or builds and give the team some examples to build on.
What’s do you plan to tackle next?
The successes we’ve had are great, but we are a long way from shifting the whole organization. The next step is to spread DevOps across whole organization, get everyone to see the benefits, and make it a real culture shift so it becomes the norm instead of the exception.
- DevOps can boost overall organizational performance, and help your team innovate. Read all about the state of DevOps today in the 2016 State of DevOps Report.