Salesforce.com: A DevOps Journey
I had the pleasure of sitting down with Reena Mathew and Dave Mangot of salesforce.com to discuss their upcoming talk at DevOps Enterprise in San Francisco (October 21 - 23), and their journey into the heart of DevOps. Salesforce.com was recently ranked the most innovative company by Forbes Magazine for a record fourth consecutive year, and it also ranked No. 7 on Fortune’s 100 Best Places to Work list.
Reena and Dave's talk is among a great lineup of speakers at DevOps Enterprise, a three-day conference where Gene Kim (co-author of The Phoenix Project) is assembling leaders from large, complex organizations that are adopting DevOps and sharing their transformation stories. For more details about the event (including a special discount code), jump to the bottom of the interview.
Reena Mathew is a principal architect in the Quality Engineering group at salesforce.com. She’s been with salesforce.com for eight years working on test infrastructure and various Force.com products. In the last few years, Reena has been the product owner for several productivity teams that build internal tools for salesforce.com’s engineers. Reena’s experience as a software developer, quality engineer and product owner has helped her to drive quality in different ways — and not just by testing.
Dave Mangot is an infrastructure engineering architect at salesforce.com. Dave has been at salesforce.com for more than two years and has been heavily involved in its Puppet team, as well as heading many DevOps transformation initiatives. Dave has more than 20 years of experience in IT, mostly as a sysadmin, and is passionate about web operations.
When and why did you embark on your DevOps journey?
DAVE: We started our DevOps journey in 2012. At the end of 2013, Gene [Kim] came in and talked to us about DevOps. Salesforce.com has a startup mentality, despite being around for 15 years. We have a strong desire to go fast, stay ahead of the industry and beat the competition. DevOps is about systems thinking and going faster, which appeals to our mentality at Salesforce. We always say that trust is our No.1 value, so DevOps is a natural fit for how we want to work.
REENA: Our teams are set up so that we have functional groups working together. For software delivery, you need to collaborate across functional groups to deliver a high-quality service. It’s not just about software, it’s also how the software works with the infrastructure. As Dave said, DevOps was a natural fit for how we wanted to operate as a company.
What were some of the business problems you were trying to solve?
DAVE: We’ve tried to solve many of our business problems by applying DevOps principles. Just going through the CAMS (Culture, Automation, Measurement and Sharing) model, metrics are really important, so we started a metrics initiative. Before, getting metrics required a lot of wrangling and tickets, and now we’ve made that whole process self-service. We have an internal tool, called Chatter, which facilitates sharing between different groups. We wanted to do more infrastructure as code, and turned to Puppet as our solution. Our infrastructure is growing like crazy, and we needed to accelerate our ability to keep up with all that growth.
REENA: We needed a lot more automation. We have a very extensive test automation infrastructure for many software products, which is how we’re able to release so frequently. We want similar test automation for the various tools and systems around infrastructure projects. There are lots of efforts to streamline our process to ensure zero discrepancies between test and production environments. Automation goes a long way to help us achieve that goal.
What were some of the major hurdles along the way?
REENA: DevOps is not one simple solution, and there was a misunderstanding of what it is all about. I think this is true in the industry, not just our company.
DAVE: One of the biggest hurdles was organizational. One of the things I’m most happy about was getting the senior executives to read The Phoenix Project last year. They then went on and purchased ebooks of The Phoenix Project for the entire organization, and eliminated existing organizational barriers by creating one cloud technology group. We stopped talking about ops and R&D, and started talking about how we’re one team trying to deliver a great product to our customers.
What were some of the major successes?
DAVE: We recently had our fourth internal DevOps mini-conference. Also, our metrics framework now enables people to do metrics. The pipeline team is a continuous delivery team for infrastructure, which is really amazing. We didn’t have that two years ago. We’re embracing continuous delivery for software and infrastructure. That’s a huge win for us. We’ve fully puppetized hosts, and have thousands of hosts under Puppet control. Things that were difficult in the past are now easy. We started using kanban in the last year. We started a service engineering team, which is about getting ops-minded people on teams that are delivering services to production. They are working on communicating and understanding what it takes to deliver a service in production and how to make it resilient using DevOps principles.
REENA: The reorganization to break down silos really helps ensure that everyone is responsible for quality and we’re all trying to deliver the best solution for our customers. It helps teams collaborate a lot more because this structure makes it easier for teams to engage earlier about infrastructure needs, especially for new services. The constant discussion between software and infrastructure teams really helps us proactively find issues before deploying to production. We all benefit from working together more efficiently. Everyone is interested in making our services better, and everyone is thinking bigger scale. We’re encouraging people to ask the right questions to understand what we need to deliver for the future. That’s a big win for me.
How are dev and ops teams structured to facilitate DevOps?
REENA: It varies from team to team, but the ideal team structure is scrum-based with a product owner, scrum master, and a cross-functional group including devs, quality engineers, performance engineers and system engineers, depending on the needs of the product. The team or business group is self-sufficient and can build, design and deploy its service.
DAVE: Cross-functional teams are the ones that get a lot more done. When you’re not waiting on some functional area outside of your team, you can simply go faster.
Do you have a dedicated DevOps team?
DAVE: We don’t want to have a DevOps team. We don’t even want to talk about doing DevOps. We just want the natural way the business operates to follow DevOps principles. It’s not one part of the organization that needs to be doing it; it’s an approach that the whole organization adopts.
REENA: The benefit of cross-functional teams is that even though engineers are associated with a specific function, that doesn’t mean they can’t work on other things. We have quality engineers writing software. We have devs involved in deployments. We try to leverage the strength of each functional role, but that doesn’t mean individuals can’t take tasks associated with other roles. The goal is knowledge sharing. We want to make sure people are learning about the other aspects of software delivery because that helps us build better services.
What’s been your biggest surprise?
DAVE: We’ve talked a lot about all of the successes we’ve had, but none of those successes have been easy. Culture is a hard thing to change in general. By definition, it’s something that is created and adopted over time, and therefore it also takes time to shift it for the future — especially in large, established companies.
REENA: My biggest surprise is why everyone isn’t operating this way.
What steps would you recommend to an organization trying to move towards DevOps?
DAVE: Don’t give up. It’s hard. It’s going to take time. Things may change slowly in an established organization. Don’t expect everything to just fall in line. We’re never going to get to the point where we’re DevOps now. We’re just trying to get better all the time. We’re going to keep trying to make the company better and make everyone’s job better, easier, and more fun.
REENA: You need to continuously socialize what you want out of DevOps. What are the changes you want? I don’t think a completely top-down approach works. You need to know what engineers want, as well as involve the executives. It’s both top-down and bottom-up. The key thing is change has to happen across the board. It takes a while, but you need to continuously work on it.
What do you plan to tackle next?
DAVE: Spreading what we’re doing to more parts of the organization. Some areas of the organization are slightly hesitant to change, because they want to understand more about how DevOps can help them. We’ll continue to share the success we've seen to keep adoption moving forward.
REENA: Continuous improvement and thinking big as the company grows. We need to make sure people are continuously thinking about all aspects of building and delivering a service to ensure the company can scale with its growth.
- DevOps can boost overall organizational performance, and help your team innovate. Read all about the state of DevOps today in the 2016 State of DevOps Report.