Real-World Platform Teams: How HelloFresh Does Developer Happiness

Let's get real-world perspectives on the benefits and challenges of the platform engineering approach.

Platform Teams Deliver on DevOps

Our 2020 State of DevOps Report focuses on two areas that can help organizations scale their DevOps initiative: a platform approach to software delivery and applying DevOps principles to change management. We found that internal platform usage is widespread — 63 percent of respondents report their company has at least one self-service internal platform. The teams that build, govern, and maintain these platforms are dedicated, cross-functional subject matter experts and they are called — to no one’s surprise — platform teams.

Say Hello to HelloFresh’s Samantha Coffman

Kicking off the series, we are thrilled to feature Samantha Coffman, senior product manager, Engineering Experience at HelloFresh. HelloFresh’s engineering organization, known as HelloTech, has over 500 engineers distributed around the world who rely on the developer tooling that Samantha manages. One of our key findings from the 2020 State of DevOps Report is that a product mindset is key to scaling DevOps and your platform. Highly evolved firms are nearly twice as likely to be highly product-oriented as firms in the middle of their DevOps evolution. HelloFresh has embraced a product mindset by empowering product managers, like Samantha, to focus on improving the developer experience for engineers at HelloFresh.

Note: In this post, we refer to “platform,” “Platform” and “platform team.” Lower-case “platform” refers to the technology while upper-case “Platform” is short-hand for the “platform team” and related department.

Alanna: What does a platform team mean for you and what are the essential attributes?

Samantha: I just read Team Topologies and it encapsulates how we operate. We treat the platform as a product. We’re building a product that will work for engineers, data analysts, or whoever is using the platform. We back that up with data.

The essential attributes that make it work are the product mindset, splitting domain expertise into smaller subteams, and evangelizing the “you build it; you run it” mentality throughout the organization. We are not an IT help desk. We're not there to build things and get things up and running for you. If you are designing a service, you should be able to build, test, develop, run everything yourself. We enable this at HelloFresh by providing a self-service platform that takes into consideration each of these steps in the development lifecycle and ensures the experience is top-notch for our developers.

Alanna: Why did HelloFresh decide to move to a platform team approach? Did it happen organically or was it a deliberate shift? What was the tipping point to formalize this team? What problems were you trying to solve?

Samantha: I joined after we had just made this very deliberate shift so I can’t take credit for it. As the engineering organization grew, the complexity of the software grew. Our VP of Engineering, Renato Todoro and Product Lead, Jessica Ulyate, spearheaded the effort to extract complexity out of the feature teams to create the platform team. The teams were initially split into green and blue teams, but the complexity within those teams grew to be too large. That’s when they decided to divide the teams by domain. That’s where we saw the product mindset thrive because each product owner was assigned to one of the teams that make up Platform.

Now we have Engineering Experience, of which I’m the product manager. We have Cloud Runtime which handles infrastructure; Site Reliability which handles reliability of services and the whole incident management process. Finally, we have Data Infrastructure, which manages the services and tooling to support our Data organization. Data infrastructure is a new squad in the Platform team. At HelloFresh, we’re moving towards a data-driven product mindset so it was important to enable this so the data team felt supported by Platform (the work they do needs Platform-dedicated support). Our platform leads always look for those types of gaps in the organization and for ways to continuously advocate product mindset.

Alanna: What were some of the early successes that made you believe this was the right approach?

Samantha: The main performance indicator for us with this platform approach were developer satisfaction and happiness. I think those two are proxies for developer productivity because if you have happy engineers, they will work more effectively. Our North Star is the four DevOps metrics popularized by the State of DevOps Report and DORA’s book, Accelerate. We call them the Fab4+. It took a long time to adopt them, but they help us dig deeper and diagnose where the problems are.

We do a couple of things to understand how happy our engineers are and if what we’re doing is actually solving their problems that were identified in our Fab4 metrics. We run a developer happiness survey every 6 months. This is a combination of questions that are consistent across each survey and new questions that come up as the organization evolves. This enables us to benchmark progress and understand areas we should be investigating. To complement the survey and eliminate bias, we also do random sample interviewing. I take the list of all the engineers in the company and interview one person per week.

I also bring along someone from my team of nine engineers from Engineering Experience because I was having a tough time organically building empathy for customers with these engineers. I felt like I was trying to sell them on something they couldn’t relate to because they are such advanced DevOps engineers that they felt like, “The logs are right there. Why didn’t they look at the logs to see why CI didn’t run successfully?” But what I was trying to show them is that everyone has different contexts, ie. they’re working on some business logic that you might never think of so they don’t have time to think about why their CI is failing — it actually holds them back to think about that. So having these interviews has already had an impact on my team in terms of building empathy with other engineers in the company. This is how we get the anecdotal understanding of how happy the engineers are.

In the survey we have a set of NPS questions that don’t change. These are our benchmarks. We cross-reference this with our internal survey of all employees and slice the data by engineers to see if it matches our benchmark. So at the top, we have the Fab4+ metrics which helps us diagnose where the problems are. For example, if our lead time to change is high, an initiative we might focus on for the quarter is improving developer productivity. We would then look at the entire journey and ask what’s the feedback loop for getting a PR reviewed or seeing if their CI has passed.

Alanna: What are the core responsibilities of the platform team?

Samantha: I always say: we’re an enabling team, not a support team.

Alanna: How has the platform approach changed how you interact with other teams?

Samantha: As a product manager, I always look at other teams as customers and encourage my team members to look at them the same way. On the reverse side, engineers look to Platform like they look at the technology radar. They look to Platform for what they should do next and they really respect and admire the platform engineers as the gold standard for how they should be doing things.

We don’t force standards or force people to build a service in a certain way. We have a standard way of doing things, but we don’t block people from trying new things. It’s a balance, you run your CI in our tool but you also have options. This balance of autonomy and standardization has created a sense of respect for Platform. Our customers feel like Platform is looking out for them, but that they can also do it themselves. This is really tricky to cultivate but this is what our platform leads have done really well: respecting engineers and who they are as people and how they operate but still providing a self-service platform that works really well for the rest of the organization’s needs.

We’ve also improved how we enable and provide support for engineers who are stuck. We created a Slack commander bot so you can type /issue in our public channel, this then comes up with a form to help diagnose the issue and provides helpful links. The form asks things like, what do you expect to be happening; show us your error logs, etc. It helps Platform feel like they have control of what’s coming in and it helps them debug issues better. We collect all the questions and answers in a Slack channel so people can search. If someone has a question, they can search and see if it’s been answered before. It’s like our own homegrown StackOverflow, but on Slack.

This has been really helpful with the move to remote working. Before, we sat really close to the squads in HelloTech and could just walk over to someone’s desk and ask a question. This has made it easy for people to ask a quick question or search for an answer and find what they’re looking for. No one notices Platform unless something goes wrong. When things go wrong people can get upset because it affects their daily work and productivity. The way that people ask questions is so much more productive now. We’re giving people more information to make decisions themselves. This has empowered our customers and the platform engineers and leveled up the respect and trust between teams.

Alanna: What has been your hardest challenge to date and why?

Samantha: Quantifying success. We don’t have a systematic way to do this so we have to take all the data we have from the survey, talking to people and using the Fab4+ metrics to see where the problems are. Once we’ve identified a problem we try to solve it, then it’s normal product management from there. (Where is the tool being used? Is the lead time for changes going up?) It’s very difficult to do this scientifically because we can’t experiment. We can’t run A/B tests on our engineers, for example, so it’s hard to figure out if what we’re doing is making a difference. That’s why we have all these different inputs of information.

Alanna: What kind of attributes do platform engineers have? What skillsets or characteristics do you hire for?

We hire very senior DevOps engineers. Our management in Platform does a really good job of communicating the vision of Platform so everyone knows how to act in public channels. If we get a request for support, it’s in our best interest to help them because they’re our customer. The engineering managers partner closely with the product owners which models the desired behavior for the rest of the organization. So it’s a combination of hiring senior talent, communicating clearly, cultivating a psychologically safe team environment, and implementing a structure with squad leaders and a product owner embedded in each squad. Also, most of our Platform engineers have worked at the company for a long time and have moved between many of the teams so they have really strong relationships with other teams.

Alanna: How does your team think about DevOps / support DevOps practices?

Samantha: The “you build it; you run it” mindset is the most important aspect within the scope of my job. For Engineering Experience in general, we want any engineer in any squad to be able to manage the full lifecycle of their service. We care about making it a good experience for them. We don't expect teams to fully operate everything but we want them to be able to operate their service, make sure they can test it and that it’s reliable. Every person in their own way should have a DevOps mentality the way Platform does. The way we do that is we make it really easy to operate in that way and we productize it.

Alanna: For teams considering this approach, where should they start?

Samantha: Team Topologies is top of mind. Identify where the most cognitive load is on teams; how much cognitive load is on any given engineer today and split by type of engineer (front-end, backend, QA, automation, data). This will help prioritize which things need to be extracted first into a platform team or guide the team organization. Focus on the engineer’s experience. I like to ask questions like, “How do you feel about deploying? How do you feel about testing, monitoring, incidents?” You get the most honest answers from people when you ask it that way.

Samantha Coffman (she/her) is a product owner, D&I advocate, and foodie. Sam works at HelloFresh as a Senior Product Manager for Engineering Experience. In addition to building products that foster developer productivity and happiness, she is passionate about building more empathy, inclusion, and diversity within tech organizations.

Check Out Puppet's Scaling DevOps Service