Rhommel Lamas is a site reliability engineer at eBay Classifieds Group in Sydney, Australia. Rhommel helps companies embrace best practices around configuration management, and he's especially interested in containers and schedulers. His PuppetConf talk, There Is a Storm Coming: Untangling the Strings of Your Puppet, will focus on taking a step back once in a while to look at the path your repository is taking, and how code review and pull requests help a team work well together.
We asked Rhommel if he'd answer a few questions ahead of PuppetConf so we could get to know him better.
Aliza Earnshaw: When and how did you first start with Puppet?
Rhommel Lamas: It all started at the beginning of 2009, when I moved from Venezuela to Spain. I joined an ISP where we had to manage something around 1,000 bare-metal servers, and I refused to do it using complex shell scripts. At that moment, people were struggling with configuration management, and the deployment of new servers was taking anywhere from six to 10 hours between the moment the servers arrived in our office and the moment they were ready to use in the data center due to data center transport availability. Later on, we started deploying Puppet 1.7, and everything started from there. Servers were shipped directly to the data center, and they were ready to use in less than two hours, including rack time. This triggered the necessity to virtualize our hardware, and with a Puppet infrastructure in place, everything became noticeably easy.
Aliza: You're using Puppet with Mesos at Gumtree Australia, which is part of the eBay Classifieds Group. Can you give us a bit of detail about that?
Rhommel: Puppet is a core piece in our infrastructure, and we rely heavily on it to provision our Mesos nodes. Things like Zookeeper, Docker, Mesos and all frameworks that we run on top of Mesos are managed by Puppet. This way, new servers are provisioned in less than 10 minutes, and that helps us scale as much as we need.
Aliza: I also see you're using Puppet with Docker and Kubernetes. How does Puppet fit in with your use of containers and a container scheduler?
Rhommel: As I mentioned, we use Docker within Mesos. There are some applications that we prefer to spawn using Docker for the simplicity, and some others that we are running as Mesos containers while we migrate them into Docker containers. We use Puppet to make sure servers are properly configured and consistent. Schedulers gives us the leverage to run dozens of heterogeneous systems per machine, and misconfigurations can potentially affect the reliability of the whole site. Puppet helps us to keep this up to date and safely delegate delta configurations to less skilled members of the team without worrying about incidents.
Aliza: What's your best advice for anyone who wants to get started with Puppet?
Rhommel: Puppet is such a powerful tool that allows you to do the same task in many different ways, and this is what makes teams debate between different config management tools. In the operations engineering field, we love to take complex approaches to solve simple problems, and this is why most of the time outages happen. So my recommendation to new starters it to think twice when writing their code, and just because you can doesn’t mean you should.
Aliza: What are the most common problems you see when teams are trying to improve their configuration management?
Rhommel: One of the most common problems that I’ve seen, even after 10 years of using Puppet, is that many teams still operate in a siloed environment. They don’t want other teams to put their hands on their code in some cases because of impostor syndrome, in some other because they haven’t built a trust relationship with their team. Changing this mindset is not necessarily the easiest work, but it pays back when you see teams working together.
Aliza: Is there a mindset or approach to configuration management you think is most helpful?
Rhommel: Keep it simple, creating complexity only where it is really needed. I’ve seen far too many Puppet repositories where engineers get scared of changing simple values like the number of processes nginx should run on a node, and they validate it running Puppet on tens of servers to ensure nothing changes before merging a pull request. This is just the wrong approach. When you decide on taking the infrastructure-as-code path, you should embrace all the practices around development, build your test suite, do code reviews, test your code. The tools are out there, and as time goes on, they are easier to integrate.
Aliza: What's a fun fact about you that we might not know, but would like to?
Rhommel: After living in Europe for almost eight years, I had to move to Sydney to visit Amsterdam (Netherlands) and Aarhus (Denmark). It is common when you live in a city to leave all cool stuff for later. This year, I had an opportunity to go back for an internal hackathon organized by eBay at one of our headquarters. Pro tip: Leaving things for the future is a bad idea; it might not ever happen.
Aliza: What's the most interesting thing you've read or watched recently?
Aliza: Go ahead, tease us: What can we look forward to in your talk about untangling Puppet strings? (Love the title, by the way!)
Rhommel: In the past four years, I’ve changed jobs frequently. I went from being an old timer at a startup to been that guy, the New Hire — the one who changes the number of nginx processes on the frontend servers, and everything goes down. The idea of my talk is to focus on self-awareness, and why it is important for teams to just stop once in a while and question what they are doing.
Aliza Earnshaw is the editorial director at Puppet.