Are you using Slack to do cool things (other than chatting about Netflix episodes at the virtual water cooler)? In today’s episode, Martez Reed and Demetrius Malbrough discuss how Hydra integrates with Slack to create and scale virtual environments. If you are interested in how your environment can be automatically created and spun up behind the scenes, this episode is for you!
Learn more ways to collaborate in our free eBook, "Fostering Accountability Across Silos:"
00:00:13Demetrius MalbroughHey, everyone, thanks for joining this episode of Pulling the Strings podcast powered by Puppet. I'm delighted to be your host. My name is Demetrius Malbrough and I'm on the product marketing team here at Puppet. And I'm really excited today to talk with Martez Reed. Martez is a principal field solutions developer here at Puppet focused on the creation of technical education content. Martez, how are you today?
00:00:42Martez ReedI'm doing excellent. How about yourself, Demetrius?
00:00:45Demetrius MalbroughI am fantastic and welcome to Pulling the Strings. I'm hoping that we can shed some light on Hydra. And first and foremost, why don't you tell us exactly what Hydra is and what is it used for?
00:01:01Martez ReedYeah, absolutely. So Hydra is actually the internal self-service platform that we use here at Puppet for the provisioning of our classroom environments in AWS, as well as for the demonstration environments that our sales engineers use as part of the sales process.
00:01:19Demetrius MalbroughOK. So it is used heavily internally, I guess, from a classroom perspective, knowledge management, etc. And I guess it's as simple as what, downloading an OBA and just kind of following through on the actual steps around configuring it. Is it that easy and simple?
00:01:38Martez ReedIt's actually even easier. So the actual integration that we've built is a Slack integration. So where you can request your particular environment via a Slack web interface, which is actually pretty interesting for the use case that we have to enable some of that ChatOps capability.
00:01:57Demetrius MalbroughSo give me like one example of a Slack command or something that that I would type into the Slack channel to get Hydra to respond and do or create something on.
00:02:07Martez ReedYeah. So an example would be like create, and then a particular name of your environment, with X number of machines for X number of days. And then after that it goes out and creates the environment. And then in 20 to 30 minutes you'll have a brand new, shiny environment that is ready to go for sales demonstrations or for a classroom environment.
00:02:29Demetrius MalbroughOK. So this is not just an internal thing to Puppet. So it's public facing, and I guess anyone can go out and download it. Is that correct?
00:02:38Martez ReedSo what it actually is right now, it is closed-source internal to Puppet. And so it's a number of integrations with sort of cloud native products and capabilities. So, we're actually leveraging HashiCorp Terraform for the provisioning of the infrastructure for the environments themselves. And then we're leveraging S3 for Terraform state management. The actual storage of the state. We're leveraging containers or the actual services that integrate with Slack and actually perform the process there. And then on sort of the pipeline piece, we're using GitHub actions to actually carry out the operations in sort of GitOps methodology.
00:03:19Demetrius MalbroughSo what what are some things that may be similar if I guess some of our listeners may not have heard of Hydra and how we're using it? Can you give us maybe an example of something that's similar out in the wild that people are more familiar with?
00:03:33Martez ReedYeah. So it's actually quite similar to a lot of the CMP's, or cloud management platforms. Obviously, this one is very much scaled down and very specific to our use case. But the goal is to be a sort of a self-service mechanism that provides exactly what you need for a business use case. In this particular case, the business use case is a classroom environment or a sales demonstration environment. And so that is very tailored to that specific use case. And so there's a lot of integrations as it relates to the automation of what's going on to actually stand up that particular environment.
00:04:10Demetrius MalbroughOK. So how long has this been around? And I guess is it kind of old technology or, you know, it's being updated and you're working on maybe like a newer version of it?
00:04:21Martez ReedYeah. So it's actually been around for a number of years now. And so it's gone through various different iterations. And so where a kind of the idea actually started from was for the classroom environment previously. What was required for the classrooms to be actually delivered was that students would have to download an OBA prior to actually coming to class and they would leverage that, that would run their Puppet Enterprise master and be what they would use for the classroom environment. And so for those that have dealt with OBA's and even downloading very large files, they understand some of the issues that come along there of bandwidth and just needing to have a machine beefy enough to run a decent-sized virtual machine as part of the process. And so the thought precipitated, we should be able to leverage the public cloud with its scalability and elasticity to actually stand up environments that can basically be stood up on demand notice and then be torn down as necessary and only incur that cost at that moment in time. And so that was really what precipitated it. And so we've gone through a couple of sort of iterations or minor versions. And most recently, we've actually been working, and actually have released internally, the second iteration or second version of the platform, which is Hydra 2, and it includes the the self-service Slack interface, utilizing the Slack forms to be able to request environments via the drop downs and provide different customizations and capabilities within a user interface, which is really a great feature because the first version of Hydra was actually leveraging the command line commands. And so there's that Regex that comes along with that. And for those that are familiar with Regex or the back inside, it can be pretty tedious and hairy trying to figure out the exact way to capture the command. And for the user, it's not a very good experience because I have to know exactly what the command is supposed to be and type it exactly as "create environment name with number of machines for X number of days." And it has to be in that exact format. And so that's the challenge and kind of the beauty of the second Hydra platform.
00:06:28Demetrius MalbroughThat sounds very interesting. And you'll be proud of me that that I've done maybe just a tad bit of research here and stumbled across HashiCorp's Nomad. So just a little bit about micro services and how architecture plays a role in, I guess, the way Hydra is set up. And also, the hot thing right now, which is Kubernetes, can you maybe get into a little bit around, you know, what's Nomad, and, you know, why is it worthy of going up against, let's say, Kubernetes?
00:06:58Martez ReedYeah, absolutely. I'll start with the architecture of Hydra. And so the underlying platform that does the interaction with Slack and interacts with our GitHub environment and handles the updating of information is based upon a loose microservice architecture in the sense that we've sort of broken out the various pieces and functions of the architecture into their own containers. And as part of this process, it was needing to handle orchestration because the goal was to be able to write some resiliency for the platform. So where, like if the single EC2 instance running the platform, if it falls over, then the service is completely dead. Obviously want to provide some sort of resiliency. And so at that point, we needed to be able to orchestrate containers across multiple nodes. And so in this case, it was looking at, OK, what are our options? Obviously, Kubernetes is the hot thing that everybody knows and is clamoring for. But it was the case of, does Kubernetes in particular meet our business case, which is a fairly small group that is managing the Hydra platform itself. And we're actually just running maybe four or five microservices in and of themselves. And so some of the services are actually just batch jobs. And so it was, is Kubernetes is the right choice for orchestration or is it maybe a little bit too heavy for our needs? And so the kind of the thought became was if Kubernetes isn't it, what are our other options? And so that's where HashiCorp Nomad came up. Prior to coming to Puppet, I worked for a consulting company. And so I was fairly familiar with HashiCorp Nomad, kind of what it did, but for me, it was always where's the use case of where I would use Hashi Corp Nomad versus Kubernetes?
00:08:45Demetrius MalbroughYeah.
00:08:45Martez ReedAnd during that time, I never really found it. And so as I was going through this process, kind of the light bulb came on of, HashiCorp Nomad in this particular case is really valuable because it reduces that sort of entry level complexity. I'm not needing to dig deep on persistent volume claims and all these other Kubernetes particular constructs. It's really just I need to run a couple of containers and I want them to be resilient. I don't really want to dig deep into pods and in understanding all that that nomenclature, it's just I need to run some workloads and I need to do the thing that accomplishes the business goal. So that's where HashiCorp Nomad came into play of being really simple to get started with, as well as being easy to use for our use case, which is a smaller use case. Obviously, there's a case where those that are running thousands of microservices, Kubernetes may be the ideal choice for them, especially given the ecosystem. But HashiCorp Nomad was the the ideal solution for us. And so as part of that process, we also looked at the other options of sort of Kubernetes as one of them being EKS since the Hydra platform is running on AWS. And so it was, does this fit the use case? One of the reasons we didn't go with that was just the cost of EKS and some of the complexity that came along with that. How did things talk to one another and some of the implementation pieces there? We also looked at AWS ECS, which is their elastic container service. And so it was, yeah, I don't want to go full-blown Kubernetes, I just want to be able to run containers. And so similarly, there's a lot of moving pieces there and needing to wrap my head around some of the complexity was just, just not what we wanted to venture down. And so we ultimately settled on HashiCorp Nomad as the underlying platform that pins it. And so, in my opinion, it was a great choice, vibrant ecosystem going on with HashiCorp Nomad, as well as a company that pretty much everyone in sort of the DevOps space knows and loves the tools that they've created, such as the terraforms, the HashiCorp Vault, the Packer, or the Vagrant did provide those sort of DevOps capabilities.
00:10:59Demetrius MalbroughWow, that was a lot. You're a little bit over my head there. I only did some, like, really light reading on that. But I appreciate you breaking it down for me, Martez. What are, I guess, some of the cool things that you've done with Hydra, maybe around what you're doing now with it, but I'm really trying to get to like something that someone would say, wow, I think that really, really, really cool besides what you've already mentioned. Is there anything else you can think about that may be worth mentioning, or something you remember now that's really cool that you've done with Hydra or you think the future of Hydra could be?
00:11:37Martez ReedYeah, I mean, I think one of the really interesting things that I kind of touched on was actually the Slack interface. I mean, I know those that have used Slack have kind of seen some of the ways that that Slack can be accessed and the commands and different things. But it was actually really interesting from a development perspective to actually create the form that a user would see when requesting an environment to be able to provide easy-to-use buttons of like, opening up the Hydra app, and I want to create an environment, and then be presented with a number of options similar to what you would expect in like a cloud management platform, like a ServiceNow. I want to request this item and then be able to dig deeper on. Okay, I want this item and then be able to be presented with an actual form that guides you along the process of like environment name and number of machines in the environment and have that be specific to each individual environment. Because what we were seeing from a usability perspective was oftentimes we would get the question of someone pinging us via Slack and be like, oh, hey, what? Why is it this command working or what's the command to do the thing that I need, like a destroying environment? So of course, from like a usability perspective, that's pretty poor usability if a large number of users need to request information about how to use what you provided. And so at that point, it doesn't really feel so much like self-service as like this distant service, and so I think actually being able to leverage the Slack user interface in the form has truly made it self-service in the sense that I may have never used Hydra before. Let's say somebody's new to the company and you tell them, oh, hey, you should use Hydra to stand up an environment to test the the integration of whatever it might be. And you just tell them, oh, hey, slash Hydra. And then at that point, hopefully it's self-guided enough to where they can use the platform and stand up their own environment and not have to understand what the particular syntax is of the creation command or a deletion command.
00:13:43Demetrius MalbroughOkay. Yeah. That sounds cool enough for me. I wanted to switch gears and while I have you on the podcast, I want to definitely tap in to some current events. And right now, you know, it's no secret that we're in the midst of COVID-19 right now and the pandemic and a lot of people are out of work and people are staying unemployed a lot longer as well. I also saw a job report that showed that I think there is about 4.6 million people out of work and still looking. So that number is definitely continuing to grow. So what type of skills do you think that, let's say, you know, DevOps engineers and DevSecOps or anyone in this space. What are some of the things that you think that they should be learning or looking at right now that's going to keep them relevant or maybe take them to the next level?
00:14:39Martez ReedYeah. I mean, it's actually interesting you bring that up. I actually just had a friend reach out the other day about what he could do as far as learning cloud, cloud certifications, things like that. That's kind of one of the key things I would say, is learn a cloud in the case of a public cloud. So like AWS, Azure, GCP. Get familiar with how to use that and sort of the next iteration would be how to automate that. In the case of maybe the one of the first things you did was stand up a VPC in AWS. So how can I automate that process, whether it's the command line being like AWS, CLI, python, terraform, whatever it might be. Learn how to automate that and then walk through the process of really understanding. I know for me having an operations background, the more I started to understand about application architecture, I think was really invaluable for understanding kind of a lot of the DevOps sort of methodologies and constructs and understanding where that friction really comes in. Because traditionally I was a sysadmin, hadn't really worked in like true development shops, and so I didn't really experience a lot of that pain because it was third party applications, commercial off-the-shelf applications that we were just installing. I didn't have really direct interaction with the developers. And so for me, gaining a lot of those concepts was really invaluable. And so, like, in addition to cloud, I would say find a programming language for those that aren't programmers by trade. And just really dig into that for like a use case, like I mention, of building out, automating in public cloud. And for me, I think that's just super invaluable skill set, especially in this climate where more and more companies are adapting public cloud and basically just needing to move faster and faster and faster. And then kind of to that DevSecOps point, one of the things that I think is really critical, especially as you see the news more and more, is all of the breaches and all the ramifications of it, like being a habitat to put on that security hat, at least to some degree, I think is super important.
00:16:48Demetrius MalbroughYeah, that's definitely spot on. And one thing I'd like to say as well, if you have an excuse on, you know, learning a new technology and you mentioned learning a programming language, you know, you're only one click away from YouTube and there's so many free training opportunities out there. I saw a free Kubernetes class out there that I signed up for. I have yet to log in and actually sit down to my, you know, and start going through it. But there are no excuses, you know, as far as, you know, coming up to speed and in trying to learn when it comes to elevating yourself and gaining those skills. But what programming language do you recommend? Because you mentioned learn a program language, is there one that you recommend, or one that you see is kind of valuable right now or easier to learn?
00:17:35Martez ReedYeah. So the one I would actually recommend is Python. I mean, there's a ton of resources out there. And if you look you'll even see some for, like, learning Python for kids. I mean, it's pretty simple to at least grasp on the basic concepts. And the more and more you delve into it, you'll have a better understanding. I think the other thing I would mention is there's a really great community in the IT space where I know for me at times when I was early in my career, it was, you don't know what you don't know. And sometimes it's just really helpful if, whether it be like Twitter, LinkedIn, whatever, maybe just reach out to somebody that you think is a little more seasoned, has a particular focus or knowledge in the area you're desiring. And if nothing else, it's, hey, I'm looking to do this. Can you point me in the right direction? Because what tends to happen at times is you see all the stuff that's out there and you just get overwhelmed like, oh, man, I need to learn Kubernetes, and cloud, and configuration management. And like, what tends to happen is you get that analysis paralysis of like, I'm just so overwhelmed. I don't know where to go.
00:18:44Demetrius MalbroughYeah, I see that a lot. And, you know, just having conversations, you know, online, you know, like on LinkedIn and Twitter, etc., I definitely have friends and also cohorts that, you know, reach out to say, hey, what should I be learning? Just pick something, right? Pick one thing. Right. You see, you know what's trending on Twitter right now or you see what everyone's talking about on LinkedIn. You know, take that one thing, you know, like containers, and, you know, take it one step at a time. So I really appreciate you sharing all of those details and making us a little smarter on Hydra and how Hydra is used internally here. So I truly appreciate you coming on Pulling the Strings. Can you also please provide how our listeners can reach out to you on social media maybe?
00:19:32Martez ReedYeah. So I'm on Twitter at @GreenReedTech. And then as well on LinkedIn at Martez Reed. And that's another great way to find me and connect.
00:19:48Demetrius MalbroughAll right. Well, Martez, thank you so much for coming on Pulling the Strings podcast powered by Puppet. I really, really appreciate your time and thank you so much. Until next time.
00:20:00Martez ReedThanks for having me.