A few weeks ago, with Sysadmin Day approaching, we asked ourselves, "What does it even mean to be a sysadmin in 2017?" Luckily, Puppet is a company filled with system administrators and folks who once were system administrators, so everybody here is passionate about making the lives of system administers better. Nigel Kersten, our chief technical strategist, and I decided to ask our folks for their insights. Here’s what we found.
“Sysadmins, the various DevOps roles, and site reliability engineers are basically generalist last-mile systems integrators who stay responsible for the system once it's in production.”
~ Daniel Dreier, site reliability engineer
I couldn’t agree more with Daniel. No matter what we call it, the role has those characteristics. However, the job title system administrator is declining. One indication things have changed: in the 2017 State of DevOps survey, we saw a sharp drop in the percentage of respondents who said their title was system administrator. The most common title for our respondents (internationally) was actually DevOps engineer, followed by software engineer, architect, system engineer and then system administrator. Just the year before, system administrator was the most commonly reported title, followed by system engineer. This shift in titles is just one indication that the job has changed enormously.
For businesses and individual people alike, being competitive means concentrating on software
As we often say around Puppet (and we're not alone in this — the "software is eating the world" theme has been resounding through the industry for several years now), every business is powered by software operations. Succeeding as a business relies on how well the organization can develop, deliver and manage software, so everyone involved with software has to align with the goals and strategy of the business.
Reflecting this shift, the role of the system administrator has changed, not only in nomenclature, but in terms of what responsibilities fall to the infrastructure layer. This doesn’t mean the classic definition of a system administrator has gone away; it means that today, the role is a superset of what was already an ill-defined and daunting job description.
The way software flows through and powers the company is now seen as a competitive advantage. Many system admins are in charge of bit flow, testing pipelines, delivery mechanisms, and error rates on production services. However, they still have to be able to take care of that legacy application in the corner.
"Sysadmins are able to represent core requirements such as supportability, monitoring, scalability, security, and performance much earlier in the software delivery lifecycle. These non-functional requirements are increasingly built in as part of the initial development cycle, as opposed to retrofitted at the end, or carried as debt."
~ Omri Gazitt, chief product officer
Definitions and responsibilities are changing
We’ve moved beyond applications servicing merely back-office needs. Software is at the forefront of the business. This means the infrastructure that software runs on is critical. It’s also been changing and evolving to coalesce with applications in service of providing a less static approach to business. We’ve seen cloud, microservices and containers gain acceptance, and as that happens, these technologies contribute to shifts in the job of system administration.
"Rather than needing knowledge of specific operating systems or architectures, sysadmins need knowledge about tools and practices to run data centers efficiently."
~ Lucy Wyman, software engineer
"[Classic sysadmin problems] are replaced with newer ones such as how to effectively manage the cost of cloud compute resources. Sysadmins start to become an expert in their cloud provider and will know which cloud services they can leverage to help build a product."
~ Eric Zounes, data engineer
It’s not about care and feeding — it’s about building
At this point, we manage infrastructure with software. To make changes with software, you write it, modify it, or glue it together with the rest of your software stack. We don’t simply watch infrastructure and care for it — we build it, defined by software. We augment our hardware. We connect data centers and clouds across the globe. We write, code, test, deploy, iterate, and change that code all over again. Infrastructure engineers are developers, but in a slightly different domain.
"Automation, coding, and development experience is what I've noticed companies really want when they're asking or talking about a sysadmin."
~ Charles Dunbar, site reliability engineer
Optimized for recovery
Because of modern infrastructure technologies, the best system administrators don’t necessarily have to be the best at troubleshooting problems, digging deep into kernel tuning, memory optimization, or IO. When problems arise, the first choice is to redeploy. Put simply, we’ve moved from a world where we designed systems to not fail (a very high MTBF, or mean time between failure) to a world where we know failure happens, and so we optimize for recovery (MTTR, or mean time to recovery).
"More and more, successful sysadmins need to expand their reach and emphasize this higher-level, business-focused work, especially with the diminishing need for break-fix/business-as-usual work."
~ Ryan Whitehurst, site reliability engineer
New tools, new responsibilities, new complexity
"Infrastructures have been getting more complex for some time now, but now the real chaos seems to have shifted to the tool space. There are so many competing tools that it’s hard to even have time to learn about them, let alone evaluate them deeply."
~ Rich Burroughs, site reliability engineer
The focus on evolving infrastructure brings many new categories of tools into the fold for system admins. We’re now caring for software delivery end to end, which begat continuous integration systems, distributed source code control, cloud account management, cloud billing tools, deployment tooling, and software for capturing test results. This is on top of core services like DNS, DHCP, LDAP, OS management and security — all things we’ve done for years. The problems uncovered in managing these new tools are also solved by software.
"The classic sysadmin role is giving way to the infrastructure developer and tool builders who have coding backgrounds."
~ Matt Young, director of product management
The big upside of this change is that system administrators are now more aligned with business objectives, because software and infrastructure are aligned. Having infrastructure engineers caring about business objectives is an amazing transition that’s come about in the past decade. I’d attribute some of this to growing awareness of the DevOps movement, but some of it is just because infrastructure is now part of the business, instead of a layer (or several layers) below it.
"The business has no need for a cost-center IT cave hidden in the basement; it needs people involved at every level and every stage, using technology to make every aspect of the business operate more efficiently than ever before."
~ Ryan Whitehurst, site reliability engineer
Automation is key
The sysadmin role is evolving, from knowledge of individual components that comprise a company's infrastructure to knowledge of the systems that automate that infrastructure. So if you’re not automating your infrastructure and applications, you’re losing opportunities to somebody who is.
Automation brings speed to the delivery process, but more importantly it brings consistency and uniformity. This means individuals can learn about automation software and have a larger impact than they ever could when working on one-off systems.
The role of IT has evolved
The IT department has moved into more of a service-provider mentality. If your IT team causes friction vs. removes it, your customers (including your internal customers) look elsewhere. In most cases, IT isn’t doing work on behalf of the developer or business — they’re enabling it through self-service mechanisms such as APIs and portals. Today’s sysadmin is building and operating those services.
"IT becoming an internal service provider changes the expectations regarding delivery and operational SLAs, flexibility and quality, forcing automation and abstraction provision services behind APIs."
~ Gert Drapers, vice president of engineering
Today, IT isn’t getting the dollars it used to. Expectations are higher, and competition is fierce. Doing more with fewer folks is expected. Ratios of what people manage are rising, and will continue to rise. Complexity and scope increase. This is why the skill level for being a sysadmin has also risen — and will continue to rise.
"'Do more with fewer people' seems to be even more the norm now."
~ Gene Liverman, site reliability engineer
Sysadmins have always been good at doing whatever is needed. In one of my early roles, even things like moving furniture around was somehow included in the sysadmin job description. When you’re handling the bottom of the stack, you end up with everything that didn’t find a home somewhere further up the chain. Honestly, I don’t expect that to change.
"Admins are now finding it necessary to look beyond their job description to gain the big-picture perspective."
~ Jenna Esquivel, site reliability engineer
So are we celebrating?
We may not all have the title sysadmin any more, but the role still exists. There’s infrastructure to run, automate, deploy, operate and learn about. To me, the fact that the title system administrator is on the decline doesn’t really make a lot of sense. I don't think we've really moved beyond system administrator so much as changed what the definition of system is.
So if you take care of infrastructure — whether you’re an administrator, operator, DevOps engineer, site reliability engineer, or something in between — today is your day. Here’s to you! The folks at Puppet are happy you’re still in there, fighting the good fight and contributing.
Michael Stahnke is director of site reliability engineering at Puppet.