published on 28 July 2017

A few weeks ago, with Sysadmin Day approaching, we asked ourselves, "What does it even mean to be a sysadmin in 2017?" Luckily, Puppet is a company filled with system administrators and folks who once were system administrators, so everybody here is passionate about making the lives of system administers better. Nigel Kersten, our chief technical strategist, and I decided to ask our folks for their insights. Here’s what we found.

Photo of Daniel Dreier, site reliability engineer at Puppet

“Sysadmins, the various DevOps roles, and site reliability engineers are basically generalist last-mile systems integrators who stay responsible for the system once it's in production.”
~ Daniel Dreier, site reliability engineer


I couldn’t agree more with Daniel. No matter what we call it, the role has those characteristics. However, the job title system administrator is declining. One indication things have changed: in the 2017 State of DevOps survey, we saw a sharp drop in the percentage of respondents who said their title was system administrator. The most common title for our respondents (internationally) was actually DevOps engineer, followed by software engineer, architect, system engineer and then system administrator. Just the year before, system administrator was the most commonly reported title, followed by system engineer. This shift in titles is just one indication that the job has changed enormously.

For businesses and individual people alike, being competitive means concentrating on software

As we often say around Puppet (and we're not alone in this — the "software is eating the world" theme has been resounding through the industry for several years now), every business is powered by software operations. Succeeding as a business relies on how well the organization can develop, deliver and manage software, so everyone involved with software has to align with the goals and strategy of the business.

Reflecting this shift, the role of the system administrator has changed, not only in nomenclature, but in terms of what responsibilities fall to the infrastructure layer. This doesn’t mean the classic definition of a system administrator has gone away; it means that today, the role is a superset of what was already an ill-defined and daunting job description.

The way software flows through and powers the company is now seen as a competitive advantage. Many system admins are in charge of bit flow, testing pipelines, delivery mechanisms, and error rates on production services. However, they still have to be able to take care of that legacy application in the corner.

Photo of Omri Gazitt, chief product officer at Puppet

"Sysadmins are able to represent core requirements such as supportability, monitoring, scalability, security, and performance much earlier in the software delivery lifecycle. These non-functional requirements are increasingly built in as part of the initial development cycle, as opposed to retrofitted at the end, or carried as debt."
~ Omri Gazitt, chief product officer


Definitions and responsibilities are changing

We’ve moved beyond applications servicing merely back-office needs. Software is at the forefront of the business. This means the infrastructure that software runs on is critical. It’s also been changing and evolving to coalesce with applications in service of providing a less static approach to business. We’ve seen cloud, microservices and containers gain acceptance, and as that happens, these technologies contribute to shifts in the job of system administration.


Lucy Wyman, site reliability engineer at Puppet

"Rather than needing knowledge of specific operating systems or architectures, sysadmins need knowledge about tools and practices to run data centers efficiently."
~ Lucy Wyman, software engineer


Photo of Eric Zounes, data engineer at Puppet

"[Classic sysadmin problems] are replaced with newer ones such as how to effectively manage the cost of cloud compute resources. Sysadmins start to become an expert in their cloud provider and will know which cloud services they can leverage to help build a product."
~ Eric Zounes, data engineer


It’s not about care and feeding — it’s about building

At this point, we manage infrastructure with software. To make changes with software, you write it, modify it, or glue it together with the rest of your software stack. We don’t simply watch infrastructure and care for it — we build it, defined by software. We augment our hardware. We connect data centers and clouds across the globe. We write, code, test, deploy, iterate, and change that code all over again. Infrastructure engineers are developers, but in a slightly different domain.

Photo of Charles Dunbar, site reliability engineer at Puppet

"Automation, coding, and development experience is what I've noticed companies really want when they're asking or talking about a sysadmin."
~ Charles Dunbar, site reliability engineer


Optimized for recovery

Because of modern infrastructure technologies, the best system administrators don’t necessarily have to be the best at troubleshooting problems, digging deep into kernel tuning, memory optimization, or IO. When problems arise, the first choice is to redeploy. Put simply, we’ve moved from a world where we designed systems to not fail (a very high MTBF, or mean time between failure) to a world where we know failure happens, and so we optimize for recovery (MTTR, or mean time to recovery).

Photo of Ryan Whitehurst, site reliability engineer at Puppet

"More and more, successful sysadmins need to expand their reach and emphasize this higher-level, business-focused work, especially with the diminishing need for break-fix/business-as-usual work."
~ Ryan Whitehurst, site reliability engineer


New tools, new responsibilities, new complexity

Photo of Rich Burroughs, site reliability engineer at Puppet

"Infrastructures have been getting more complex for some time now, but now the real chaos seems to have shifted to the tool space. There are so many competing tools that it’s hard to even have time to learn about them, let alone evaluate them deeply."
~ Rich Burroughs, site reliability engineer


The focus on evolving infrastructure brings many new categories of tools into the fold for system admins. We’re now caring for software delivery end to end, which begat continuous integration systems, distributed source code control, cloud account management, cloud billing tools, deployment tooling, and software for capturing test results. This is on top of core services like DNS, DHCP, LDAP, OS management and security — all things we’ve done for years. The problems uncovered in managing these new tools are also solved by software.


Photo of Matt Young, director of product management at Puppet

"The classic sysadmin role is giving way to the infrastructure developer and tool builders who have coding backgrounds."
~ Matt Young, director of product management


The big upside of this change is that system administrators are now more aligned with business objectives, because software and infrastructure are aligned. Having infrastructure engineers caring about business objectives is an amazing transition that’s come about in the past decade. I’d attribute some of this to growing awareness of the DevOps movement, but some of it is just because infrastructure is now part of the business, instead of a layer (or several layers) below it.

"The business has no need for a cost-center IT cave hidden in the basement; it needs people involved at every level and every stage, using technology to make every aspect of the business operate more efficiently than ever before."
~ Ryan Whitehurst, site reliability engineer

Automation is key

The sysadmin role is evolving, from knowledge of individual components that comprise a company's infrastructure to knowledge of the systems that automate that infrastructure. So if you’re not automating your infrastructure and applications, you’re losing opportunities to somebody who is.

Automation brings speed to the delivery process, but more importantly it brings consistency and uniformity. This means individuals can learn about automation software and have a larger impact than they ever could when working on one-off systems.

The role of IT has evolved

The IT department has moved into more of a service-provider mentality. If your IT team causes friction vs. removes it, your customers (including your internal customers) look elsewhere. In most cases, IT isn’t doing work on behalf of the developer or business — they’re enabling it through self-service mechanisms such as APIs and portals. Today’s sysadmin is building and operating those services.

Photo of Gert Drapers, vice president of engineering at Puppet

"IT becoming an internal service provider changes the expectations regarding delivery and operational SLAs, flexibility and quality, forcing automation and abstraction provision services behind APIs."
~ Gert Drapers, vice president of engineering


Today, IT isn’t getting the dollars it used to. Expectations are higher, and competition is fierce. Doing more with fewer folks is expected. Ratios of what people manage are rising, and will continue to rise. Complexity and scope increase. This is why the skill level for being a sysadmin has also risen — and will continue to rise.


Photo of Gene Liverman, site reliability engineer at Puppet

"'Do more with fewer people' seems to be even more the norm now."
~ Gene Liverman, site reliability engineer


Sysadmins have always been good at doing whatever is needed. In one of my early roles, even things like moving furniture around was somehow included in the sysadmin job description. When you’re handling the bottom of the stack, you end up with everything that didn’t find a home somewhere further up the chain. Honestly, I don’t expect that to change.

Photo of Jenna Esquivel, site reliability engineer at Puppet

"Admins are now finding it necessary to look beyond their job description to gain the big-picture perspective."
~ Jenna Esquivel, site reliability engineer


So are we celebrating?

We may not all have the title sysadmin any more, but the role still exists. There’s infrastructure to run, automate, deploy, operate and learn about. To me, the fact that the title system administrator is on the decline doesn’t really make a lot of sense. I don't think we've really moved beyond system administrator so much as changed what the definition of system is.

So if you take care of infrastructure — whether you’re an administrator, operator, DevOps engineer, site reliability engineer, or something in between — today is your day. Here’s to you! The folks at Puppet are happy you’re still in there, fighting the good fight and contributing.

Michael Stahnke is director of site reliability engineering at Puppet.

Learn more

Share via:
Posted in:

 Great blog, Mr. Stahnke,

As someone who has watched this fierce transformation in a very short amount of time.  Getting stuck isn't an option...  Not constantly keeping up with technology, as to miss out on increasing product worth, customer service, reduction in "employee workload or requirements to maintain" and to drive down cost.  It takes a great group of diversely knowledgeable individuals, as well as a mix of old school and new.

I agree with Rich, regarding tooling.  The bleeding edge of technology can be a hairy place to navigate.  The more tools, the more complex the system... and similarly a required, larger, headcount.  A successful team would limit the tools used, to those which only improve the process without tying one or more engineers down to maintain it.

A couple quotes I like:

"If it hurts, do it more often"

"Getting a new Ops team all on board with a new idea can chafe. The best thing you can do to ensure a smooth transition is to get everybody on board, early"

"If you automate a mess, you get an automated mess"

"The most powerful tool we have as developers is automation"

Thanks for putting this together, Mike and team.

 

 

"The classic sysadmin role is giving way to the infrastructure developer and tool builders who have coding backgrounds." 
~ Matt Young, director of product management

This guy has no idea what he's talking about, as far as coding is really important today for sysadmin/devops eng/sre.
Still the best people in that area come from sysadmins grounds who learn how to code, than people who were coding and now learning infrastructure... Having an good idea about systems and infrastructures takes years of experience. You can be brilliant coder in less than year nowadays... One of reasons why devops/sre jobs are so overpriced today compared to coders.

Add new comment

The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.