homeblogdevops and security cultural changes bring dev sec ops together

DevOps and security: cultural changes to bring Dev, Sec & Ops together

DevOps has been around for some time, long enough to have some commonly accepted terms — for example, CAMS, which stands for Culture, Automation, Measurement, and Sharing. The inclusion of security in DevOps is, as a culture shift, still in its infancy. While some people talk about DevSecOps, I’m not sure we need a whole new acronym, or a story for how to get Dev, Ops and Security all working together. I think the answer is, we just start to do it, and we fumble, adjust, and improve.

Luckily, people are starting to talk about it. I see Francois (@F_Raynaud) of DevSecCon, the AppSec USA conference, and many others, too, starting to talk about DevOps and Security together.

I’m partial to this talk from DevOpsDays PDX (transcription can be found at the bottom of this post). It was given by Adrien Thebo and Chris Barker of Puppet, and Ben Hughes of Etsy.

You might also enjoy a similar talk given at AppSec USA. This one was given by Adrien, Chris and Bill Weiss, who also works at Puppet. I've worked with all four of the people speaking in these talks, and I feel lucky to know and work with them.

What I learned from and appreciated in these talks

First of all, I really loved the emphasis on how important it is to talk to each other, be open, and feeling empathy for the other person’s (and other team's) job challenges. It’s hard to build and maintain infrastructure and software, and you're not the only one finding it difficult. Few people are trying to make your job harder, and it's important to remember this, especially when things go wrong.

I also paid attention to Ben raising the issue of how difficult it really is to shift the culture. Even though we all talk about the importance of getting together and communicating, note that Ben actually says this presentation is the most he'd talked with developers and operators since he started working in security.

I’m of course partial to the idea of doing an internal three-day mini con. A three-day mini con? What the what? Actually, it’s a cross-functional group comprised of stakeholders from within the organization, and together, they go through multiple sessions of brainstorming. These sessions are lead by the product owner, who provides the roadmap as planned, manages feedback, and keeps conversations on track

The brainstorming provides two critical things for future collaboration: 1) a shared knowledge of future plans, goals, and possible friction points, and 2) a sense of communitas among the participants that enables better asynchronous communication via tickets or email. In short, it forces everyone to talk with each other, and hash out any gotchas early and often, thus breaking down silo walls.

Finally, I really appreciate how both these talks emphasize that moving faster means we need tighter communication loops between Dev,Ops and Security. (Also, pancakes. Pancakes always help.) These tighter communication loops work to everyone’s benefit. Because really, who wants that 3 AM page?

There are a variety of ways to improve communication between the different groups. Sending developers and operations people to security conferences, or vice versa, is a great idea. You could host lunch-and-learns or other gatherings to make sure Dev, Ops and Security all hear why a feature is important for the company and its users — and get a chance to ask questions, too. Encourage finding ways to the compromise and the yes, rather than getting to the point where Security wants to unplug everything, throw it in a ditch, and pour concrete over it. That’s a symptom of a problem, not a solution. Whatever means you decide on to encourage communication, empathy, and sharing, the important thing is to break down the silos, so we aren’t tossing Jira tickets or angry emails at each other.

Remember this: We are all humans. Change is hard, it’s scary and it can’t happen overnight. Regardless, it is well past time for us to start making these culture changes. Once Dev, Sec and Ops are all communicating regularly and amicably, we can all succeed together. If we don't make these culture changes, we will all fail together.

Beth Cornils is a senior product manager at Puppet.


DevOpsDays Portland 2016: Adrien Thebo, Ben Hughes, Chris Barker - Everything is Terrible

Alice Goldfuss: Today's speakers, for this first panel, were the unfortunate consequence of failed cloning. One of them came out British, which was especially disappointing.

Male voice: Ooh, shots fired.

Alice Goldfuss: It's okay. They're not part of Europe anymore.

As they were being sent to the furnace to be disposed of as medical waste, they managed to escape by making a significant Rube Goldberg machine out of Cat 5 cables and ever since then they've been wandering from city to city performing mediocre rock shows and bank heists.

And while we do have the cops standing outside, we'd first like to welcome to the stage, they are, Chris Barker, Adrien Thebo, and Ben Hughes.

Chris Barker: So this is our talk, titled "Everything is Terrible: Three Perspectives on Building, Configuring, and Securing Software (An Inspirational and Aspirational Talk)."

First, some introductions. I'm Chris Barker, otherwise known as @mrzarquan. I tend to give people a lot of suggestions. Sometimes they follow them. I've been doing operations for a while, then about four years ago I decided to go to Puppet.

Adrien Thebo: I'm Adrien Thebo. I started in Puppet like five or something years ago. At some point I decided that I wanted to cause people mild retina damage when they look at me, so there's that. Currently a developer and very angry about security.

Ben Hughes: What have I done? I'm Ben Hughes. I used to work at Puppet and I'm not convinced I don't still. I'm now at Etsy, which is relatively obvious, and I work in security so I consume a lot of gin.

Chris Barker: We wanted to talk about how the world was before DevOps came about for all three of our roles, being in operations, development, and security. On the operations side, I'm sitting there kind of waiting for somebody to hand me a tarball maybe, or just a bunch of files to deploy on a server while I'm in the middle of doing all my other work, of trying to actually spin all the plates, keep things from working, or not working in some cases.

Adrien Thebo: So I'd be in the role of the developer, given any number of requirements, some of which may be physically impossible, varying deadlines, scope change. I want to ship the software as quickly as I can. Get my builds out, and hopefully it will pass QA and then throw it over the wall.

Maybe it'll be in Google Drive. That's a good deployment mechanism, right?

Ben Hughes: In a Google document. A tarball in a Google document.

Adrien Thebo: Challenge accepted.

Chris Barker: After I'd get that I'd get it up and running hopefully along with trying to get all the other things working as well, then maybe a couple weeks later I'd tell Ben, or he would find out.

Ben Hughes: Then I'd — what's the polite way of phrasing this? — panic slightly? Lose my decorum and go, "Why is this on the public internet? Why are there all these ports open? What idiot wrote this code? I'm just going to turn this off because I like saying no to people, and I'm quite mean."

Chris Barker: So, after this it goes back to maybe a three-month cycle from the time that Adrien actually built the software to somebody actually says we can put customers in front of it. What this really leads to is one continuous tire fire.

But now, Adrien and I have gone to DevOpsDays. We maybe use some configuration management tools. We're in this really awesome agile DevOps world, right?

Adrien Thebo: Yeah, we're sitting around the campfire singing "Kumbaya." We are having very quick iterations. I'm deploying four times a day. I'm shipping all sorts of random new code all the time. I'm delighted because I can put something into production right away. It's great.

Chris Barker: From my standpoint, I'm able to build servers really quickly. I've probably moved a lot of my workflow over to AWS for dev and test. I'm able to deploy these things called security groups, which are kind of like firewall rules. I don't really know how they work, but they kind of have set up the way that I want.

They're firewall rules written in JSON. That's all that they are.

Chris Barker: That makes so much more sense now. We're still very happy but for some reason there's still some kind of friction going on.

Ben Hughes: What I see is, my orange-haired friend is able to pick any library he finds on GitHub or SourceForge and just throw that in the code today, or even like over lunch. Whereas before I had three months to try and find all the bugs, now I have three minutes. There's even more random libraries going up.

There's even more random servers going up. No one can configure files in JSON. Of course the natural language of files is XML, because that wouldn't go wrong. We've moved from a world where I had three months to panic and — Am I allowed to swear? Is that allowed? Organizer, can I swear?

Alice Goldfuss: I thought I was the insult. Do not do gendered slurs.

Ben Hughes: All right. I lose my shit — thank you, Organizer — because there's now way more stuff on fire than before. Whereas there was only slow-burning things, there's now the slow-burning things from before, and the tiny, miniature, artisanal tire fires in AWS.

Chris Barker: So what really becomes the problem here is that there's actually this whole different misalignment, right? Even on the DevOps side, we're still trying to get to this point of people collaborating together and working on things.

But what does that really mean when an organization maybe hasn't embraced this change throughout the whole structure?

Adrien Thebo: From a development perspective, my responsibility is to take product specifications and ship them. I'm trying to figure out ways where I can ship new code. Ship things rapidly. Take the things that I was given and make them reality. I want to be able to build new functionality without necessarily having to reinvent the wheel.

If I have to do image processing, it would be convenient for me to use ImageMagick, or something like that. Because I don't want to —

Ben Hughes: Please do.

Adrien Thebo: And this is how I'd do the bank heists. Building new software and minimizing the reinvention of wheels and trying to work quickly, effectively, and this creates a whole lot of fire.

Ben Hughes: I'm also not convinced no developer want to reinvent the wheel, because every developer thinks they can invent a better wheel.

Adrien Thebo: I don't think, I know.

Ben Hughes: I love you developers, really.

Chris Barker: As someone who is working in operations —

Adrien and Ben hug

Ben Hughes: That's the magic of DevOps right there.

Chris Barker: That's for the end of the talk.

Adrien Thebo: More hugs are better.

Chris Barker: So from the operations standpoint, my priorities are really trying to keep the plates that I have spinning, the backlog of work minimized. I'm sitting there working on trying to triage all of the existing projects. Some of the work that a development team may have done has gone off to another project. Now I'm the one who is actually in charge of the care and feeding of a project.

Probably a bunch of Red Hat servers sitting around as well. Some of those are enterprise Linux Red Hat, other ones are just Red Hat, and maybe NT4.

Ben Hughes: Still newer than CentOS.

Chris Barker: Yes. And so really from my standpoint, I like for things to stop burning. I'll take stability over any new features that can come into the environment. And I'm really trying to keep my 3 a.m. pages down. Honestly, in areas, I'm making decisions that are based on how to minimize getting woken up at four in the morning.

I'm also doing things that are causing Ben's problems. I turn off firewalls because, dammit, I need to just make this thing work and nobody told me what ports the service is actually communicating on.

So I'm just going to open up everything until things start working and the app starts running again so I can close the ticket. I'd like to be able to work on my backlog but it's getting longer and longer and longer. Most of the time that means I just can kind of stand around and watch stuff.

But it's in a perimeter, right? Myself and my colleagues, if I'm lucky, are able to sit there and kind of keep things cordoned off.

Ben Hughes: Security has, of course, a different set of wants, which are the most important, because if there's one thing security has, it's egos galore. You can advance because I don't have the clicker.

We want things not to change so that we can try and get a decent grasp of what's out there and find as many bugs as possible. Things really not changing ever, because that just brings in unknowns and bad things you want. Except when there's patches which must happen this instant, and you must drop everything you're doing because there's nothing more important than me and my work.

Which is why, I don't know what Ops are doing all the time, but they should be patching my stuff. It's ridiculous. You are ridiculous. Go rob a bank.

Chris Barker: That's great. I'm the one who didn't get to dye his hair.

Adrien Thebo: That's after.

Ben Hughes: I thought you were blonde?

Chris Barker: No. So how do we kind of start resolving these things? Some of it is really just getting us to work together. The fact that there's three of us up here right now talking about the problems that we have, in part with each other but also with our jobs and our priorities, helps us try to get in alignment so we actually know what's going on. Also so we can understand and empathize with each other on here.

The next thing that you can start doing around this is integrating some training, some things across the teams around this. But really, you also have to do things such as address some of the structural issues that you might have at an organization.

So ensuring there's interaction points so that devs can actually talk to security people, and that operations and security people talk to each other more than just over angry emails and ticket wars.

Ben Hughes: What can't JIRA fix?

Chris Barker: Nothing. Actually wait, no. JIRA did give me an email alias really quickly yesterday. That was a human, but it went through JIRA.

Ben Hughes: I'm pleased. And I still maintain that for a lot of my career, this conversation is more than I've had with dev teams and operations teams while being in security. That would have been useful 15 years ago. Well done us for sitting on this for 15 years.

Traditionally operations people like sitting in their cave. Developers like, I don't know, sitting in their Jacuzzi writing mad code. Security people, let's not talk about them, but the generally don't play well with others. I think that needs to change in the way that certainly DevOps has tried to bring operations closer to devs. I don't know if it's brought devs closer to ops.

Adrien Thebo: In a lot of cases you have the security team coming in after the fact, after the code is live, after things are already burning, but these are issues of, "Hey, maybe I need to do some image processing. I need a library." Instead of waiting for the, "Oh, this library is horribly vulnerable, and we've never seen so many rootkits on one server," we talk to security up front and say, "Hey, we would like to do X, Y, or Z.

What's your advice on this? What libraries are safe? What things should we know up front?" Because this guy has a wealth of knowledge. The security teams that have so much information.

Ah, maybe we're just screwed. No, they do have a wealth of knowledge and experience and this is something that, if you tap into earlier, you can do things right from the beginning just by talking to people. It's easy to forget, but it turns out that the simple things —

Ben Hughes: Turn... into magic.

Chris Barker: So we keep talking about structure, and some of this is also going to the cultural points too. Such as, always assume that people aren't trying to screw you over, unless they're Ben.

But really, truly actually trying to build a space, kind of security as a safe space. People can come to you and say, "Hey I clicked on this email, there's a problem with that." Or, "Hey, I want to build this thing out." Or, "Help me figure out how to write better JSON firewall rules so that it doesn't actually suck."

Ben Hughes: I blame Amazon for everything.

Chris Barker: Part of this is also why we have this idea of doing rotations, right? If you can work together and collaborate together, then people actually have a better understanding of how things are going.

Adrien Thebo: For developers, once again, consulting security for best practices, but just going out, getting training. Attending DEFCON for instance, or going to an all-ops training or anything like that. There's a lot of things that you may not think about, like timing attacks. Which, it's like, "Oh, I'll just do a for loop. Oh, I can optimize this, make this faster." That's bad.

So learning what you actually need to address instead of trying to bolt on security afterwards. I think we're talking about tooling later.

Ben Hughes: Also go to BSides Portland, because they're our media sponsor, and they're lovely people.

Chris Barker: So some more specifics around this that we actually would really want to call out is the idea of coordinated planning. At Puppet, one of the things that we've started doing is whenever we build a new product or feature — actually Beth here is the one who introduced the idea of holding — so we can single her out, she's the person who's laughing loudly usually when we're making fun of each other — is the idea of a mini convention. We actually have a three-day internal con that has subject matter experts across the field, including folks in operations, people who are handling some of the security aspects, and the development team, so that everyone actually has a baseline at the start of the project of what, one, who the people involved are.

But also as the project goes on, there's more interaction with that. So all of a sudden if I'm getting a ticket from Adrien about something, I actually remember back two or three months ago, the actual conversation we had. It's this empathy. We had time to sit down and work with it so I know what the hell he needs these library services for in the first place.

Or I know what database is in place. Security knows ahead of time that yes, we're going to be doing testing in AWS, but we're going to be using maybe this pre-built or verified firewall process ahead of time. Or just tell them that it's actually going to be running in AWS in this availability zone, so that's not somebody leaking their GitHub keys.

Adrien Thebo: It's hard to understate the importance of empathy. It's very easy for me to think, here's my responsibilities, I don't have to go any farther. The server is not my problem. Security, I don't care. It is this lack of empathy that, it's like there's a ticket at 3 a.m. Things are on fire and it's like, finger pointing begins and there's blame being thrown around and the issue isn't being addressed.

We're just creating hostile relationships and it becomes harder to work together. But with this mini-con idea, you sit down in the room and you have this shared goal. You are trying to achieve this thing. You can see that everyone there is in good faith. You actually are working on the same project together.

I know we're all paranoid of like, oh we can't have touchy-feely talks. But the basic human interactions do really matter. The difference between the, oh God, we had another vulnerability, and the, oh, we, as a team, may have made a mistake. Let's fix this together. This can mean the difference between hating your coworkers and having hugs onstage and such.

Chris Barker: I don't know if you want to add anything about hate.

Ben Hughes: I have a record collection about hate. Yeah, security has traditionally been a very unfriendly, unwelcoming place. I hope that's slowly changing. Certainly as the industry matures. But it's very much been a, we say no, we block things, we turn things off. Which, even ignoring how much you hug someone, is not always the best way to do business.

And security does lose sight of, these things are put in place to enable business, not — Most companies' job isn't to be the most secure company, it's to actually try and make money. Because yay capitalism. Just turning everything off and covering all your servers in concrete and burying them would be secure, but you would also be quite unemployed quite rapidly, unless you also had a sideline in making concrete. And who doesn't have one of those.

Chris Barker: I guess I need one. Also on this, we've mentioned rotations a lot. There's a presenter, you might have heard of her, her name is Alice Goldfuss. She does an awesome talk called, "Rock Stars, Builders, and Janitors." If you can, catch it. Is it online? Is the video up?

Alice Goldfuss: Yeah, it's on YouTube.

Chris Barker: Sweet. Really watch that as one of the ways to look at — If you need a quick intro to how you can possibly start doing rotations, for people to start interacting with each other. Going really from the standpoint of, what are the things people don't want to do first so that you can actually understand why the work that no one wants to do is the work that everyone should first figure out how to do.

So you build empathy and realize, "Wow, I don't want to make more mistakes for other people to have to clean up."

Ben Hughes: Someone else will do that via libraries or kernel exploits. You'll have to clean those up anyway, so try and add fewer of those.

Chris Barker: Yeah. The other thing on here is what comes up a lot too when we start talking about security, is people start talking about things like doing bug bounties, doing pen testing. If you're at the point where you're still trying to figure out how to have the conversation with a security person, you probably don't want to spend all the money on a bug bounty yet because you're going to fix that first.

If I can actually work with Ben, we'll get rid of all of the low hanging fruit so we don't have to pay out a full salary to a bug bounty team, when we realize just working with Ben can find all those ahead of time.

Ben Hughes: When Etsy first launched their bug bounty program back in, I don't know, four years ago, the first two weeks were just spent triaging bugs and getting mostly the security team working with developers to get them all fixed. That was everyone on the security team just doing bug bounty stuff.

We'd already had a security team for a number of years and had already had some audits at that point anyway. There are bugs out there so don't rush into bug bounties.

Chris Barker: As everyone talks about how DevOpsDays isn't about tooling, but then we have the slide where we talk about how let's talk about some of the tooling that you might want to look at.

Adrien Thebo: One of the things that I've been working with a few other people is embedding security testing right as part of the CI pipeline. So if you make a mistake, if you write code that doesn't do proper SSL verification, or doesn't respect CRLs, your tests pass, your code doesn't get merged.

This is really nice. It means that — Isn't that slide hypnotizing? It's been really rough when we've been preparing for this.

Ben Hughes: Just really hungry now.

Adrien Thebo: Pancakes.

Ben Hughes: They're gluten-free.

Adrien Thebo: Nothing is ever too soon for you Ben.

So yeah, doing this very aggressive like, if your code is not secure, if it has issues, it does not get merged into the build. It means that you can work with at least a little bit more confidence. An example of this is no go to fail, where it does some really rigorous SSL verification.

Or tools, I'm not sure if Gauntlet's been mentioned much. Gauntlet is a kind of Cucumber syntax sort of driven, penetration testing tool. It will hook into Nmap and maybe Metasploit and such like that.

So you can say, okay, we have a live build on this server, let's bring it down, or let's do everything we can. You can hook this up to monitoring, where you have security tests that call Nagios alerts. When you ship code to your staging server, you get an alert saying, "Hey, if this code would go live then you would have new and exciting services running on your server." Having that continuous checking is quite powerful.

Chris Barker: Also on that area is obviously infrastructure-as-code tools. I'm a little biased. Puppet is pretty good at this. But there's other ones here too. Chef is also here. At least it's a configuration management tool and not someone's shell script, which is what I used to have to spend most of my time fixing.

The reason for this is, how I see it is, my infrastructure-as-code, my actual code, my repo that I'm using to build servers, is something that I can work with the security team on.

One, so security can validate my server configurations and I can repeatedly deploy them, but also becomes a contract I can provide to my development team. So they can say, "Look, this is what you're going to get." I can pin specific libraries in place so that OpenSSL is pinned at a specific version. We know that this is it. Or in some cases we can just ensure latest so that it always has the latest version on it, and then reboot everything.

The idea is I can have an easy, programmatic way to instantiate my systems, which makes Ben's life a lot easier because he can inspect that code, possibly raise issues on it, rarely send a pull request, but at least know there's a baseline in which I'm building my systems now.

Ben Hughes: What we've actually done for some of our stuff is when we've found — when there's been vulnerability announcements and CentOS eventually make a patch — or, sorry, recompile a patch from Red Hat — is putting that information in Chef into a fact. No not a fact, I've got that wrong, haven't I?

What does Chef use? Ohai thing? I don't know. My infrastructure brain is gone. And then we have Nagios. I did remember that. Alerting on that so that if any hosts are suddenly vulnerable to this, then someone will get notified. Probably not paged, but emailed. Then as soon as new servers are integrated, or spun up, or whatever you do with servers, it will get these Nagios checks probably.

Then we can always have, is everything patched to at least this level, and that's all done for us automatically.

Chris Barker: That's a similar thing we do on the operations side in other tools. Also with the monitoring aspect,s you can do some very creative things with your config management. Some of the things you can have set up are being able to deploy — That really is still going. I should have had that pause.

If you're using tools like Sensu you can actually broadcast events to the Sensu system before or after your config management tool is running. Just being able to alert to Sensu that, hey I'm going to run Puppet right now, so if a service restarts in this window when Puppet is running, you probably don't need to raise an error.

But also, if Puppet has just finished running and a service stopped, that is a good way of indicating somebody just ate that service and you probably need to alert whoever just triggered the Puppet run, and if you can tie that back to a code change or other system.

Lastly, in terms of tools, this is much of equipping yourself and your team on this. At Puppet we do a DevOps survey. The folks out there can give you a copy of it. It includes a section talking about security and how security, as part of your DevOps process, is a useful way — it's research on this.

If you need to go back to the organization and get buy in to help with the structural changes, these cultural changes, there are numbers and graphs and data to help prove the point instead of just saying, "Hey, I just say these three weirdos standing on a stage for 30 minutes talking about this stuff." There's actually some science to back this up.

Specifically, organizations that have these sorts of policies literally spend half the time they otherwise would on security events and remediation, is the biggest finding we found out of it from the survey this last year.

Ben Hughes: If the media of the last year hasn't convinced you that maybe a security is a good thing, then I hope the survey will, because it's been a pretty rough year for security.

Adrien Thebo: There's the aspect too that having a very secure server does not inherently make you money. Having a vulnerable server will cause you to lose money, maybe, at some point. Being able to say, here are the numbers, here's your risk factor, here's all the time you're losing, and this is how you can once again save money and be more efficient.

You're putting a monetary cost onto something that was otherwise an unquantifiable good thing. We always knew it was a good thing, we just may not have been able to justify it. Saying, "Hey, you want to save a whole lot of money here, or do you want to save face in front of the entire world and be more efficient? Hey, maybe should do these smart things."

Ben Hughes: Proving security is starting to save you money due to the horrible world of cyber-insurance, where if you have more security you will pay less on your cyber-insurance premiums. What a horrible word that is. But it's sadly now a very real thing.

Chris Barker: Otherwise known as a gin budget.

Ben Hughes: It's generally where a ridiculous number of insurance companies say you should allocate this much money to spend on Mandiant when you're broken into. And then you argue with them about why that's a ridiculous number. And then you hire more security people.

Chris Barker: Just to wrap this up — Ben's going to talk about more of the cultural changes later, at his next talk. We managed to wrangle him in for this one as well. So you get twice as much twitchy Ben today.

Ben Hughes: My chronic insomnia is really bad at the moment, so I've not really slept.

Adrien Thebo: I've got a really good view of his bloodshot eyes from here. But it is worth emphasizing, we've already done this with Dev and Ops. We've broken down that silo. But we're not done yet. We're very far from being done.

Once again, start these conversations. Be engaged with these people. Remember that we are all on this common goal together. You can achieve a lot just by having those mini cons, bringing people into the fold sooner, making this a priority. It saves you money and you meet new people and all that great stuff.

Male questioner: It's probably included in the conversation you had earlier about dev and security needing to be closer together, but one of the things that development tends to be unaware of is blast radius. If something goes wrong with this component, what does it mean all the way down the line?

This goes into major vendors. I just finished teaching a class with a major vendor that left open a container port, and suddenly I got a nastygram from a large cloud provider saying, "Why are you DDOS-ing my people?"

Is there a formal system right now for evaluating the effects of bugs in particular areas that you can communicate back, and say this is why security needs to be part of this reference architecture?

Ben Hughes: I'm sure a vendor will try and sell you something to do that. I think it's, as systems get more complicated, it gets harder because it used to be you compromise one system, you have that system and its network link and as much disk as it has.

Now, you compromise a machine, then you find the AWS creds. Now you have as much AWS as you can use until the accounting department starts crying. Not that that would ever happen. So I think it's a really hard thing to quantify. I think sadly some of the best people who will quantify are people trying to sell you insurance, because they have a very strong monetary interest in working out what that exposure is.

I think that still very immature because it's so new and it's not really that explored. Yeah, good luck.

Chris Barker: Some of this might be able to be derived from maybe staging and QA areas. To actually get an idea of what's the scope of this? If you're building and developing a service that's going to only have five systems, but you're going to throw it behind an auto-scale group that's going to go up to 250, it's really starting to keep those things in mind.

That's something I found. It can sometimes be a perspective that I've always had from working in the operations standpoint because my view has always been, I'm going to be responsible for N number of systems. That becomes more complicated as things go on. Versus just having that development environment that you're focused in on.

Just try to get this work on three and then we'll scale it later. I think getting an idea of what is actually building the applications and what's in there. Having at least communication going on so that doing a formal assessment may be difficult, but it definitely is something that would be easier to surface once you're having these conversations.

Ben Hughes: If you have the money and it's on a particular thing, then just contract someone to do a red team engagement on it and they will tell you how big the blast radius is and then everything else they've broken into, and all of your PII that you didn't know was in another system, and go from there. That doesn't scale, but if the specifics — It's terrifying and exciting.

Male questioner: Good morning. I was just curious, in a DevOps world, or model, if any of the responsibilities of security change. The security team is still accountable for overall security, but do any of the responsibilities move to some of the Ops or Developers?

Adrien Thebo: They don't change because they always should have been, everyone is responsible for security. I would say that security is, they are there to help. They are there to advise. They are there to say, "Hey that's a really dumb thing you're doing. Let's not put that into production." And at the end of the day, security is there to say, "Hey, we have found these holes. Let's resolve these problems."

Developers have to be security conscious, 100 percent. We were all just at DEFCON, and oh man, the vulnerabilities we saw were fascinating because of programming flaws. You can't put a timing vulnerability behind a firewall. You can't hide a lot of things. Developers have to be security conscious.

Operations, that's where the rubber hits the road. Unless security starts standing up servers, which might be cool, these guys are responsible for where everything is run. This is effectively the first line of defense. The responsibility was always there, so the answer you want is yes, but we always should have been saying, "security is everybody's responsibility."

Did I get it all?

Ben Hughes: I mean you got most things ever said.

Adrien Thebo: I talk quickly.

Ben Hughes: The things that change for operations and devs in this wonderful DevOps thing also change, in some ways, for security in that you're now not a wall to be thrown over, you actually have to have conversations rather than just mandate things. Which leads to, I don't want to say leads to compromise because there are too many double meanings on that.

But that leads to a shared understanding. There we go. Thank you, thesaurus. Rather than just, it's this, you must do this. You have to communicate more, which security people are not good at, as I'm proving by rambling over the same point.

Male questioner: I'd love to hear your thoughts about, say you have a web app that used that used HTTP Proxy, and there was a big fire drill when everybody had to go turn that off and sanitize. What if that breaks your application in the process? I'd love to hear your thoughts on that.

Adrien Thebo: I think I heard someone in the front row say, "That sucks." If you're saying this specifically with respect to a security fire drill — What was a vulnerable thing that happened, like some bad software being shipped?

Ben Hughes: That's cool. You learned something. You found a new thing that you can make better. Depending on your website, taking it down may or may not be a big deal. You don't work at Delta, do you?

I ask for no reason. I just need some cheap flights and some bitcoin. I don't know. Gamedays and security are definitely a thing. It's a thing we've been pushing in our organization. What happens if we break this in this way? But as an attacker, not as an ops person doing whatever gamedaying Ops people do. Sipping whiskey.

Chris Barker: Tripping over the network cable.

Ben Hughes: Yeah. I don't know. At Etsy we like learning, so we turn it into a learning exercise and then hug for an hour. It sounds like a painful thing but a thing you now know more about.

Chris Barker: In the area of having a reactive to that is usually a triage team or something would come together. That would kind of compromise the representatives of all three areas of interest to work on that to find the resolution going forward.

In some cases that's something that may happen as a symptom of not having that communication happening. It's really hard to be like, "Hey, we found this problem, now let's immediately implement all of the DevOps and security integration to fix it." You can kind of do that where folks talk about, let's do a firefighting drill on that, but then you realize that you're really good at fighting fires and you wait until they happen.

In some ways you can use it to pilot internally. Let's not dissolve the team once we've actually fixed that. Let's have them keep going forward and keep iterating on that as they keep working.

Alice Goldfuss: All right. We are out of time. Thank you so much. Can we have a round of applause?

Learn more

Puppet sites use proprietary and third-party cookies. By using our sites, you agree to our cookie policy.