Real-Time War Rooms: Building Proactive DeFi Security Operations Centers

Transcript

Testing. Testing. Amazing. Hey everyone. Good morning. Excited to start day two all together. Talking about a very exciting topic, real-time war rooms. Who doesn't love a 9:30 a.m. war room uh on a Friday, right? Uh thank you everybody for joining the session. Uh really excited to talk about something that is very close to my heart, uh close to my schedule. My friends all know this is where they can find me most evenings and weekends. So, uh very curious. I want to make this as collaborative as tactical as possible. So, going to be doing a lightning A to Z on everything war rooms.

Little bit about myself. My name is Sam. I'm the founder and CEO at Guardrail. Guardrail.ai is our website. Must plug for the for the whole talk. But, if you're curious, do check it out. Guardrail.ai is the is what we're working on. And like I was mentioning, going to be doing a little bit of a quick walk through of what is the role of defense when we're talking about incident response. And then also, what are the outcomes that it creates uh for the people that are in the war room, right? And I think it's important to recognize the kinds of roles that our people are playing in the war room. Followed by which, we'll talk about what good detection looks like. And then also, what good action looks like based on detection.

Pretty simple. Hopefully, we'll have some time for questions. Awesome. So, let's jump right in. >> [clears throat] >> I think when we're talking about real-time war rooms, it's usually a security lead or a protocol lead who's the first person in that war room. And I want to take a quick second to talk about what the day-to-day for that person looks like. Since I think it's pretty talked about, there's the security audits, there's the pen testing, even some, let's say, SOC 2 or ISO. But, I want to take a quick minute just to list out what a security engineer, security POC is doing day-to-day. Just to talk about where the war room would fit in into those responsibilities. So, we've got security design and code review.

As you all know, a very important piece uh when you're talking about SDLC. We've got proactive threat hunting, also a very important piece of being proactive in terms of determining the threats for products that your team is building. Um security testing. Uh a lot of developers will do all kinds of different testing, but they're not always thinking about the security design, security threat model. Um then there's monitoring and tuning for live products, products that are going live. Um again, threat modeling, and the list goes on as you can see. Um I bring this up because not meant to be exhaustive, this is just a way to understand what the person is doing day-to-day.

And if you look at that list and and within our industry, we found by working with so many different protocols that there ends up being two main hats that are worn by the security lead in the war room, right? It's we one out of two. It's usually either a protocol lead, so this is somebody who's worked on the smart contract, the design. They're not necessarily a security expert, right? They're the ones who are able to find where something is when something goes wrong, but they're not proactively looking at the IAM configuration or looking at, you know, EDR logs or anything like that. But this ends up being a very de facto lead when it comes to security in in our industry. Then there's also the security practitioner.

This is somebody who takes security as like the core principle of what they do, but sometimes may not have necessarily all the on-chain context. And I think this is important to call out, kind of identifying which version you are and also identifying what balance you need in that incident response war room. And the reason I bring this up is in our industry, this is sort of what we're up against, right? It doesn't matter if you're the security practitioner who's on the Zoom call or the protocol lead, um we're actually more on the left-hand side here, right? And then we're actually managing security in a remote environment with teammates all over the world, managing millions, if not billions of dollars.

There are much well-funded defenders with entire teams with all kinds of different uh resources behind them. And meanwhile, we have our adversaries that are just waiting for either the left-hand side or the middle-hand side to make any kind of mistake, and they are very much really, really well outnumbered. And that's kind of why we're today in the position where, you know, their choices are left-hand side or middle-hand side. And then the outcomes that this creates is where things get really interesting, right? Uh So, in 2023, when when we were first starting Guardrail, uh we spoke to 20 different protocol teams, and we asked them, "If we're on this call right now, and there was to be a hack right now, um would any early detection system catch it?"

Two of them said, "Yes." 18 said, "What's on the slide right there." Uh that they'll find out on Twitter. Right? Um And And this is because they said, "We've got the Peck Shield alerts on. We've got uh you know, incident responders on standby. They'll tell us." And then at that point, we're already in in the reactive state rather than proactive. And I mean, I do really sympathize with the challenge, right? Between the scale, speed, and complexity of what we're building here in this industry, uh it can be really difficult, and that's why most teams today do rely on vendors, rely on 80/20 solutions, and I think that's really the only way that you can scale a security function.

That being said, the most successful teams make sure that there is always a security owner on the team. And that's because in general, security can never come for free, is what you notice. It always comes with either a time element or a cost element. And you do need somebody on the team to be a consistent voice of that. And I I I can't stress this enough. Um There's teams you work with where everyone's responsible for security, and as we've heard, that means no one's responsible for security. And you the the third thing cannot be stressed enough that somebody on your team needs to be the one who says, "You know, we need to pay for that tool or we need to spend the time and slow down and add the security in.

And not everybody should be thinking about it, right? Everyone's got different specializations, but if you don't have that person who is your security champion on the team, it's going to be a really painful situation when something were to happen. And then, I think about this quote from Sam Sun all the time. Safe components can come together to make something unsafe. That's a good reminder that even the best security practitioners who do their job perfectly, you know, threat model, prepare their team, have the incident response runbooks. We're going to see a couple of examples of this in a second. You can do everything perfectly, but you have to be prepared for things that could go wrong.

That's part of the job. So, let's talk about what good detection looks like. Again, not meant to be exhaustive. I thought we'll get a little bit tactical here, so just four quick pillars of what that looks like. First of all, simulation. This term means so many different things in our industry. We've got testnet simulation, we've got mainnet simulation. I want to talk about security simulation. And security simulation is saying, this action that's about to be posted to the blockchain, what is the effect of that action? And we want to explore that from a security lens. I say that because security means who's the counterparty, what are the assets, how much value is being exchanged. And a very easy way to do this these days is using some kind of pre-transaction simulation product,

service, or even something open source like what Seal 911 has has put outside. What you're seeing here is two screenshots from Guardrail as an example of what we've built. This is an example of when you're about to transact something like say what happened in the Bybit hack. In this case, you're actually getting some results directly sent to your Slack that are giving you passive information on what's about to be signed. On the left-hand side, this is a hack transaction that your team, maybe somebody on the operations team, somebody on the finance team might actually accidentally be signing. You can see here we're calling out a lot of signals, you know, high gas usage, it's a flash loan, things like that. So, we're able to simulate transaction and push out some key contacts and I think at this point, this is pretty bread and

butter for what teams should be doing. Uh and as anybody who's been on the other end of this, if you can prevent anything like this before assets are moved and and funds are transferred, it's night and day difference. So, wherever you can, bring in the prevention, bring in some kind of simulation across the whole stack. And again, there are many platforms out there. You should not be relying on one, just manual, open source, and also pre-packaged platforms. So, definitely worth adding to your detection stack. Um the second is multi-threshold alerting. Um and I know the the term alerting, nobody loves an alert, but think about the context of something were to go wrong, and every time we see uh some kind of postmortem, the the very first timestamp, uh every single person

remembers that because that's the first indicator of something were to go wrong. Um and you don't always need like a a critical incident, but what's really helpful we have seen is when something like the Stream Finance hack is happening or or in this case, the Big One Exchange, um there's a couple of medium alerts that can give you an early indicator that a critical is about to come up. And that crucial time, and and getting the first indication something's about to go wrong, is actually really really helpful from what we've seen in practice with teams. So, this is just one example uh in the Big One hack that happened, I want to say July this year, um there was more than 15 different assets stolen over multiple chains. Um they actually had cascading alerting policies. Um so, they were able to get a early heads-up, but

here's some examples of, you know, not just looking at uh volume transfer, but looking at behavior changes, and being able to set up, you know, different thresholds for alerting can actually be again a really big difference when it comes to detection for your team. Now, the next thing, uh I call this correlation metadata investigation. You can think of it as as right? So, let's say an alert comes in and you have like the most perfect alerts that you can possibly set up with different thresholds and you've done the simulation. I would say this is where when something happens, it's helpful to know who are the other dependencies upstream or downstream that your protocol depends on.

And this is really really important. As we've seen, you know, Penpie was hacked, but it was on top of Pendle, which was completely secure. We saw Griffin AI was hacked, which was on top of Layer Zero, which was completely secure. Then we saw this La Booboo fake ERC-20 hack, which happened on top of Alex. All really sizable TVL lost in these hacks, but all were underlying protocol was completely secure, but somebody used a rogue integration or a rogue entry point into a contract. Think about if you were in this situation, being able to understand what those different entry points are, having that context where you need to go to be able to investigate, upstream dependencies, downstream dependencies.

It's crucial. And for us, we see this every single week where we're able to find teams with different kinds of Oracle issues, different kinds of dependency issues, where they're not getting hacked every week, but they are saving hundreds of thousands of dollars every week with the kinds of insights that we're able to find both with upstream issues and downstream dependencies that could be the trigger of for some kind of hack. So. That's the third piece. And then lastly, I say coordination. And we're going to get into action in a second here, but I would say when these incidents are happening, there's a very natural tendency where the whole team's, you know, ears are perked up and every minute matters.

And again, being the security person in the war room here, it's helpful to be prepared with the the who, what, why of what's happening, right? Every single person's going to want to know who's the counterparty, what's the risk, how much, which assets are affected, and then even having a clear chain of command. I would say this takes about an hour and a half to two hours one time and then maybe like about an hour every month just to keep things refreshed, but this can save you a world of pain if you do find yourself in some kind of live war room incident. Um I think I've seen teams you know, hope they're not in this position but have to decide when to send an on-chain message, when to engage a third-party incident responder, you

know, which agreements to sign whether it's information disclosure. Um and again, a lot of these things can be determined proactively ahead of time. Hopefully never to be used and and you know, refreshed passively. However, if an incident were to happen, uh you're spending 30 minutes going back and forth with legal and information information sharing agreements which again, could all be avoided if you do end up in a in a tricky situation like we're describing here. Yeah. And then and then lastly, I think in our space everything's really really decentralized which is amazing, but think about an incident discovery. So, routinely Guardrail ends up being a point of contact for teams that are finding issues, vulnerabilities that they want to get exposed to a protocol.

So, if you're a protocol, think about how when those motivated folks are finding out issues in your protocol, is there a good avenue for them to reach you, right? Whether it's through Seal, that's great. Whether it's through your security docs or your Twitter, but do think about how as you're growing and as people are trying to use your protocol and and growing TVL, is there a good way that they can they can reach out to you? Awesome. So, let's just say you've got those four pillars of detection in place, right? We have simulation, alerting, correlation and and coordination. This gives you a lot of preparation. Um when we're actually talking about being in this war room incident, right? When we're actually being able to respond on time, here's some of the things that

we've seen that's been helpful. Um first of all, being able to coordinate off-chain actions and letting the team know what to expect. So, um you know, when when a crisis situation happens, we want to avoid the thing where everyone's saying, "What do we do? What do we do?" When they see the right Slack message come in, they understand, "Okay, there's a security incident that's going on. XYZ is the is the incident lead. You know, person B is the one who's doing the investigation." So, I think setting the expectations across the team of if something were to go wrong, here's what to expect. Um, having a lots of flexibility in there for when things do go wrong, and I think we're going to hear from some of the speakers on this after me.

And then, I would say again, using simulation again where you do have it available to help you investigate really what's happening in those transactions. Um, being able to assign roles, uh, like we were just mentioning here. And then, also having some pre-drafted messages that either say, "We know something's happening. We're working on it." or "Something has happened and our team is investigating it with XYZ." Um, but again, in in the when the situation is happening, you really don't want to be able to, you know, necessarily use AI tools to draft a message like this, uh, because it will be the the legacy. It'll be the in in the postmortem after the fact. So, it's really important, I would say, to think about it with a clear mind ahead of time on the off-chain side.

And then, on the on-chain side, um, go through each contract you have, each asset, and each user you have at the company, and understand what actions you can take place. Um, you know, we do see teams regularly that think that they have some kind of pausing functionality, and then they realize certain contracts have it and some don't. Very difficult realization when something is already happening to to figure out what you can do in that situation. So, I would say, ahead of time, know which tokens you can revoke, which tokens you won't revoke, who's going to be doing that revocation action on chain. Um, sometimes this has to go through a time lock, sometimes this has to go through a governance, what not. So, um, be aware of what you can revoke, what you can pause, um, which protocol parameters are available to change, um, which ones are absolutely, you know, no

man's territory, no one can change them. Um, and again, be prepared for those on-chain actions. Um and then lastly, um this is a big one which I think wasn't as big of a factor 2 years ago, but as of today, virtually every team we work with is multi-chain. We're talking they have assets on different chains, they have contracts on different chains, um and cannot stress this enough, different chains might have different security, availability, and options. Um some teams are some chains are extremely accessible, some chains are are a lot harder to work with in the case of an incident. Um so, just understand which environment you're operating in, right? And and that can affect what kind of controls that you can implement, um and again, will will could make the difference between, you know, a couple of million versus hundreds of millions

in at stake. Um so, just to wrap it up here, so you know, I'm not not talking at nauseam, and hopefully you guys got some tactical tips out of this on, you know, if you do end up in a war room, who are you going to be in the war room with, and what tactics to employ. Um here's a like a just putting it all together, a a 10 checklist example of you want to click a picture, you want to reference this later, uh feel free uh to look at this this, you know, 10 check mark uh checklist uh to say. And with that, I will um once everyone's done taking a picture, wrap it up. Thank you guys so much for um an engaging first session of the day.

>> [applause] [music]