Story time! A Year of SEAL War Games

Transcript

okay okay great working so this session is about the security Alliance War games and War Games is just one initiative of the security Alliance so if we go to the next slide perfect so this is how the security Alliance initiatives fit together so you'll hear from a number of people that are involved um over the coming sessions in this block and uh really they're all complimentary so try and think about um each part of the picture kind of fitting together and so my name is Kelsey nabin I'm kind of I'm a seal but I'm also Laing as a seal in a way so I'm actually an academic by training and I'm doing research on the security Alliance as an example of

security in the digital economy and obviously it's an exceptional example because uh they're First Responders to you know a number of issues as well as building um social social and Technical infrastructure across the ecosystem to secure the future of crypto uh so next slide so this was my first experience of War games uh stepping in uh to this welcome message which happened uh on a scenario which was an exercise with a major dii protocol and now I'd like to hand to Isaac PKA yeah hi everybody um I'm Isaac uh I lead the war games Initiative for seal I'm also the co-founder of a company called Shield 3 which focuses on incident response

training um I started working with Sam a couple of years ago um when the security Alliance was just forming um about how do we solve the problem of teams just completely melting down in panicking in war rooms once an incident finally happens and so what we wanted to what we were trying to solve was just how can we prepare teams for these high pressure scenarios so that when an incident inevitably happens as it does like no matter how big you are or small you are um it's not uh it's not a meltdown scenario and like you uh trust your team you know what to do you know your tools work and so what could we do to prepare those teams and so when we do in these war game scenarios or you could also think of them as just like incident response drills but we like to think of

them as war games um is that we' just help teams test and improve both the social and the technical resilience of the team it's one thing to know that all of your tools are going to work uh Under Pressure but it's also another uh to know that uh you trust your team and you know that everybody's going to be able to uh perform properly and so we're not just trying to incite panic in teams when we do these drills but we want them to practice emergency response and so in these drills there's there's three main phases uh we do just a lot of Open Source intelligence gathering which just basically means we read every public dock we read every contract and we understand how the protocol Works uh we do a tabletop scenario in which we come up with four or five different uh possible things that we think could go

wrong tailored to the protocol and then we ask questions like how did you discover this issue do you have internal monitoring that alerted you to this or did you wait for somebody to just post on Twitter like looks like you know this exchange is getting hacked again and so like is there monitoring that exists how does the is there a Playbook that exists who is responding who's awake how are you communicating with these people the the tabletop exercise helps us understand uh th those components of like what is the structure of the instent response maturity of this protocol and that helps us lead into the third phase which I think is the most fun where we actually do a live attack simulation and so we set up a forked Network we Fork all their infrastructure we replicate all their monitoring and then they get uh and then in some uh

period of some hours on a certain day um there is an incident that takes place and then everybody has to join a war room and respond as if it happens for real um and we learned a lot of very interesting things about how teams respond under pressure and it helps teams just understand like uh how ready they are uh so one thing one reason why I think that these war games are important and uh we could also zoom out and think of this more as like an operational security uh in uh training um a lot of the hacks have been not due to Smart contract bugs which is great and a testament to the tooling that a lot of people here have built it's like we're making fewer and fewer of the same dumb mistakes on the smart contract side but we're making as the protocols get

more uh complex and large and become really critical Global Financial infrastructure the operational side is suffering and so what can we we do to help teams prepare operationally for how to actually run these things long term these are cross functional exercises so we're not just going in and working with the devs um but we're working with their Auditors often one thing that surprised me that I learned in this is that a lot of the times Auditors will have like these really longgo retainer arrangements with these protocols where they're not just auditing your contract but they're there for you to join a war room maybe they're on your guardian multisig um to help support the protocol uh so we're dealing with the devs the Auditors the Guardians who are actually supposed to be there and available to

pause like are they going to show up on the day of when you actually need to pause um or are they like off in the woods and forgot to bring their Ledger and can't pause the protocol um and then as far as like the communications team there's a lot of tension we see in these War rooms between like okay the devs want to say publicly this thing is happening on the protocol right now but the communications people that control the Twitter are like no maybe we should uh rephrase it so you want to have the communications people in these things as well and legal because once you start having these like uh conflicts day of between like devs legal and Communications of what can we actually do what can we actually say you want to have that worked out in advance far before uh an incident takes place um so

we've had the opportunity over the last year year and a half to work with a lot of protocols in the space and we've learned a lot from how uh the best teams operate um but the thing that I wanted to emphasize on this slide is like how can you work with us uh there's a few different tracks one is we just published a open- source toolkit for how to run war games um on the seal GitHub there will be a link later in the slide with like a long video walkthrough that I do saying how do you do this stuff what to look for and so you can look at the open source uh the open source Repository um I'm holding a workshop at Devcon on how to use it um there's the like seal sponsored non-for-profit track and so if you're a big protocol that

wants to get one of these drills um it at no cost to you you can come in and like get on our schedule there's a waiting list but like we can get on the schedule and we can work with you um to do a drill at no cost of the protocol and then there's also like a a new like paid track that we're working on with like external uh companies including my company where if a Project's on an accelerated timeline they can get a drill uh Within uh within like a month or two um so the reason why we do war games again is uh uh that Under Pressure like here's this quote that we found online that I actually found very hard to attribute um that Under Pressure you don't rise to the occasion you sync tier level of training I just think that

that's very applicable um and it was either attributed to an ancient Greek poet or the Navy Seals but we couldn't figure out for sure um each build for this is actually quite custom uh there's a lot of moving pieces that goes into this and so for each drill it changes how we actually set up the live environment because so many Protocols are different but the core components stay the same when we're doing a live drill we of course need a Forks Network so we could use like Anvil for that we can use Tender Leaf for that we need a block Explorer because like we need to make the environment feel like a real scenario when the team is debugging and so they need to have some ability to go into like a tenderly debugger or or like an ether scan looking UI and actually click around and figure out

what's happening we need to have their monitoring replicated so if they're using for toot or if they're using like a custom solution or if they have like things like hyper native or hexag gate setup like all of that stuff should also be connected to this forked environment um so that it feels as realistic as possible uh we create a lot of bots that simulate user Behavior so we'll have like the these networks are like very noisy and so we like we actually have Bots that are scripted to be if we're working with a lending protocol we have Bots that are doing deposits and lending and swaps just so that like the Explorer is filled up with a bunch of junk as it would be in a real scenario if you were like dealing with an incident and working on ether scan uh we have like of

course the exploit Bots where at some point they'll like ramp up an intensity and exploit of the protocol and then often there's a additional like internal uis and things that we need to Fork so there's a big like weak two we build process that goes into each of these drills but we're trying to make it more and more consistent so that we can do these at a at a faster Pace yeah I think it's difficult to emphasize how much background research and work goes into preparing the scenarios and actually building the infrastructure to run these and I don't know if anyone noticed in the previous slide but on the optimism base which was our first um multi-chain war game uh seal 911 was actually called in the response which was really cool and so these are some of the scenarios that uh

have been included for different war games that we've run and one way we think about preparing these is through uh vulnerability uh mapping which is a methodology that I've um written about and um especially in relation to Dows as well so thinking about the soot technical legal and economic uh dimensions of a protocol and protocol ecosystem and like I said as we've expanded the war games into like multi-chain scenarios looking at those cross chain uh dependencies and potential Cascade effects as well and so obviously as a social scientist I'm really interested in the human and organizational dynamics of uh security uh and so uh these are some of the things that have come up uh or the

this in the scenarios and also in the responses or or room for improvement uh in the various things and we do a a massive debrief with every war game we work with the team on that and try to provide them written documentation and help them reflect on how they actually went in that and uh each war game uh has also um sort of builds links into the other seal initiatives so as I mentioned 911 um one really interesting aspect is that mindset of uh security which uh p p Pascal kavio um as Sam mentioned as well said in a quote you know absolute effing paranoia is the mindset that you have to have when it comes to blockchain and then also uh a recent

initiative um by the security Alliance uh from the seal um ISAC so the intelligence um and Analysis platform uh is around uh dprk workers the IT workers and some of the social engineering that's happened um behind the scenes that has that have led to to Major exploits and so um that's another human and organizational Dynamic which we're looking at in the scenarios that we're running and so we aim to help uh Foster a an attitude of continuous Improvement uh each Warg game results in a big debrief um the ethnographic observation which is uh myself is allowing for kind of generalizable insights to be published across um not just academic

papers but um blogs uh book chapters books Etc forthcoming uh as well as practical teams for tools so as Isaac mentioned um the open source War Games toolkit and then we're also running a live event in Bangkok on the 10th which has some really uh exciting uh scenarios that Isaac has prepared and so we did want to save time for questions we've still got about uh 10 minutes if anyone has one on their mind while you think about those I'll just throw to Isaac for um some best practices that are arising from s war games and I don't know if we have a microphone Runner but I okay I will run

if I need all right we do yeah so in before we get to uh the questions just while you have a chance to think about them uh I just wanted to go through like just a few slides of like what have we learned from how the best protocols respond uh act um so that we can uh so that we can just share some best practices one is is like maintaining multiple channels to reach not just guardians but everybody who's kind of critical in incident response something that I always like to ask teams to think about is like what happens if there's an incident and telegram is down the day of the incident like how are you actually reaching these team members do you have backup ways of reaching them um are you sure that all of your pager systems are working uh in cases where pre-signing

critical like pause transactions is appropriate that's another thing that we've seen a lot of teams do um where there's certain like you know one bullet in the chamber paus transaction that sits somewhere accessible by certain people um in case certain things have to happen or more teams I think are also using kind of these sophisticated autop paws like uh service providers which uh which is really great to see as well um one thing that we learned we learned a lot of things from from how the UR team operates they like this was like one of the uh this was like a really cool one for for me to learn how it works um and this is all about like Risk isolation and so a pro a protocol like urine which has like assets deployed into many many many different protocols throughout the EOS system it was to me very interesting

to understand how minim how like little contagion there was from like one strategy to another and so the idea of like each uh uh component of a protocol being like Atomic and isolated so that it can't leak and cause issues with other parts of the protocol and also every time a new strategy was deployed um there were there was like an extensive like Risk diligence process and an emergency procedures card so that when the incident happens it actually says step by step one like this is the right person to call but also if this person's not available you take these steps to rescue funds from where they were deployed so that no one's questioning the day of like how do I actually do it because the emergency procedures are defined before the funds are actually deployed uh another thing

on risk isolation from a which I thought was very interesting was how like different collateral assets are can be isolated with these different like Risk Flags so that it can also minimize like contagion uh between different asset classes um and individual collateral assets being able to be paused or Frozen without affecting The Wider protocol like Risk isolation I think is a key thing that we've learned um instant postmortem is super important this is another like shout out to the urine repos GitHub just because like they have a really great like a postmortem uh process for every time there's been something that happened and how that gets integrated back into the Playbook uh and F uh also monitoring mirroring or mirroring monitoring and so like if you have an extensive monitoring

set up on mainnet you should also be running that monitoring thing on any like test net or any like Forks that you have so that you know uh so that you know that these monitoring tools are working and also perhaps like do things to trigger those monitoring tools because uh the biggest stress of a lot of these like backup tools is like how do I know it's actually going to be there the day of when the thing happens and so if you have like a test net setup or a fork setup run your monitoring against your fork setups as well uh in the same way that you would do it on on Main net um yeah so coming back to the high level these are some of the actionable recommendations uh that have uh Arisen from seal war games and we're going to

pause there and invite any questions from the floor which I believe we have some hey a quick question on what are your findings for best response teams have like from psychological not necessarily technical perspective operational psychological how do you prepare for the incident yeah something that we saw in some of the with some of the larger protocols is that there's a very clear like Incident Commander who just is taking lead of the of the scenario and so there's one team member who is like always sharing during when when there's a war room call like there might be like 30 40 people on the call but there's one person who's like The Incident Commander who's sharing their screen and they're like taking notes live where they're like okay there is a Playbook that we're following we did this step we did this step and then

they're keeping track of like all the different things so if they've directed the communications team to prepare a public statement they're like that's like they're they're the person that's putting that on the Google Doc and saying like hey communication team what's the status of this hey did we check with legal on this hey the root cause developers that are looking or the people that are analyzing exposure like there's one person kind of managing the incident and keeping notes in a in a very public way um versus like kind of teams like kind of fracturing out um that that's something that I thought was very important yeah I think one of the most um stressful to look at the other side of that um scenario as I saw for the response team was calling Guardians on a multi- so they went through the actual process process and called the

real guardians but that was I think five people across time zones so they woke up people in the middle of the night um and yeah that was quite interesting because I think it's a common practice across protocols but for them to actually go through that process of making contact UM was very precarious in terms of getting on to them and it was um quite a time delay as well in terms of um slowing down the overall time to to to pausing the protocol and ultimately stopping the the hemorrhage any other question hey guys um you're amazing uh

question about uh compound I think it is cuz did compound have something happen after they had run a war game and I was just wondering if there was anything on the feedback cycle there because obviously it's somebody that you worked with and then they you would have hoped would be more prepared for what was coming uh we haven't like fully closed the loop on that and I think that maybe we'll just chat with the open zeppin or confound folks to ask like did the training we do actually end up helping you because that would actually be probably a useful data point for us to understand uh so I mean we could ask Michael I guess yes thank you let's I won't give numbers but there was a like uh amount of time for all the emergency signers to

sign and we found very specific specific areas where some signers were not as fast as the others and we were able to correct for that which was very helpful there you go anyone else I don't see any hands raised excellent uh if not then I'll just explain what we have on the slide here pleas uh so on the 10th we have a event that I'm very excited about with uh co-hosted between seal and Tenderly with participation from a number of uh of of of great protocols including the folks from a and Unis swop and and urine Ur audit um so we're going to be doing an event which is like a like live War Games kind of CTF style event where everybody will

be competing against each other to white hat rescue funds uh on a bunch of uh on a bunch of challenges that we deploy um and so this will be a really good chance to show off your white hat skills uh and then so if you can make it to the event fantastic also if not I'm working on kind of like a take-home exercise version of it that we'll publish so that you can also like kind of go in and and do the challenges um very much inspired by the work from the uh red Guild with Dem vulnerable defi um and with the benefit a lot of like retrospective hack proof of Concepts from the immuni blog so very much thank you to those teams but yeah we're doing some very cool uh live white hat exercises on the 10th and is there prize money uh there is some

prize money but uh it's mostly for the glory um there's this is a link to the war games drill template I think will just tweet out all of this stuff after uh so that you guys don't have to see this or like write this stuff down but I recorded like a long video of me just walking through actually in detail how you do um a war game including all of the Bots and stuff that I've uh that I've written for that uh if you are a big protocol that wants to get on the wait list for like a seal sponsored drill you can uh fill out a form on the security Alliance website um if you're a protocol that is on like an accelerated Time Track and really wants to like hey we would love to do some like internet response training um and do a game uh

there's also a for on my website Shield 3 where you can fill out just information about your company and we can uh and fit you in as soon as possible uh if you're also interested on this on the open source contribution side or working with me just feel free to find me uh anytime throughout this week or even after thanks so much [Applause]