Rekt Pilled: What to Do When Your dApp Gets Pwned and How to Stay Kalm

Transcript

hey guys can you understand me okay is that is that good um I'm really excited to present this to you today um I've had a lot of conversations over this past week with people about um incident response and there are a lot of different takes on this most people say oh yeah we just need better monitoring we just need better monitoring and then you know I go and talk to various different projects and I ask them what kind of monitoring are you doing today crickets usually right and they're not exactly prepared for when they actually get wrecked right and we're going to be talking about exactly what wrecked means today now um first and foremost I probably should introduce myself my name is Heidi I'm on the unit 0x team at coinbase our Big

Goal here is to make web 3 more safe as coinbase is getting more into the web 3 space we want to make sure that we are creating sort of the most safest apps but also making these space a lot more safe one thing that you might say is well wait a second why does coin days care about web 3 security yeah sure you guys are launching base but why do you care about maybe the Euler hack well first and foremost any ecosystem threat is bad for us right it's it's bad PR for us but there's also Fallout from things like the Euler hack that you might not think about for example all the CBE being drained right there's also other Fallout from these sort of events also happening for example a threat actor might want to cash out with us so there

are a bunch of different facets that we care about and we really are interested in making sure that dapps in the ecosystem do not get wrecked now I've got a couple different things on the agenda today but we're going to keep it really light first we wanted to find as what does it mean to get Rector to get pwned right after that I just want to get a set up with basic 10 questions you as a project should be asking yourself in terms of being prepared for getting wrecked right I'm not saying you need a whole monitoring plan today that's a little over ambitious I think for a lot of projects today so let's keep it easy and then last but not least you might be asking yourself the question of oh my

God my dad just got wrecked what do I do how do I even set up a war room how do I even think about incident response so let's get started what do we mean by getting pwned now I want to make it very clear and I love the presentation right before mine about from immunify because getting pwned or wrecked is not the same thing as an incident an incident is kind of what you know immunophyte was talking about where a white hat or maybe you know someone on the dev team might actually find a buck yeah sure it might be a critical bug but it doesn't lead to funds loss of users right and so let's define getting pwned now getting pwn will depend on your DAP in and of itself and your adap's risk tolerance that might be defined by the

jurisdiction that you're in it might be defined by um uh your lawyers it might be defined by what your total tvl is and also maybe your LPS but generalized Pond basically means the unexpected sudden and significant loss of user funds now we're going to go into all of the specifics that I've underlined here but that's the basics of it and I think we can all agree on this definition because it also includes things like rugs right it includes things like hacks too and incl it's it's very Broad and it's done for that reason right because we need to all come to an agreement is what we what what do we mean here now how do we think about getting pwned now

this is all going to be as I talked about before is based on your Project's risk tolerance so what is an unexpected event for you so that might be the usdc d-pag I don't think anyone in this room saw that coming until you know the banks I don't know on that Friday went under but I don't think anyone in this room expected that what's a sudden loss for you how do you define that what percentage of tvl would you actually classify that as right or would you say a sudden loss could even be something like maybe losing funds from your treasury that's something that you want to think about right so for example a sudden loss might be suddenly one million BNB being withdrawn from the BSC bridge now initially when this BNB Bridge hack went down

I didn't think that that 1 million withdrawal was that bad I was like oh it's probably just some big Trader it was only until like I want to say it was like less than five minutes later there was another withdrawal for one million BNB and I remember thinking uh this probably isn't just a big Trader will now the last thing you also want to think about is what's a significant loss like I didn't think that one million B and B was that significant of a loss at the time but maybe in for example in the case of Euler that 34 million usdc which was that was withdrawn from the pool that was the entire you know uh that was the entire Euler you know pool of usdc they had no more money left that is a

significant loss and so when you're like just as a project start thinking about those three very high level things now many of you probably have been in incidents have been in war rooms before but I want to bring us back to the more basic cyber security stuff and how the industry kind of thinks about incident response and then we're going to try to apply it to web3 and all of the kind of pitfalls that we might see with this kind of flowchart now first in the traditional sense there's some preparation that you do for incident response that could be doing a whole bunch of documentation that could be um uh writing playbooks that could be creating a contact list that could be

even starting to monitor right next you have detection so that is all about your monitoring program setting up alerts Etc followed by patching obviously the bug so you find a bug you want to triage it you patch it followed by obviously the recovery okay now you've patched the bug you want to maybe bring whatever your dat this back you bring it back and last but not least you also have lessons which you then are going to apply to your preparation okay this sounds all great right but how is web 3 a little bit different from this well first off detection we all know right from crypto Twitter that a lot of the detection is done publicly that's not generally a thing that happens in web 2

or in traditional cyber security where you're literally put on notice by the entire community that something is up another thing that's also different is when you're actually thinking about applying a patch let's say for example you're operating a bridge your Bridge goes down right and you want to okay you found the bug you want to bring it back well when you bring that back when you want to bring the the bridge back you want to create sort of recovery right what do you think is going to happen everyone is going to flee from your Bridge you will not have liquidity anymore right and so there's a very strange kind of balance that you kind of need to reach when thinking about how you're actually going to go about doing

a recovery right the other thing too with recovery is it's not just bringing that bridge back online it's also recovering whatever funds you might have lost right and funds recovery is a whole other subset whole other story and finally with lessons it's not just the you know the lessons are learned by the bridge in this case it's in and of itself but it's also by The Wider ecosystem we constantly are taking away lessons from various different exploits as to how we can build better gaps um in the future and I'm sure you guys all are as well and that's something very different from web2 sometimes web 2 companies will publish incidents that occur but a lot of times they'll keep it as quiet as possible right because they

don't want the bad PR but of course because everything's public on chain you kind of can't help it here which is cool now let's go through this exercise real quick with two examples and I actually chose these examples because I thought they'd be a little different but in reality they're not so um and before I get into that one thing that I also want to mention is that today I'm only going to be covering mostly these two sections here that are highlighted in red detection we've seen a ton of different companies do a way amazing job of talking about monitoring and I think they do such a good job there's no point me like rehashing that all over again so let's go into the Nomad Bridge example that happened last August now we don't know what kind of preparation they did for this kind of

event we can assume very little though probably but we don't know what kind of preparation they did now one thing with Nomad that was kind of fun or problematic was they didn't really detect it they didn't detect when suddenly massive withdrawals were happening in one specific block that happened I want to say around 9pm um uh that night and unfortunately crypto Twitter saw this and I don't know if you guys remember in that one block there was one threat actor who was a part of the group of attackers that was really hell-bent on participating in that initial Block in the attack and they made the attack public because initially it wasn't and we couldn't

unhash it but this attacker was just very greedy and wanted to push through their transaction and that of course started the free-for-all where everyone could basically copy pasta the code just update write the um uh the input data to include obviously their destination address and boom they could withdraw a ton of um with uh rap BTC and all sorts of currencies now one thing that I have here is that it took the team three hours to respond right in reality and I updated this slide actually while sitting over there that's not actually the case because I was going through the Nomad tweets um the team didn't really respond when in reality happen is if you look at the bridge contract is the bridge contract

just got drained within three hours right of course the attackers initially went for the highest denominations right but once they had gone for the highest value coins like weth or or um uh rap Bitcoin they then kind of waited around and then they went for like lower value currencies like cards for example so they didn't really patch the problem and then it was took them like almost an entire day later right because remember other people are copy pastaing this hack and I remember that there were several people in various group chats I was in they were like hey shouldn't we be withdrawing funds from this bridge as well because we can just hand back this money if you know if they publish a recovery address right it takes Nomad and day to publish a recovery address and even so it's been

almost a year now since the incident they've only recovered about 17 of all the stolen funds that's a very very small amount and of course Nomad has never come back in fact they've hardly made a statement since then I don't know if anyone from The Nomad team is here but I'd love to talk with you to learn whether or not if you ever are considering coming back but to me uh you guys are just ghosts at this point now okay let's talk about something more recent right I thought okay we'll go for an older case now let's go for a newer one multi-chain that literally happened like what a week and a half ago okay so first and foremost preparation let's think about preparation they've been hacked three times so let's hope they prepared right this is hack number four by the way guys so let's hope they did

some preparation not sure though we've they've never published any of it now of course again how was this hack right we don't actually know if this was a hack we think it might be CCP where it's still kind of unclear based on Twitter um based on what they've written on Twitter up until now but again the attack was detected by the public who identified this oh was Peck Shield PEC Shield at 1206 PM UTC detected us on the sixth this is kind of important because it took them literally almost 12 hours later to then respond right on Twitter to say that funds have been moved to an unknown address that's pretty bad that is really really bad right so it was

detected by the public and then it took almost um I want to say it was like maybe a couple hours later funds were frozen then so we kind of tried to recover those funds and why do I mean we it's because some of the funds were frozen on uscc so that was the one thing that you know um that could be done in some of these instances where uh protocols get wrecked or dabs Get Wrecked is that certain assets are freezable so usdc which is a conglomeration of Center and coinbase or rather Circle in coinbase um uh we're able to freeze those funds and that's something we were able to do in this case but it took them then it looks like an extra three minutes to then finally then make a post saying hey don't use our bridge at all right and by

the way if you try to use our Bridge transactions won't go through that's pretty rough right it's it's it's not a good look right and since then as we know they've then published saying hey we're still not sure of where the funds went to but we think it's the CCP who took them because our CEO is still gone but it's unclear again this is pretty bad response right and this is even I would say that in this case a lot of us knew what was going on we thought that it was rapidly kind of happening rapid rapid response was going on but in reality it wasn't in reality it was pretty disorganized I would say so once you start thinking about things in terms of this kind of flow chart things start to seem a little bit less organized I would

say now okay now we know what a basic incident flow chart looks like let's let's talk about some questions to ask before getting pwned and this is where I would say once I finish talking about the slide just take a picture of it because it'll be useful um but some questions to ask right first off with monitoring do you have basic monitoring basic monitoring does not mean crypto Twitter it does not mean creating a tweet deck of your favorite monitoring tools your favorite Auditors they're great and all don't get me wrong Peck Shield slow Miss we all know who they are who then published like these insane like um tweets like hey you might want to check this this transaction out uh

tweets that is not monitoring sure that's a good interim solution that is not monitoring you want to have those alerts set up you want to actually be looking at those next legal slash comms this is something that I've noticed a lot of teams don't think about unless they get a little bit bigger and they actually have lawyers but many teams when they first start they're literally a bunch of devs so the question that I like and would ask them is do you have a lawyer right because when an incident like this goes down there's a lot of legal liability and we're going to talk about that in uh just a minute but the next thing that you might want to ask yourself is do you have pre-written comms in case of an incident like this happening so let's say you don't have a lawyer right or

let's say you have a lawyer and you're kind of on a retainer with them have them just approve a pre-written thing just in case you ever get wrecked remember in crypto communication is key we know with Nomad we also know with multi-chain that the reason that like that their response was so abysmal is that they weren't public about what was going on but instead kind of Let The Chain speak for itself which is not the right move ever right the next thing is is do you have an insurance plan do you have any insurance plan whatsoever and have you notified any of your users or LPS of that right it's not to say that you need necessarily need an insurance plan but it's something that you might want to

think about now the last question is a little controversial is do you have connections to law enforcement now some of you guys might be like no no no we don't we don't want to deal with law enforcement at all nope nope nope we're out of here we're we're about a free financial system here but the thing is is law enforcement might come in contact to you and you kind of don't want that you kind of want to already have someone on your side from the get-go it's just some or something worth thinking about next is incident planning do you have an incident lead so when you have a war room and it was really cool actually the previous presentation here they were talking about having a war room now what I'm talking about here with incident responses going from war room

all the way forward to trying to recover and actually learn from an incident which is a lot which encompasses a lot more right but when you're in a war room what you need to have is an incident lead that person needs to be ascribed at and I've seen various different War rooms where it's a hot potato and no one wants to take it on just sign someone up immediately and just say hey you're you're down for this you got to take all the notes next what you need to have is you need to actually have a tech lead and what is a tech lead a tech lead is someone who communicates with all the devs to figure out where is the bug right that's followed by a calm legal person having someone in those War rooms is critical they can help very quickly help write the tweets help send out

Communications help communicate also with law enforcement Etc whoever reaches out and another question is before incident planning is in order for those Dev to figure out where the hell the bug is how well have you documented your infrastructure and a lot of you guys might still be sitting there especially if you're in a smaller project is like we don't even have time to document it out like that's a luxury that larger projects have we just do not have that just start thinking about keeping it simple and doing some sort of form of documentation and another question is is do you even have a list of contacts that you can reach out to in case any of any anything goes down right and what I mean what do I mean by contacts I mean law enforcement I mean people like uh incident responders even here in this

room who have been through various different incidents Etc and uh last but not least obviously economic analysis right what projects have invested in you and what are the wider ecosystem threats that might actually happen um uh if you try to turn off your tool or your dap or if you try to turn it back on and one thing that I didn't get to but I really want to even though I have one more minute left is the main point here which is what if my dab actually gets pwned and I just want you guys to First not panic and then take pictures of these slides because I'm going to go through them really quickly um but this is the step-by-step plan you want to prepare you want to set up monitoring and all this other stuff that I talked about then obviously you want a

triage and you want to create that team that I talked about but here is the real step-by-step flow chart so first off let's say you get an alert from crypto Twitter I don't want you to but you probably will right monitoring and and then um let's say you triage it and you know that uh oh we lost money what do you do okay so let's say you don't actually think you lost that much money it's not that big of a deal not an incident close it forget about it this is the one part of the slide that I really want you guys to take a picture of is when you call an incident what do you do so there are three big areas that you want to be scoping out you want to be asking yourself legal comms questions that are critical engineering questions but then

also investigations questions because if you get wrecked you are going to want to try to recoup that money you really are um and now you guys have taken pictures of this I want to show you guys two different kind of um uh offshoots of that let's say you find a patch right you need to then start asking yourself what your liability is for implementing that patch and do we have any insurance and what if we what are the wider ecosystems of us ecosystem threats of us turning this thing back on um and you know uh last but not least investigations wise we're still out all this money how do are we going to pay these people back are we actually even going to track down this money and then because a lot of people like this is

they believe that when attackers try to negotiate that maybe they're going to get money back I would say 99 of the time do not even bother but in case you try to bother I wanted to do this slide for you um here are the questions you should be asking yourself if you want to negotiate the main point is here don't get distracted by negotiation it's not worth it and let's do some takeaways here real quick just prepare prepare prepare that's like the main point here and um obviously conduct exercises and if you guys want to talk to us more about some exercises we've conducted uh let us know but thank you so much thank you Heidi we um we have a bit of time before lunch or lunch is starting

now but if you guys have any questions yep okay I'm gonna start you can go next uh I wanted to ask isn't it a bit unfair to assume that the public response on Twitter is correlated with the time that the team actually began to assess and respond to the incident because assume like if your project and you get an incident that the first thing is not go to Twitter it's like actually try and solve the issue right you used to work for nomad no that is a great question and you know what we should give the project team some credit here's the problem though the blockchain is public all of this stuff is public if you don't make a

statement people are going to make assumptions about you so you may as well just make a statement and it should just be a coverall statement like hey we've noticed something weird we'll get back to you something along the lines of that but something that will protect you maybe a little bit more legally but at least that's the bare minimum because if you don't say anything the assumption is you're not doing anything is that unfair yes but have we all seen that time and time again absolutely yeah the uh great talk by the way um do you have like a example of good insulin response it was kind of difficult so I was thinking initially when I was looking through the flow chart of good incident response and a lot of times the best

incident responses have been from companies like block SEC that actually presented earlier where they try to literally front run an attacker right but I didn't want to include that in this presentation even though that's something that's very unique about this industry is that you can front run an attacker right and that the public is already aware and that there are companies out there doing that but it felt a little strange to include that in there because it should be actually the project team doing that right um so yeah any other questions yep are there monitoring tools that you recommend to help front run the attacker I'm going to stay unbiased here

is that right enough any other questions we've got time don't be shy well Heidi's going to be around for a bit longer so feel free to grab her and ask questions thank you so much epic talk thanks [Applause]