Building the War Room Before the War: Proactive Incident Response for DeFi Protocols

Transcript

Hey, hey, hey. Actually excited how how much people actually turned up like thank you. Thank you very much. Um I'm Vladashuk. I'm security auditor at consensus diligence. Basically I do audits. I do different ZK related stuff and also I do different blog posts. I have a telegram channel about web3 security and um today's topic would be incident response. So I have I would say quite unique experience in terms of firstly I was a solidity developer who being um on the side of being exploited and not time not one time but actually

two times um but I wouldn't say this this wasn't my mistake at this point this was a miscon wrong management of the smart contracts but overall it's not the topic uh Today the topic would be how actually it looks from the protocol side of view when you get exploited. Um and that's quite funny I think so because if the protocol is getting exploited usually the first people who noticed it would be the user the end user. Why? Because the end user have tokens inside the protocol. So if the token is going to zero, you probably want to ask the protocol why it's going on, what's going on? And that's kind of

an example what's usually happens. And u the person who will be get a notification in that case would be a basically chat telegram support group not the person who should be uh receiving a notification. The second case would be um it would be just a pack shield alert or somebody who will be kind of saying hey guys you're being exploited you might have a look and also the on this side there would be a marketing person looking at it and um in this both cases not anyone from the protocol team would actually have a look and I think this is mainly the reason

why the protocols are not managing the exploits in a very fast way. So it's takes lots of time for the protocol to get a mitigation and yeah obviously the third point would be a monitoring tool alert. It's not really hard to do it anymore. Um but unfortunately it's not really widespread. Um so as we know let's see the hypotentical scenario the team is already notified right and uh what we can do next um so most probably um there would be a management call um management will

notify like look at it and then they will see that um they will give a task to the development team let's uh figure it out the situation very stressful. They need some extra data to basically understand what's going on and um in the meantime they will jump on call and uh start basically preparing to get mitigation. Third step would be is when um the devs after some time actually understands what's is the reason what is the root cause of the exploit and can have some extra like output to the management team and the fourth step it's usually not the last one but in my case

would be contracts get post but usually the protocol can have different problems like it could be not stoppable there is no like post modification or the storage s already broken. So it's not really always help help the protocol and um as you can see there's lots of steps and the thing is that people usually makes mistakes during having lot of steps uh lots doing lots of steps under under the stress. Uh so you can see that there is a cure code and this is one of the uh basically articles I did about penpai exploit. Um I liked uh in this case study that uh the

it shows a lot why people how people can make a mistake and it would be quite sill one and will cost really a lot. So the exploit started at 6:23 it was just penai uh protocol on ethereum and at 6:45 22 minutes later pandal not penpai but basically different protocols post their contracts because they could uh because uh they were also losing money because of the exploit. Um 51 minutes later after the first exploit attacker uh stole additional $600,000 from the arbitum chain. Uh why? Because Pandal haven't actually thought about this case

and uh penai is still not responding and only 56 56 uh minutes later pendal fully post its arbiter platform and 75 minutes later penpai post its protocol. So as you see from the previous like slide uh there are lots of steps usually I would say that having 20 minutes uh time response is pretty good based on the current I would say standards or um situations but uh still um so people tend to make mistakes and uh in this case uh there would be there was more than half a million dollars lost and um

we need to have some plan before the exploit right uh we need to have some monitoring tool we need to fire alert um alerts or something so uh the topic of this call uh the topic of the speech would be basically to find out um why we need uh to prepare for the war room first one would be uh what do you want to see uh in the war room you want to see right people there there would be people who can interpret pretend data. There shouldn't be some people uh you probably don't want to see their marketing team or like Telegram chat support group. Uh you want to see all the accurate information. This also

especially from my experience is uh not happening all the time. uh if the protocol have um current deployment for example like V2 uh they have for example in the best scenario a list of the all smart contracts for this V2 contract but usually the exploits happens it can affect all the versions so are you sure you know all the deployments on like v1 v2 on like 20 chains probably not so uh you need to have accurate information straightforward v visuals It's more about the simulation like uh pro different tracing tools like tenderly or so on. Um clear communication um you

need to set up um place where you can have a call with your team. So basically it's the best approach to have a call not this telegram group skill meeting liter um there should be a person who would be managing all of the response uh I would say and sense of urgency it's very important to make it as fast as possible uh because if you do it fast you can minimize the risk or basically even stop it um how the war room actually starts uh So runs you should design the invariance first of your protocol right after that you

assemble runbooks for common cases for example if uh my protocol have upgradability pattern I can get compromised from the deployment address right so then I need to prepare for that uh so if you have for example a bridge the likely you will have some like mint problem, burning problem or something. So the idea is to basically find the invariance and um prepare for it for for each case. Um document the list who own the roles. Document what addresses you actually using in the protocol. Uh what current deployments uh decides uh who will be um on which side.

So who would be on the war room? Uh set up monitor info on chain metrics and of chain surfaces. Um also not really hard one. You can just uh have a check on the back end who will be monitoring the balance difference with every transaction your protocol with every block. Um and incident flow is quite straightforward. You classify incident and document what you know. Pull the right people into the forum and assign roles. uh use monitoring transactions um traces to gather data.

Then pick action from the pre-approved list to make it as fast as possible. Tell users and other protocols what's happening because usually as for example from the previous case not only you getting wrecked most probably there would be another protocol who who is also suffering and uh keep updating log because in the varants there would be a finished postmortm for you um how can you prepare I think that currently there was there was a lot of different cases which I really liked uh post functionality is basically the most basic one. Um but um emergency implementation is something that I

really like. For example, you have upgradeable protocol and uh if you see that something is going on and you don't want to have your story changed anymore, you just basically upgrade your protocol to the dummy implementation with uh no bite code and every transaction which will be going on after that there would be no difference. Yes, of course. So if you have d if you have uh if you don't have um if it's not possible for you to deploy it like that it will not work but in some cases that's great uh contracts list u that's what I already told you need to uh basically have uh to remember who is actually integrating you who is working with you and volume limits also

quite popular one for example in the uh Ronian bridge incident there was an exploit of around 4,000 ether $12 million and uh it was uh stopped by just a cap on the smart contracts. There was roughly 72 million dollar state on the contracts. U root cause I would say was quite not like really complex but uh also the very basic step of having a cap prevented losing about 80 $90 million. So if it would be $90 million I would say probably Ronin would never survive after this exploit.

Um but now it's just a bad day for your protocol assigning roles in your own war room. So this is overall how the war room structure should look like. Um you need uh several people who will be in charge. You need operations who will be initiating the war room who will be like monitoring the monitoring the chain who will be basically saying hey guys we need basically to be on the war room right now because something bad is going on.

You need a person who will be um keeping the timeline uh and um uh decision logs. Uh you need a strategy lead uh who will be deciding what action would you do next. You need a protocol lead who will know all the protocol who will basically be able to understand why the protocol is hacked. um web infrastructure lead. Uh it's in web 3 not really common but uh also sometimes the exploit could be mitigated from the front end. For example, you can stop the users from continuing to interact with the protocol

uh by uh blocking the web and external communicator. So I would say a person who will be able to reach all of the people in different Telegram groups uh your integrators and so on. Um also important to note that u maybe when the exploits happens all the time you see some very interesting messages on the other address scan about uh I want a bug bouncy I want I'm like a white hat or like if it's kyber swap exploiter you will you will think about um they will tell you a lot but in the most

cases I would say negotiation is not your tool in this case and every negotiation that you will start will basically spend your time first and in the protocols overall in the security uh I think that um sorry lost a little bit um so about negotiation I would say um if you want to actually negotiate in 99 99% he will not gain something good from it and if you actually will decide to do that probably you have to involve legals uh before you do that um

FAC security layers to the protocol there would be probably first of them that you're not vip coding the smart contract you're just developing smart contract then you have uh internal review security preview you have a bug bounty setup and uh then if there is all of these steps of security layers are not protecting you from the exploit there is a very last one seal safe habber uh this is not very popular for some reason but it's quite new also uh you can have a legal framework for white hat which is a smart contract predefined set of rules you can sign it on the smart contract level so if the exploit at some reason will actually decide

whether they want to be to proceed with the ex um sec uh returning funds I I guess this is the best case for the protocol to try seal safe harbor and uh final summary I think it's almost important uh almost impossible to basically have zero incident incidents in your protocol but at the same time you can easily minimize it. First of all, you can always uh contact us and we can help you with doing the security audit and uh because of that most likely you will have not a crazy exploit. Uh

and hopefully no exploit at all. Um I think that um a must have is to have a monitoring tool with all the modern tools. It's uh not hard to do that. Uh then you have to implement a checklist and threat model. Just sit down and think about what attack vectors your protocol could have and uh based on these attack vectors think about how can I protect it? What would be my action in this case? And uh third basically assign the roles. If you would have an exploit, you don't want people just screaming on each other. you want people

uh that everything should be planned. That's that's how I see it. Um so by doing the steps you will have not 50 60 80 minute response. In the best case you will have a response time of likely hopefully like 15 minutes which will save you lots of money. Um so that's all. Thank you very much. >> [applause] >> Thank you very much, Vlad.

Um, as a follow-up question to to the presentation you just had, we have obviously developers, we have product managers. Would you recommend doing uh dry runs on test nets for the war room incidents? And if so, how often? >> Okay, that's an interesting question. Do you need to have a driver? Um, yeah, I think it's a good idea to try them, but at the same time um, honestly, just from my experience, I haven't heard that protocol doing it a lot. So that's uh something that it was just interesting to me. And uh I think

the reason for that is just obviously that uh the protocol I think that I have described all the cases how you can protect from it. Uh I don't really understand what is would be the dry run. So for example you can connect seal in advance. Yes I guess so. But at the same time you can just prepare it uh on the chain. So I don't think it's really important to have uh basically to protect yourself on the test net or something. Just having a plan most obviously would be the best case. If it's uh not uh a plan but an

execution of scripts um it's very hard to to have uh to know what would be the exact exploit. So I not think that in most cases it would be relevant for the protocol. >> Okay. Thank you very much. A wonderful presentation. A round of applause for Vlad. Thank you.