You Deployed Your Project, Now What?

Transcript

[Music] All right. Well, hello everyone. My name is Emra Sabos. Um, I'm a blockchain security engineer at Coinbase and my coworker Tom Ryan was in the security operations team. And this is you deploy your project. Now, what? So, let's do a quick real check before we start. Before you deploy your project, you did multiple audits. You made sure that every single findings from those audits are actually addressed. You maxed out your code. You even went step ahead and set up some bug bounty program with the hope of incentivizing researchers to submit the vulnerabilities rather than acting on

them. And you deploy a project thinking that you're done. But we don't think so. Deployment is not equal to mission complete. This is why today I'm going to talk about insights we build at Coinbase over years to secure, strengthen, and effectively manage your project after deployment. But before that, let's step back to pre-eployment phase, which is a very important phase that we need we need to make sure that you do this. So insight number one says get monitoring requirements from your thread model/ audit. So when you do thread modeling and if you don't do you absolutely should you will identify some invariance/ requirements that you want

to make sure your project holds true at all the time and if you don't do thread modeling you can ask your auditor to provide these. So you need these requirements and invariance to set up monitoring and you want to make sure you are actually getting notified when there's a change on your invariance. In fact, one of the biggest hack in DeFi, which resulted in $600 million, didn't have any notification or alerts, which is why they didn't realize they got hacked up until a week later when someone tried to withdraw their money. And before you implement anything, there are three questions that you have to answer. What are you monitoring? Why are you monitoring? And how are you monitoring? Let's answer these questions using a case study from base fault

proofs. So optimistic rollups rely on the assumption that transaction are valid which is why their optimistic nature is coming from and fault proofs is this cryptography primitive that enables anyone to challenge permissionlessly the state transition and dispute games are these games that participant plays if they think something is wrong in the dispute process and they just find the where the dispute where the fault transition has happened using the dispute games and if you think fault proofs as a big component we can think fault proof system as a core and we can think dispute games as another component

and to answer the what we are monitoring question for each of these components I'm going to start with the full proof system we are monitoring for ownership changes and privilege actions these are the contracts that we make sure there is no ownership change happens that we are not aware of and we are also monitoring ing the functions with an access control that are that performs privileged actions and on the dispute games we are looking for more nuanced invariance such as early withdrawal and challenge proposal. So early withdrawal when the participants of dispute game plays the game they deposit some ETH amount and we want to make sure that those ETHs are not withdrawn before the predefined time is set time is met. um if they are

something is wrong and we have to act on them and we also want to get notified when there is a proposal on there's a challenge on our proposals and we want to act on them. So that is the what question and on the why are we monitoring side we want to make sure that our system is integrity is protected. So when there's a change in the poll pro system that our integrity will be at risk and we want to act on them and why are we monitoring the dispute games is because we want to ensure the ST integrity every time there's a new proposal the state integrity they can part they can participate in this game and challenge it and we want to make sure that we want to get notified when that happens so we

can respond to them and before I talk about how are we monitoring them this is a slide where you scan the QR code which will direct you to a GitHub repo that's public that will talk about that talks about how are we monitoring stuff and that details implementations of them and we also have a cool blog that we detail all this and at the bottom you can see that we have a dispute game and for each dispute game that is created we have different monitoring solution and now the how are we monitoring let's answer the last question. So on the left I mean we are using hexagate as a partner to monitor all the contracts and

on the left we are looking for some events. When these events are emitted that means we will get alerted and on the right you can see the UI that we can write more custom invariants and these are we are using the domain specific language of hexag and this is more custom invariants and more nuance invariants such as early it visual and challenger proposed insight number two says test your monitors on test net. So before you actually deploy a contract and start running your monitors on mainet, you want to make sure they are project owner to deploy a contract on test net and run this test to make sure their monitors are working. That's the only way to make sure that they are live and they are

working and are your monitors finding alerts as expected. So this is where you're going to find all those edge cases and you're going to make sure that you actually fix them before you deploy everything to mainet. And pro tip here is that use thread model scenarios identified within the audited models. So this is more realistic testing scenarios. So you don't just come up with some nonsense testing scenarios. You actually use realistic scenarios. You deploy a contract. Congrats. Now let's look into the post deployment base where we start with the runbooks. So runbooks is a part of documentation. No one likes to write but we assure you that you want to write one because so runbooks essentially a

step-by-step procedure that one follows on a certain action and action here is your alert and inside number two it says that document runbooks for each monitor alert. So if you don't have a runbook for each alert, you are not going to know how to react on them and you can just assume that every single person that knows how to react on those alerts will be will be in the in the chat that receives all these alerts. So some questions to answer here is that do you know what to do when alert is triggered and do you know who to escalate it? So these you should clearly identify there so that the person who thinks the alert is true positive knows who to escalate this. And for the last questions here do you know how to mitigate true positive

alert? So what are you going to do? How are you going to react? Who are you going to contact? These are some questions you should definitely answer and write detail in runbooks. And pro tip here is that make sure to include runbook for each alert in your notification. So when you receive alert on your Slack or whatever platform that you use, you want to have a link to a runbook. Otherwise, in that chaos, you're going to go look for this documentation and try to find it, which is way more stressful environment, right?

Inside number four, establish schedule if you haven't done so already. So you want to make sure that your alerts are hitting the correct channel and in that channel you have a correct people who is going to be on call to respond to it. At Coinbase we use Slack plus page of DD for this and depending on the severity of the monitors our Slack notification pages the on call engineer directly and the goal here is that severity levels depends on the importance. We can just bring the people immediately.

And do uncle personnel know where to find runbooks? Well, if you link your runbook to your alert, you will have answer this question and you will be able to identify the runbook and what to do immediately. And insight number five says plan tabletop with key stakeholders. So tabletop is exercise. It's a guided exercise where the teams gather together and to work on a scenario and the scenario details are here it's not very detailed. So you have a scenario and the teams that you have they are trying to speculate on what will you do and how you will react. And tip here is that make sure to include all stakeholders in

your tabletop. Your engineers, your security folks, your coms, your legal if you have any. So the goal here is that every single team that will be alerted when you have actually an exploit and how are they are they going to communicate. Some tips here is that do not reveal the exact instance scenarios in advance. So you don't want people to know how they going to respond when they come to the guided discussion. You want those people to come without knowing the details so that you can actually identify gaps in your instant response and you can improve on it. The goal here is to not shame anyone but actually prepare and even better you can just use um independent engineer/party

to conduct the tabletop and for this we worked with seal last year which my coworker Tom going to go the details of case study. Sweet. Thank you everyone. So I'm Tom Ryan working in the security operations team. Um today I'm going to go over uh a SEAL wargaming simulation uh that we did with the SEAL team a couple years ago a little it was prior to fallroofs but we have some blogs about it so we wanted to share some of the insights out of it um so those who don't know SEAL suggest you you get to know them they're the SEAL alliance they run tabletops they also work uh you know across the industry to help resolve incidents as they occur um so the scenario that we executed with the SEAL team was a faulty

upgrade that introduced a vulnerability allowing withdrawals on layer 1 without a corresponding backing withdrawal on layer two. So, we can split the tabletop into three phases. There's the setup of it. There's actually getting the alert, and there's actually conducting the war room and starting the incident. In the setup phase, there's two pieces to it. There's actually getting setting up the scenario, getting the environment ready, getting the test chain up, and there's also applying your monitors to that test chain to see whether or not your monitors are actually going to be effective, where they're not going to work. So, as previously mentioned, what we want when we're setting up these these tabletops, we don't want to exactly know all the details of what's going to occur in the incident. The objective isn't about figuring out what instance or what exploits going to

happen. It's about stress testing your people, your processes, your runbooks to make sure that you're prepared when something does or if something actually goes wrong. So, uh, for this war game, we had the SEAL team determine the scenario. They helped us set up a conduit test chain. They were simulating block production. They also had some additional transactions occurring within it. So it wasn't just we were seeing like only the exploit transactions go through. Um and they also set up a block explorer for us. So when the event actually did happen, we could dig into it more and make it more like a real life scenario.

Once the test chain was live, we then had to go in and replicate our monitors. This is kind of an extra step, but it is necessary if you do want to stress test your monitors and see whether or not you're getting alerted. It's better to miss an alert during a tabletop exercise than miss an alert during real a real life scenario. Otherwise, you're not going to feel as prepared and be much more stressful that way. Um, okay. So, at this point, once everything's set up, you're at the stage where you're now waiting for the alert. So, we set up a date and a time range to execute this. We gave a bit of a heads up to the teams that would be involved to make sure that they knew that they were aware that we were conducting um a simulation because you don't want to kick off an incident,

have everyone get really scared. Oh my god, what's going on? So we wanted to make sure that they had a bit of a heads up. They had some time allocated and we had some engineering resources to jump on the instant call to mock uh the simulation. So fortunately in this war game our monitors did work and we got an alert uh indicating that there was a successful withdrawal on L1 without a backing withdrawal on L2. So when that happened now we need to kick off the incident. This is where you have to get everyone in the room and actually start conducting how do we mitigate this? What's going on and start identifying the root causes. I'm going to pause here and say that if you don't have the capacity to set up a test chain and set up all your monitors, you can kick off a

tabletop simply starting with the alert and say, "Hey guys, we just got this alert. This just happened or Twitter notified of this us of this. What do we do about it?" So, if you don't have the capacity, any sort of tabletop is better than no tabletop. So, if you don't have all the capacity set up, the test chain, the monitors, you can pick it up from the alert. So, once you have the alert, the next step is getting everyone in the room. In our case, we had to get the base team, their engineers. We had to get optimism. We also had to reach out to SEAL. We had to figure out what comms teams need to be involved, what legal people we need for any sort of decision-m that we have to have. Uh, and one of the big challenges that you'll face is figuring out whether people are

available and how to get in touch with them. So, this is where it goes back to our runbooks. The runbooks are really important because when these things happen, there's nothing worse than being on an incident and not having the people available or even knowing how to get in touch with them. If you're sitting around waiting for XYZ to join and they're, you know, in a different time zone, fast asleep on a Sunday night, that's never ideal. So, you want to make sure you have everyone's page duties, not only for the people on call, but the people you also want to get in touch with. Um, all right. So, this leads us to our next piece. So, when you actually connect the war room, everyone's going to be in the room. It's going to get a little stressful. You're going to have to figure out what's going on. But the

point of the tabletop is to ensue the chaos now so you feel prepared when the actual event occurs, feeling like you've already done this before. So at the result of the tabletop, the key is then to refine your runbooks. What you're going to find is that your runbooks suck. The runbooks are kind of are always the last piece. You actually do your engineering work. You set up the monitors. You're like, "Ah, I got to do the runbooks." You have one person go in do it quickly. You'll find it's never enough. If that one person who wrote the runbook leaves and then someone else comes joins in and actually works to handle the incident, if they don't know what to do and the runbooks don't have enough information, you're going to be sitting on your thumbs trying to create a runbook from scratch during incident and that's never going to be any fun. So

you want to make sure that from the result of your tabletops, you're going back and refining your runbooks. The tip of it too is that runbooks are not one and done. You're not just going to write it once. You don't want to go onto a Confluence page and see at the top that it was edited two years ago. You want to make sure that's up to date and you have all the relevant pager duties, you know who to contact so that if something happens, you're prepared. All right. Finally, we'll stop we'll do quick rapid fire insights. Um, so the first one we want to talk on is inventory your smart contracts and other vendor dependencies. So sometimes you think, oh, I just deployed a contract, you know, I can always see it on chain.

I know where it is. But if an engineer leaves or something happens and you don't know what your threat landscape is, you don't know what your onchain assets are, you're going to be in trouble. So inventory your assets. Say say you are using some vendors or an Oracle, anything like that, you want to know which ones you're using. Say something on Twitter where a vendor gets compromised. Are you using it? How are you going to know? Maybe the engineer knows but the rest of the org doesn't know. And if that engineer is not available to tell you we if we care about or not, it's gonna be really difficult and really stressful in the scenario to actually act upon it and pause the contract. So very much recommend inventory your smart contract and your other dependencies that you have. The next one is key livveness

checks. So a lot of times we talk about key security but not always key availability. So something were to go wrong, you need to make sure you have the people that are on call that actually have the keys and the ability to invoke the pausing actions to invoke the upgrade and actually do all the onchain activities. This is an often overlooked piece because let's say something happens at 3 a.m. on a Sunday and someone's in different time zone or someone's at e ECC at some event, right? And they're not available to actually invoke this. You are sitting on your thumbs and the exploits happened and it's it's likely too late.

This brings us to our next insight. Uh automate responses where possible. So if you can do any sort of pre-signed pausing um this will speed up or improve the impact of your mitigations because once something happens a lot of time it happens so fast they don't even have enough time to get on the computer and actually pause. So if you can set up a pre-signed pause in a secure way it's highly recommended. It's a lot more impactful. I will say you have to be kind of cognizant of the user experience impacts. recommend using alerts with extremely high if not 100% fidelity where you know if I get this alert we need to pause. If you're getting alerts that are showing up as false positives and your contract is pausing all the time, you're not going to have a good experience and that on call engineer is

not going to be too happy waking up all the time and having to unpause and resolve these things. Uh and the final insight is request assistance from Seal and other community initiatives. We're all here together to help secure the ecosystem. If something goes wrong, don't be afraid to reach out for help. Make sure you have within your runbooks how to detail what's going on because if you get someone involved, you want to get them up to speed as quickly as possible. And we really and you really need to leverage Seal and the other people that are available and ready to help um in case something goes wrong and we want to make sure that you know we're mediating it and keeping the the ecosystem safe.

So, pause there. Final slide. We are hiring. So if you're interested in helping securing problems like fault proofs base and other web3 projects we're building at Coinbase, we would love to have you. Um you can explore more of the positions on the Coinbase page. Uh and I'll pause there and see if we have any questions. and that made may also create some unwanted chaos uh um especially for

onboarding people in the team and so on. Have you already kind of built an internal like dashboard and that brings it all together for for monitoring and I mean exactly for this purpose to be really reactive Yeah. Uh, all of our monitors are actually documented and all the onboarding process goes over them and every single alert that we get, we actually have a runbook that you can go and just we have a dedicated space for that. And um yeah I think also for the smart contract inventory we have actually built internal system that tracks of all the contracts that we have and all the dependencies and in case that we need to check if our internal contract is using some external parties

such as something getting hacked as he mentioned on Twitter we will be able to identify quickly and find the faulty contracts that we need to work on. Not currently, but it's something as we build tools for ourselves, we also want to enable people building on base to use the same tools and leverage kind of our best practices. We do have a program with hexagate where if you're building on base, you can get basically um simple monitors. You get up to 10 monitors, 10 contracts, and you can use their threat modeling as well. So, as we build things internally, we actually use them and they're securing our systems, we look to open them up more. Um, back to your

comment earlier too, we also just we try to use as much paved roads as possible, whatever it currently exists. So, we have our smart contract inventory, we use Slack, we use Jira, Pager of Duty. We don't want to over complicate the process, make it harder for the devs. We kind of want it to be streamlined. So, we've been using uh the existing things that we have and then building upon those. Thank you.