Security doesn’t stop at code complete

Transcript

uh welcome everyone uh my name is Dan McKeon I work at open Zeppelin we created a popular smart contracts Library probably a lot of you have heard of we also do security Audits and uh we also build tooling and that's mainly what I'm going to be talking to you today here on in security doesn't stop at code complete so why does this subject matter at all we have a big trust problem in web 3 that causes a lot of people that are kind of right on the edge uh maybe would be interested in some of the elements but they see headlines like you see on the right here there's 1.9 billion dollars stolen in hacks and we sort of wear it like a badge of honor of not your keys not your

coins and do your own research and we kind of put a lot of onus on users and so everything we can do to try and reduce that so that does include security audits it does include uh um coding securely by Design but it also includes what you do in deployment and thereafter and that's what we'll be talking about today uh so in web 2 where I came from a couple years ago before I was at open Zeppelin this is a very robust set of processes that whether you work at Amazon or some little startup you know everyone kind of does things in a similar way so we follow strict CI CD processes we do a lot of monitoring logging we represent our infrastructure

as code so it's reproducible we monitor performance and we kind of automate just about everything we can and there's a million Tools in that space they've been around for a long time these are some very popular ones probably many of you have heard of some of you are probably using and there are literally thousands Beyond you see on this list that you kind of hear about a new one almost every week when you're working in that space so in web 3 it is highly variable I think there are definitely a lot of protocols that are more developed have been around longer maybe have larger teams that are maybe have more robust processes but there is also a little bit more of a cowboy

mentality in a lot of the space and and that things are a little bit more freewheeling in some cases um so I think in many cases we're doing a CI but CD is more like deploying when someone sends you a slack message or something like this we monitor the Discord server uh and uh the deployments are often happening you know via some script that the the senior developer is firing off but we do have tooling now and so some of this tooling has been around for a little bit but there's been a lot of maturation here in the last year around the landscape of tooling we have available to us and even tooling that has been around there's been big improvements particularly in ux that have happened here in this last year and I think a lot of folks who maybe tried

various tooling options as of a year plus ago didn't have the greatest experience that today those tools might look actually very different so we're going to talk talking about a very wide array of topics today deployment monitoring and incident response we only have like 12 more minutes so we can only kind of touch the tip of the iceberg here but here we go if you want more details uh certainly there are a lot of great resources out there we have a Solutions team at open Zeppelin that actually uh Dives uh much deeper into into these areas so deployment and upgrades we'll talk about first we we want reproducibility and especially because multi-network has become so important so when you're doing something by Script when you're kind of like

changing Constructor arguments you're going to inevitably have human error and so we want to use an automated process you see the GitHub CI Runner here at the bottom many different options for this but but at any rate when you have reproducibility and you're able to just kind of change one config to change networks and you're able to have it that next time you deploy it is the exact same as the earlier deploy you also have auditability and logging so if something went wrong you can fix it at the system level and not have it be that someone fat fingered like a key in the Constructor argument and screwed everything up um private Key Management is also kind of a big thing in this in this world so there are relay Services out there um opens open Defender does have one uh

called relay there are other ones as well but one of the benefits of private Key Management uh is that uh if your API keys are compromised you can do key rotation where if you have a private key that you're relying on and that that address is very important to you and that's compromised in some way it's a little bit of a different story byte code verification so when we're upgrading and especially for non-technical users because I think technical users are are sometimes more diligent about this but a lot of signers on safes are not necessarily technical users and so you can use tooling to validate that the byte code that is in GitHub that you've proven that yes this is the right byte code the tribun chain is actually on chain and uh enable someone to kind of run that validation

for themselves at the point of signing a proposal and same deal with type safety and so this is more of a ux issue but it can turn into a security issue if you upgrade contract and it turns out you're you're not being backwards compatible or you're maybe exposing something you didn't think you were exposing and finally testing in lower environments so Fork networks test Nets and this you know I think probably a lot of us are us are doing to some degree I just use a tool by the way called conduit the other day that is fantastic for uh for fork networks deploying a fork Network I highly recommend folks check that out I think it was a paradigm a project um so in monitoring uh we're kind of in a unique place where we have a

tremendous amount of data uh in web3 and we also have uh really good signals as to um what kind of transaction uh we can map to an exploit and so uh systems that are characterized by those two two elements are very uh um uh they're great candidates for AI to be a good solution I don't think AI is a fantasy for everything but it's when got when it's got those two elements it's a really good uh good option for for kind of making a system work better than you can by a traditional methods so forta is a decentralized monitoring protocol one of their uh the the set of bots they've built is around machine learning and uh

and there was there have been a number of of instances like this but I just kind of had one use case here uh or one case study of the urine attack on April 13th we're a forta mlbot fired this uh this alert uh 529 UTC just over 20 minutes later you're involved attacked leads to 11 million dollars in losses and so there's a lot we can do with AI I think that potentially broadly like I'm kind of describing here but possibly also at a contract level um and I think that that area is kind of just uh just beginning but um but it's a big thing to look out for especially over this next year as we're now kind of gaining this this big interest in in AI

so it should be taken with a grain of salt because no monitoring solution is perfect uh this attack detector bot I reference it does have an 80 roughly uh recall that means that of the actual attacks that happened 80 percent of them this spot could detect however uh it's only a 20 precision and so this speaks to the false positive rate so of every five uh alerts that are fired four of them are false positive so we can't just every time bot like this fires or can't just pause the protocol that's not really a good solution and we'll talk about that more in a minute but uh suspicious activity screening like this this kind of like generalized approach that I'm talking about this applies to all contracts I think this is something

that that we've used necessary but not sufficient there is also certainly work that needs to be done uh based on your protocol and every protocol is different in this next section is gonna there's kind of a set of questions here and these may look very different the answers to them for each protocol so this is one that we all think about which invariance signal a threat or an active exploit so if there's a highly profitable trading strategy and you have a lot of value being taken out of your contract what does that look like what are the parameters you might monitor to to validate that and then you know if you've just made a deployment how do you know that that sound how do you know that's behaving in the way that uh that you think it should behaving

um what exposure do you have to changes in external dependencies uh we make assumptions sometimes like that uh one stable coin dollar is going to equal one dollar and then all of a sudden you know crazy things happen and then that assumption is violated and uh monitoring other contracts where this data is coming from is also uh kind of part of your security framework so if there's price oracles or conditions on other networks like if you run a bridge for instance um finally if your protocol is exploited um so you know maybe you're monitoring fires but like that was the hack so it's kind of too late to to fix that now or to prevent that from happening but uh even after that happens uh what What's

your action plan then and and how quickly uh can you execute on it um so uh that's kind of monitoring sort of the triggering that is going to lead to your incident response method uh and so we I mentioned this a minute ago but pausing at the protocol level is kind of something I think we often think about at a high level like that's how incident response works there's actually a lot of other ways you can approach this you can be much more targeted with your pausing you can uh pause based on uh you know specific functions uh maybe specific senders uh you can also like limit withdrawals you can see something that's highly suspicious and you can set a cap that's like somewhere lower than the 13

million or whatever and that maybe you know in all these cases you're not can't necessarily prevent everything in some cases you're kind of mitigating the issue and that way you know with false positives which any type of monitoring is inevitable if you try and have no false positives you're going to probably end up missing a lot of the real exploits and so we use tooling for this and this is uh kind of this this burgeoning area of web3 that they're now uh you know a number of different options um that uh that we have available to us so I'm just going to show kind of a trivial example here but this is sort of like roughly how tooling can Aid in the incident response scenario so we have this transaction and this for whatever

reason we've determined is a malicious transaction and you know like the scenario I just referenced a moment ago this for tobot detects it uh it fires uh it can fire uh it can do a variety of things in this case it's going to trigger a Defender Sentinel and then that Sentinel then is going to execute some arbitrary code so this is Javascript code we have in an auto task and what we're doing here is creating a proposal that is in effect uh sending it's going to do a couple different things it can send a message to the gnosis safe transaction service and so now you have your transaction ready to go and gnosis safe like the moment after

the exploit has occurred or that there's a suspicious transaction that indicates an exploit is about to occur you can also trigger services like pagerduty and so you need to think about you know if this does happen in the middle of the night how are the developers whoever's managing this incident going to uh address this situation uh and how are they going to wake up from sleep and do something in 20 minutes um and uh and so now we have our gnosis safe proposal finally hopefully everyone jumps on it we validated that yes this this proposal makes sense it's probably a defensive posture where something is again we've already gamed this out and everything and we've uh kind of made the determination that that this is safe

hopefully beforehand um and and we go ahead and actually execute it so uh finally this this area of after an exploit uh there's a wide variety of things you can do this a guy named Gaul who gave a presentation earlier that was excellent on much more detail of what to do after an exploit but high level uh you know chain analysis has an API for this there are others too um but uh but essentially like you have things you want to do on chain you might want to execute a pause at that point because there are a lot of cases where attackers will will double dip and maybe it is multiple transactions spread over you know 10 or 20 minutes that actually take everything um and uh and you know you want a common strategy in all manner of other things

uh that that go along with that with having an exploit occur uh so three key takeaways uh from the presentation here uh would be first to identify opportunities to replace any manual processes with tooling and so I particularly talked about deployment but there's lots of other opportunities as well probably here of folks who are running scripts themselves uh develop a plan around your incident response uh you probably have you know a generalized strategy in a number of different scenarios based on what exactly is occurring and finally simulate an exploit uh and and test your systems and see if this actually occurs like how quickly can we actually execute on this um so finally a few folks who contributed the presentation uh

Christian Seifert at forta uh Mike Casey and Jonathan uh it opens up when we're all very helpful in putting together the content uh other uh opens Upland talks here coming up in the next few days we got Barto on July 19th and Hadrian talking about the contracts 5.0 uh release that we're very excited about finally there is a pull app if anyone cares for a co-op and uh on the left is a link to a sign up for Defenders so thank you very much good [Applause] okay we can take one question anyone good question on the incident response

when do you think is the right time to keep processes in-house versus bring in you know external white white hats or third parties to be able to maybe rescue funds or coordinate uh response after an exploit has occurred they're talking about I think that um so in terms of like chain analysis that is an external service I do think that probably it's like a lot of these things like you you know they're handling a lot of the leg work you would need to build in-house otherwise so I think using apis like that probably makes a lot of sense um but you know teams that are big enough to have like a PR team uh and so forth like that's part of your incident response is you want to get on Twitter you want to have dedicated people uh responding to it so I think there are

external entities you want to bring in pretty much right away but certainly your team would be very involved as well thank you pretty sure