Incident response at OpenZeppelin Contracts and how to be in the loop

Transcript

hello [Music] hi okay hello everyone my name is Francisco or frangio I'm an engineer at open Zeppelin where my team and I maintain a number of Open Source libraries and tools among them open Zeppelin contracts the most widely used library of smart

contracts on ethereum and ebm chains today and today I'm going to share with you some of our experience and learnings doing incident response for this project so the library has was launched in 2016 and our goal has always been to contribute to the security of the ecosystem by providing reusable smart contract components and we believe this can help by multiplying the efforts of the community in developing these features testing them reviewing them auditing them and potentially also in doing incident response for them since then our Ambitions have increased the scope of what we want to cover has grown and with that you know we want to

cover governance D5 and with that comes increased complexity and risk so it really requires us to take a very disciplined approach to security I like to think in Frameworks so I put it down into these four foundations that I I think the project really has to maintain which are engineering and open source best practices secure design principles multi-layered reviews as well as incident response which sorry incident preparedness which is the focus of today what does incident preparedness means it means that you know we're human processes can fail humans can can fail and so a bug may get through all of

these other layers that we've put and we need to be ready to to respond to that when that happens so we need to know what procedures we're going to follow what people we're going to Loop in what tools we're going to be using and need and in the past couple of years we've had a few incidents at open Zeppelin contracts and we had some learnings from this and developed some tools so let's talk about that mm-hmm an incident for us is a bug in the code we will usually receive this as a report on immunify which is where we host our bug Bounty and there will be some initial screening initially because many of these reports are invalid uh 80 of them that we've received so far have

been invalid so we need to filter those out but when we're in the other 20 it's usually pretty clear that we're looking at something that is real and so we need to put in place here the response protocol that we've developed the first thing that we have to do is understand the potential severity of what we're looking at Here Comes one of the first things that is peculiar about open supplement contracts which is that we put the code out and users download it and use it and we they do that on their own on environments that we don't know about without our supervision with without our control so we don't really know the way that these users are using the code and what are some maybe tangential things

about their environment that may make the the particular bug worse so we have to get really creative and think about what is the worst possible scenario of this bug that we have identified so what is the worst possible impact we like to use this framework which I think is pretty common which is to separately evaluate the impact of an issue that is what are the consequences it might have uh maybe is it stolen funds Frozen funds denial of service stuff like that and separately to assess the likelihood which is how easy or difficult is it to exploit that is there a profit motive by some some attacker might do that does it require some prior access and so we

grade each of these Dimensions between uh from low to high severity and from that we get a combined severity which may be from critical to informational so one example of this is in August 2021 we received a report about an issue in the time lock controller contract that we have it was a re-entrancy issue that could lead an attacker to take over the contract at this stage we will very likely identify different sets of conditions that might lead to different tiers of severity in this particular case depending on the way that the timeline controller was set up it could land on the critical severity because anyone would be able to take over the contract or it could land on the highest severity

which was because of lower likelihood which required some someone with prior access that presumably was trusted to exploit it and those are the kind of kinds of things that contribute to different grades of severity so once we understand this it's still hypothetical right we've we've done a thought experiment and try to come up with the worst possible thing that we can think of but then once we have that we can't just do okay we'll do the fix we'll publish a report with the right severity and be done with it because the code may be deployed on chain and so doing that wouldn't be responsible from us so we have to really assess what is the real practical impact of this bug

that we found this is where the second thing that I think is peculiar about open sampling contract shows up which is that we don't deploy the code right our users do and they do that on however many chains they they know and we we never hear about those deployments so when we have an issue like in this case in the timeline controller how do we know where are the instances that might potentially be affected uh-huh so we found two two ways to identify these instances the first one is by looking at events in this particular case in the Constructor of the timeline controller there was a event that uniquely identified the contract that we could look for by looking uh looking querying an RPC endpoint or some other

kind of API and that way get a list of all of the instances of the code that we could find on chain now once we did this we found a list of in this case it was about 300 instances which doesn't sound like a huge number we've actually seen larger numbers in other cases but we have to think what do we what will we have to do with with the contracts in this list and and some of those things that we have to do like get more information maybe get contact information our manual steps and 300 is quite a large number to go through one by one manually so we have to also find ways to either filter or prioritize this list in this particular case like I said there were different severity tiers and

in fact by looking at the events we were also able to identify for each instance in which severity or it fell and to get that information so we wrote some tools wrote some scripts which we later sort of say it for future response future incidents to look for those events to generate a Json or a CSV file to dump it on a spreadsheet which is what you can see right here and then go through manually one by one to to continue with the incident response right but when we did this the critical ones were very very few thankfully um and we were still left with a bunch of other contracts that we didn't really know how to go through right we wanted to prioritize them further so one idea

that we could have is well if we knew how much value is that each of these instances we would know how much risk each of those instances has and we were able to prioritize them and sort of go in order would be a lot more uh efficient for us but actually it turns out this is actually a really uh difficult thing to answer you look at an address how much value is under the control of that address you can't just look at the eth that is held at that account you also have to look at erc20 tokens once you do that you need to price them somehow to under sort of put them in a common unit of account and then you also have to ask well do you look at nfts and what if the contract is the owner of another another contract that is the contract that holds the

assets and however many levels of interaction that you might want so this is a tricky part we haven't yet cracked and it would be great to get an automated way to do that foreign in some cases the events may not be enough like in this case we were lucky there was this uniquely identified and datifying uh event but in other cases it may be a more standard contract that is an ERC there may be multiple implementations and it doesn't help us in that way or it may be that the bug was introduced in a later version and we would like to filter what contracts were deployed using that particular version or the versions that have been affected and the event doesn't help us there so

the other thing that we have found is useful is to look at a verified source code verifying source code is a very important part of our culture in this industry so we can sort of rely on that for most projects and if you're not very fine your sales code you need to understand that this is one of the consequences of that right you are not giving the information for free security researchers or People Like Us who may need that to act on an incident on an incident here I want to highlight one thing which is that this industry has benefited a lot from the open nature of blockchains there are things like Dune that are amazing to do a bunch of analysis on the data but source code is is a little bit

lagging behind because we rely mostly on centralized providers so even though the data is there and you can get it it's a lot harder you have to jump through some hoops and we've been able to do that but it could be a lot better so here what we have done is uh we scan uh all verified contracts and we are able to match them against our code that we know from the library and in that way we're able to identify what features of the library they're using and what version of the library they're using and this gives us the information that we need to filter and sort of act on these incidents so once we have this information we will have this big spreadsheet that I showed you and now we need to act on it we need

potentially like in this case in the timeline controller to execute a mitigation on chain so what we did in this case was we put together a mitigation kit with very clear instructions that we sent out we sent them out to the critical affected contracts first then to the high severely affected contracts and they sort of staggered way to sort of control uh more tightly how this information was disseminated because of the sensitive nature of it um it was quite hard actually to find this contact information for some projects we had to go and ethers can sort of see who had deployed it um identified the project maybe through some other connections on chain

and for some we found the project that they corresponded to but there was no contact information right so if you are developing a project it's really important to have very clearly available contact information for for this sort of security Communications so once we have done this we are clear the mitigations have been executed and now we can go ahead and publish the code sorry publish the fix and also publish an advisory The Advisory part is really important and it's also the reason why we recommend installing using npm and that is because there are tools like dependabot or Snick or stuff like that that will scan these known databases of security advisories and let you know

when you have a dependency that has an issue and known issue in it so we publish a security advisory for the npm package and if you are using the dependency you get notified very likely by one of these Bots and you can act on it one of the challenges here is that because open sampling contracts is a library that has a lot of features maybe you're just using one of them you're still going to get alerts for the contracts that you're not using um so we uh that's kind of one of the downside so we try to make it very clear in the title and in the description how to know which is the contract that's affected and also in the description how you can

know if you're using that contract whether you are affected so making all of this communication very clear is key to us to deal with this issue with a it's like a lack of granularity of alerts we're developing a tool currently that we call dependency Checker not very creatively for now and the tool what it does is it's able to scan a uh your project maybe your repository or maybe even a contract that has been verified and it is on a block Explorer and use this information that I mentioned before right what is what are the features that are in this contract what version of the contract is using and if there are any known security advisories for it in a very granular way

that is able to distinguish the features that are actually in use um this is a thing that you're going to be able to install uh maybe we might do it proactively I'm deciding just yet but you I think it will be pretty interesting and valuable so finally um regarding the other security foundations that I mentioned we've recently released a security Center page where we share a bunch of relevant information about the security processes that we follow the audits we've done the test coverage that we have the formal verification coverage and some other relevant statistics for example uh we can't see it on the screen here but

using this scanning technology that I told you about we are able to get an idea of how many instances of the code that we have built and shipped are deployed on chain and we can see that there are like hundreds of thousands of them it's actually quite interesting and we can get really good statistics of that that data set by the way we're still working on the accuracy of it but I would love to make that public at some point so that it could be hosted on Dion for example and you could use it to um against other data from the chain I think that could be really interesting so if that interests you please let me know finally we've recently started doing some incident response training for clients we teach how to react mitigate

respond to security incidents model threads identify weaknesses develop a plan and also perform simulations we've done this already with compound and sandbox it's not something I'm personally involved in but the team in open Zeppelin is doing this and I think it could be really interesting also if this is interesting to you you should reach out to us and that's it thank you hope that was interesting there may be some time for questions hi yeah thanks for this great talk I was

just curious about the custom tag you had in one slide you heard of a Custom Security advisory yeah this one a security contract here hi okay uh yeah so you asked about the Custom Security attack uh what what is your question specifically like how do you how do you use that right so you develop your code you would put this at the top of your contract um so in in the NAT spec right this is

an ad spectacle so it could go next to the documentation once you deploy it and you verify it it's going to be there and it's going to be we were able to pick it up in the same scanning process that we do through all the verified contracts and have very easily available contact information yeah and sorry it's not clear here but you would put next to it the email right or some other way to contact you that's the important part thanks okay thank you very much [Applause] [Music]