Quality of Auditors or Quantity?

Transcript

all right hey guys um cool so my talk today is going to be a little bit of a continuation of the auditing panel that we just had about an hour ago um about kind of conventional auditing versus the community contest approach to auditing um the title here is quality of Auditors or quantity you know which one matters more um and so last defy security Summit Sherlock announced released the hybrid audit contest model which is sort of a combination of an audit contest and reserving

well-known top Auditors to also audit the code base in a more traditional fashion alongside it and it's a really cool model uh basically there's incentives for both sides to perform well if they don't perform well there are negatives they lose out on money they lose it on ranking they lose out on potential leading opportunities in the future um so last D5 security Summit was end of August so we're about 10.5 months into this new hybrid audit contest approach that Sherlock has been kind of pioneering we've done about 85 contests and we found around a thousand higher medium vulnerabilities in those 85 contests and so we're now at the point where we have like a kind of

statistically significant data set that we can sort of look at and try to find some interesting things in um and one of the important questions or one of the questions that people are very curious about is who actually is finding most of the issues in these contests in these audits that kind of combine both the traditional and the community approach so just as a reminder in Sherlock contest there's usually one to two highly ranked Elite Auditors we call them lead senior Watsons they get paid a lot of money every week to show up and do the audit and then there's roughly you know 200 Security Experts on average

I think it's closer to three to four hundred at this point but across that data set it's about 200 on average that are looking at these contests and so the question for you all is who do you think finds more of the critical and important bugs in these audits raise your hand if you think the one to two highly ranked Elite Auditors are finding more of the critical bugs all right maybe like 30 hands uh raise your hand if you think the 200 kind of random Security Experts all over the world are finding the most critical bugs about the same all right to split uh the

split audience so we'll start with non-critical first these are like the medium vulnerabilities these are things that are not super critical they're not um kind of crazy in terms of could hack your protocol immediately as you can see one to two leads in your Watsons they find 46 percent the community finds 81 percent um and that's not super surprising because a lot of these things are sort of creative they're sort of edge cases that maybe you would find maybe you wouldn't it's not the worst thing in the world if you miss one so having 200 people looking at code just kind of yeah you just find a lot more of these but moving to who finds the actual critical vulnerabilities it's really close so the one to two lead senior Watson's found 66 percent of them roughly the community found 76 roughly

um so I mean I would almost call that a tie but as you can see the community kind of edges out the senior Watsons when it comes to finding those critical vulnerabilities in Sherlock contests over the last 10 and a half months so we think that's really kind of interesting and the fact that both numbers are so high is really a testament to the skill of both sides in my opinion and another way of showing this data is basically if you don't have lead senior Watsons if you don't have those reserved Auditors doing this audit then you will miss roughly one quarter of the critical vulnerabilities you know in the code base and if you don't have that Community aspect if you don't have the contest with hundreds of Auditors participating then you will miss about

one third of the critical vulnerabilities in your code base um and so here's a small case study Sherlock did a audit on the Bedrock system of optimism their upgrade earlier this year it was the seventh audit so this was like a pretty well audited code base at this point Sherlock was still able to find three kind of critical High vulnerabilities we kind of categorize those together um and the interesting thing about those three vulnerabilities wasn't so much that we found them but it was how they were found um and you can see here one of the vulnerabilities was found by the lead senior Watsons and it was solo so only the lead senior Watsons found him shout out to Zach and Trust I think Trust might be here

um for finding those they were kind of the team the lead senior Watson team that took on this audit and then one person from the community found a solo finding that the lead singer Watsons did not find and that was he-1m really crazy bug we can talk about that in another talk and then there was another critical vulnerability that was kind of that was submitted by both lead senior Watsons and the community and the interesting about this is it's kind of a perfect case study for if you had only had one of those parties if you only had the community you would have missed a critical vulnerability if you only had the lead senior Watsons you would have missed a critical vulnerability um and so Sherlock's stance on what's better the community or you know the

traditional style of auditing is having both is better you know there's not like this you don't need to choose one or the other you should have both you should combine them in the audit and Sherlock is the only one that kind of provides that right now we're the only auditor that has both the contest model and that more traditional approach of including well-known Security Experts and paying them a fixed amount of money to show up uh and another show for Sherlock is that we have millions of dollars of smart contract coverage that any protocol that goes to a Sherlock audit is able to activate after they do the audit [Music] um so yeah that's pretty much it and the last thing to talk about is Sherlock is shipping for the people who compete on

Sherlock which I know there are a decent amount of them in the audience uh Sherlock is shipping a new feature that will go live for the index contest starting tomorrow which is anonymous submissions which people have been asking about for a while and that means that your Sherlock username will no longer be shown in the judging contest uh basically if you are an unknown security expert someone might see that you're less well-known than the lead senior Watson and they might say I'm not sure that issue is real now everybody gets the same green spicy unicorn made up name and there's no way to tell who the lead senior Watson is you have to only judge the issue on its merits so

that's coming out tomorrow we're excited for it and hopefully all the Watsons are too thank you