Lessons learned from over 300 security audits

Transcript

my name is sebastian everyone calls me sebby today i'm going to be talking about lessons learned from over 300 security audits just a bit about myself because probably no one has any idea who i am working in security since 2011 so have around 10 years experience and i've been at big companies but like i'm having a blast uh at quan stamp really like the the atmosphere in this space super innovative fast-paced that's enough about me about to talk over almost

four years now we've done 307 audits completed and through those audits we found a total of 2445 issues and you can see on this slide they've been broken up uh according to their severity level it's pretty normal that like the high severity issues are less than medium low or informational interestingly the undetermined issues are some things that we weren't able to judge how severe it is due to a lack of documentation for instance

right we didn't know how bad it could be so we marked it as undetermined and we were expecting the teams to tell us it's high it's medium it's low sometimes that happened but like for these 283 it didn't happen but interesting enough interestingly enough you're going to see that many of them were fixed more than the informational or yeah close to the low ones right so here it shows here this this chart shows that 87 of the high issues were solved which is good everyone wants to solve the high issues less and less issues are solved as you go down

to informational level but now it gets it gets more interesting when you see the undetermined issues so there were several cases where these undetermined issues were actually high severity or medium severity so first piece of advice don't ignore the undetermined severity levels they don't mean that they're not important here's the same chart showing the absolute values so ideally all of these bars would be this red reddish color which mean which would mean that everything has been fixed but no as you saw the percentages on the previous slide

certain percentages are fixed certain not but it's interesting to see this so given this these stats like how many audits do you think out of our 307 had all issues fixed zero yeah zero had all issues fixed here i just uh added some histograms of how many issues are unfixed according to their severity levels and then also the total so you can see like here basically if you sum up all of these bars you get to 307 so out of the 307 audits somewhere

around 80 had one unresolved issue okay then around 50 had two unresolved issues and so on right and there at the far right you can see an audit or a few which had like 28 unresolved issues right so given all this information bear in mind that if someone says that our project has been audited it doesn't mean that that project is secure okay and i also like to quote uh this saying from dijkstra he says like program testing can only be used to show the presence of bugs but not

to show their absence so yeah take take claims from from projects that have been audited with a grain of salt look at the look at the uh audit report okay moving forward so what can you do right is it game over like is everything unsecured no like but what can you do right you could use defense in depth so here i'm gonna walk through each of these layers so defense and depth is basically a methodology to add multiple layers of protection and not just rely on one like many projects they do a security audit and that's it

other projects just to a bug bounty and that's it you should you should use multiple things right to help your security and uh yeah if one of the layers layers fail then the others could could help protect here at the bottom i'm showing the crown jewels basically those are the assets that you have in your contract that you want to protect and we will be going through these uh layers one by one starting at the top first layer develop using best practices oops so here we have two steps first one is the process and the second one is the coding for the process

it's not very different from any other classical software project right you should have great documentation right that means that you can start simple you can start from user stories and derive functional and non-functional requirements which are crucial for audits you should you should draw diagrams whatever you prefer just to help the auditors understand how you imagine the project working you also have code comments these are missing quite often so have comments either or at the beginning of your function in line however you prefer but have them and least but

last but not least in documentation you should have a description of all the roles all the stakeholders and what they're supposed well they should be allowed to do and what not right regarding the process there's a few more things here so pull requests you should have like peer reviews and not just like one developer merging into master every time right each new feature merge it to master right that creates a lot of potential for for bugs other than that you should have tests for each pull request don't just add the code for the functionality also add

tests right you should be careful because if you just keep adding code at the end you're going to be so psyched to launch that you're not going to write any tests and you should also make sure that the dependencies that you're testing with right a project runs over several months so when you start developing you set up your platform you do npm yarn install whatever you download those versions which are at that point uh set right and if you don't update them in the process and you keep testing with those old ones like maybe half a year a year later when you actually deploy on chain those versions will be different and the behavior might be different now that

i don't have a good example uh off the top of my head of some project that got like totally destroyed because of that but it might happen right finally here on this slide this is super important we see this a lot when we audit you should fork a secure version right we get a lot of um audits where it's just a diff between some very well-known project and their code right and um yeah that project is well known it's been audited a bunch of times but the commit that we're given that is forked from

has not been audited it might be even like it was upset success susceptible to a hack right which was published but the the the developers didn't know at the time right they might not be uh security savvy they might have not looked at when the hack happened right so it happened that someone forked and built on top of something which was hacked right so be careful with that regarding coding this is a list taken from the the cmu cert website and you can see how well it applies to coding in this space as well right basically the biggest culprit validating

untrusted inputs and data sources right that means malicious users don't trust the users that they're going to give you good input even if they're admins right they might type mistyped something right and that's a problem so you should always validate also data sources like oracles right should check if the output of oracles are as expected compiler warnings this is something which is oftentimes totally ignored right we sometimes build projects that we are auditing to run the tests to dynamic analysis and it's full of compiler warnings which like off the top of the bat like raise

some findings right so these things are something that you should fix immediately when you're when you're building if you see compiler warnings fix them another important one access control right there's so many hacks right the parity wall attack right due to poor access control right other tons of hacks due to this kind of thing so you should carefully design this and where do you start well you start with that documentation about what each role should do and what shouldn't do right then you can design a proper access control policy the fourth one is is

in italics because it's quite interesting so in classical security it's default denied right but in typical solidity it's default grant right when you write a function anyone can call it you need to explicitly write a modifier or introduce a condition which is not going to allow anyone to be able to call that function right so it's tricky because the default in solidity is allow right so you need to be careful to add these modifiers to each function that needs it fifth one and my personal favorite keep it simple so there's there's plenty of cool

patterns in this space right the diamond pattern super cool but really hard to audit so if you can do uh a project which is quite simple in a way which requires two solidity files don't use the diamond pattern instead right because maybe it's going to give you headaches if you don't plan on using the full power of that standard right sixth one you need to give your roles again going back to the roles the the least privilege right so you shouldn't allow admins to do absolutely everything that other roles do because they're admins

and they're trusted right an admin key could be stolen we've seen a lot of rug pulls lately so if the admin doesn't need to do magical stuff just leave them like the basic stuff that they need to do and let governance or other roles do stuff like uh which is which is harmful to end users seven you should sanitize output data this means again going back this is one of one of the the biggest problems that we see today with with defy composability right you see a lot of projects working uh interacting with each other building on top of each other and

using those projects as like yeah this is going to give me exactly what i want right but then surprise surprise you call i don't know swap on uni swap v2 and you get a way lower value than you actually wanted to get for that swap right or you call some whatever d5 you have a d5 yield aggregator and you're calling all sorts of d5 projects and you're not getting back the amount that you want for the shares that you're minting right so be careful with sanitizing outputs and don't assume that you're going to get the right amount always defense and depth we're do we're talking about right now

uh we're at layer one still layers to come use that and it's pretty good number nine testing so have unit tests have integration tests and most importantly main net forking tests right you should be able to test your project on the real network last but not least define security requirements and model the threats if you're not experiencing that contact an auditing firm they're going to help you they're going to be happy that you came to them early and you're able to model threats at an early stage second layer automated scanning and testing here you

leverage the power of oops of continuous integration systems so basically these systems allow you to automate a lot of things that you should be checking so first thing the tests the things that you've been writing all the time make sure that when you add a new feature all the tests pass and you're not allowed to merge otherwise okay you should also compute coverage and mandate that something like branch coverage using a tool like solidity coverage is always above 90 run static code analyzers right there's plenty of them they're really good they have a lot of false positives but still

you can have someone mark things as false positives and be sure that you haven't missed stuff okay also linters are important right if you want to have a smooth audit you shouldn't have your code all over the place in different styles and so on so linters play a good role there if you do all of this before you start an audit your auditors are going to be really happy and the process is going to be much smoother i don't i want to stress this a lot this is something that that should be done much more in this base testing so sad when when we get projects which have zero tests okay always test the extreme cases

when you're integrating question can compound run out of liquidity yeah it can so write tests which take that into account you don't want your project to explode if that happens same thing on the previous slide fork testing do mainnet fork testing that's going to help you test on the real environment and just an example of things that could go wrong number of decimal issues right this is like very common nowadays surprisingly um it's because mock tokens they have like their vanilla tokens they always use 18 decimals right if you don't have someone who's like very eager to change the mock values

like the default values you have 18 decimals and then when you try it with wrap bitcoin surprise it doesn't have 18 decimals and you're gonna like transfer more than more bitcoin than exists also you might have tests with just a subset like some everyone may might want to test with dye or wrapped ether again those have 18 decimals right if you if you want to test with subset test with those tokens which are interesting right usdc usdt they have different behaviors they're going to break some stuff if there is if there are like issues in your code same thing as we we talked about before

um you should document what the parties can do and cannot do and test the negative cases if you're saying the admin should be allowed to transfer whatever test if a non-admin user can transfer that amount right code coverage is not the main thing in testing okay like we've seen test suites which have 100 branch coverage across the board they're just unit tests which just aim to maximize code coverage they have zero assertions right so if you have like 100 coverage with zero assertions it's useless so check the quality of the tests

the assert statements assumptions right make the assumptions explicit like if you have assumptions like okay this test should only run in those conditions make that explicit and the auditors will tell you if that's realistic or not cool okay i'm getting to the core of the talk with the lessons learned so this is where all that those lessons learned are and first question who here has gotten a security audit okay and who wants to get one awesome okay um this is for this this part is is is focused for you

so the process that we use at quan stamp is first you're contacted by an auditing firm or you contact the auditing firm you share the quote you ask for you share the code and you ask for a quote the auditors are going to scope it and send you the quote the audit is performed the report is delivered afterwards you have time to discuss the report and implement the fixes after the fixes are implementing the the re-audit is performed and the issue the final report is issued now we're going to go through lessons learned in each of these parts because the process is quite simple

but there's a lot of like the devil is in the details before we go into that i wanted to give you like an idea of how the space has evolved over the years like when we started in 2017 there were just erc 20 audits in 2018 started seeing the first d5 primitives in addition to those erc20 audits 2019 much more d5 we saw also a bunch of bridges really diverse 2019 i would say 2020 defy summer a lot of nft projects as well really exciting but complexity grows right and this year

we're seeing like massive complexity with all this composability with cross chain d5 it's quite quite overwhelming given that there's so little documentation in testing so lessons learned while scoping the first thing the first phase right first decide what you want if you want a high level review a manual audit formal verification wherever you want just decide on it contact the firm ideally the code that they scope is going to be the same that has been audited right there have been cases where we're contacted to scope something and when we get to audit that thing it's like

double the size or even more so that is not cool and it's going to lead to delays and everything because we haven't planned for that much capacity right you should also set up a call a scoping call right where you tell the auditors here's our tech stack here's the repo structure these files are most important we're really afraid of those files because the developer was on vacation when he wrote them show them if you have like a setup right if you have like a test network or it's deployed on a test network give them documentation but don't overwhelm them right we had an audit where we were given like 100 white papers

and that was a bit too much but like don't like give them something and also set the expectations like what's your timeline when do you want to launch when do you think you're going to have time for fixes and so on right those are the things that you can do on the scoping call really early you figure out if that's the right company to audit your project here's something which i computed based on the data which we have at quan stamp it shows the number of auditors on the x-axis and the number of issues found during that audit on the y-axis and the reason why i wanted to plot this just to see what the trend

in number of issues found is so not surprising the more auditors you put on an audit the more issues you find of course there's exceptions here right but like some projects are really good so it's hard to find issues in them right they probably use defense in depth i heard it's good so in addition to that what can you do you shouldn't save try to save by having fewer auditors because that's just going to mean that you find fewer things you also should make sure that the backgrounds of your auditors are diverse because what we've noticed is that across them we have like auditors with like trading backgrounds for

quant traders we have people who've been in the security field for a long time we have developers which have been at top companies right and if we mix if we makes we also have academics right and if we mix all of those it's going to lead to an awesome report there's so many different issues right the quant traders they're going to look at all the financial modeling there the security guys are going to look at all the like language specific issues and everything like that academics they even try like really fancy stuff that has been recently published so it's really good to have diverse backgrounds of auditors don't try to speed things up by asking to audit just a div

because sometimes even a one-line change could lead to a huge hack right this happened this year okay one line might be super important if you just look at that line you're gonna miss it you need to look at the whole thing you need to make sure that your auditors have the necessary background to audit that div also don't try don't wait until the last minute okay it's good to plan these audits in advance because if you do it last minute and the auditors have given you a smaller scope because they weren't underestimating your project due to lack of documentation it might lead to delays right because

they need more time or if the things that they find are very hard to fix that might also lead to delays right so make sure that you're either flexible in when you launch or do the audit well in advance during the auditing phase this is pretty easy when you start the audit make sure that documentation is ready end user documentation is often ready because it's close to launch but these other things technical documentation interactions the math formulas those are often missing and it's really hard to figure out if the implementation is

correct without that information i think without this information without these kind of technical documentations and math from you can't even do formal verification right you need to know what the system is supposed to do should also have comments these speed up the auditing process a lot if you if you don't have comments the auditing process is going to be two times slower okay also do a deep dive or a walk through with the auditors when they start and record the video the auditors are going to keep watching that trying to find subtleties we use this a lot it helps

also shared communication channel right from the back right off the bat so answer questions as soon as possible and politely if you don't answer politely don't be surprised if they give you the cold shoulder or like are afraid to ask right you say like oh that's a stupid question then probably no one will ask anything anymore and this one is super important for for re-audits right like during the the fixes right don't make code changes while the audit is in progress that's going to lead to problems during this this period when you get the report

and you read the report it often happens that you say well these are not high severity issues these are i know these things i don't care about them right so we see a lot of uh issues being dismissed because it's not going attacker's not going to exploit this come on the attacker doesn't care about griefing attacks right who performs griefing attacks why would they do that well it turns out that they do right and anything like auditors report anything that is strange fishy right so have a discussion with them i asked them like what do you think would be the scenario why why why would an attacker do that also pay attention if the auditors

mentioned documentation issues and test issues often these things are totally ignored during audits or the re-auditing period so it's really good to to pay attention to those and fix those during the fixes when you're addressing the findings you should ask for a re-audit guide we have a great re-audit guide basically walks the customer through what they should do right some things which are important is open a separate pull request when fixing each issue don't mix everything together right you should know like the auditors should be able to see how they are fixed how each issue is fixed and not everything together but of

course you should also merge everything together so that they can took take a look at the ensemble right because if like some fix might invalidate another fix right provide argumentation why certain issues are fixed and why others are ignored and also mitigated right you might not want to fix an issue because you have a way to mitigate it which is lower cost and you're willing to accept part of the risk do not add extra features with the fixes like sometimes like some people sneak them in you know like come on it's just a small feature it's like nothing don't do that it's it's going to lead to to delays

and avoid refactoring like we've had these these problems when we get the re-audit the things have been fixed but like the code has been renamed files have been renamed everything has been realigned because the linter has finally been used and that creates really big difficulties when looking at the fixes right you're like what am i looking at like of course like there's tools to help you with that but it wastes time add the missing documentation and test issues that were were signaled that is going to help the re-audit because sometimes when the documentation has been

fixed during the re-audit other issues which are more important might come up those undetermined issues that we talked about in the beginning they become clear that their high severity medium low or informational that's super important super important on this slide is that you should also fix as soon as possible don't wait too long because auditors have a mental model of your project and if you wait too long and you give them the reaudit after a month or two they won't remember anything about your project right so it's like they're gonna take five times as long to re-audit just to remember what they were talking about when like what your project was about

and the last step when you're when you're finished when the final certificate has been issued it's cool to ask the auditing firm to deplo to publish this as a neutral third party so that your community doesn't say oh but maybe you've tampered with the audit report and you've marked everything as fixed how do we know that you didn't do that right if you ask the auditor to to publish this those kind of rumors might go away and also social media helps to spread the word right most importantly on this slide point number three here make sure that the code you deploy on chain is the same as the one that has been audited

because if not that one line could cost you like a lot of a lot of funds monitoring the fourth layer here you basically should define a strategy how you're going to respond you should have some monitoring in place alerting in place and once you get an alert like we sometimes see projects they have monitoring in place it's awesome it catches everything but they have absolutely no strategy how to deal with them right a hack happens what do we do right what should you do like depends when the when the issue was found if it's found by an internal team extending the project or whatever you

have maybe plenty of time to fix it until a malicious attacker finds it but if it's a bug bounty you might not have enough time right 90 days to responsible disclosure then they can publish it or if it's an exploit of a live system like how do you immediately react to reduce the impact of the hack right you should have some kind of alerting in place and a process so should respond to incidents collaboratively together with your auditors that that shared channel that you set up during the audit use that send the auditors the transaction which is fishy and ask them hey guys can you help

they're going to help you find the root cause find a way to maybe mitigate reduce like stop the attacker from exploiting and also to fix right we've done this successfully with plenty of our customers also perfect your post incident response during reviews right if it didn't work out the way you imagined it when you responded to some incident figure out an improvement of the strategy to make it smoother next time finally you can employ but there's a bunch of automated tools right that can do this and here i'm just posting a picture of our kwan stamp monitoring service which is deployed in production used by binance to monitor plenty of

contracts last issue last layer of defense insurance this is a an exciting one if everything else fails make sure you're covered right these are the layers right so you could you have quite some some space to maneuver here there this this uh area is quite new and there's they offer a bunch of different configurations right you could have a a b2b policy where you as a project you go to an insurer and say i want coverage of 80 of my aum right if it get if the project gets hacked you're responsible for giving back uh your end users the funds that have been

lost but you could also like say oh that's that's too big of a headache i don't want to be involved in that kind of stuff so do a b2c where the insurer directly is responsible for allowing end users to file claims and to get their uh insurance back you can choose between crypto native and traditional native there's a bunch of projects out there very cool very nice ideas there's also like traditional insurers which are starting to dip their toes right into into the space not so strongly in d5 but other spaces definitely and d5 is definitely up and coming for traditional insurers as well you have companies such as aeon and

lloyds who have been dipping their toes for like more than a year now in this maybe have a combo because in crypto native ones we've seen hacks right and then there's the question who insures the insurer right if the if your insurance pool is hacked and drained you might be left holding the bag also decide if you want an automated trigger for claims or a manual trigger right we know projects which have a manual trigger where there's a committee some experts decide this was a hack which was exploding a zero day and it was covered by the policy or not or there's projects which are up and

coming which have an automated trigger which don't need any decisions from human experts also try to tailor your policy as much as possible try to make sure that you know what you're covered for many projects might expect that they're covered for everything right like um if if someone buys insurance i'm insured i'm fine right but if like a reordering attack happens right or something like maybe at the at the at the layer one level right that's what i'm talking about that's not covered right you check out the policies those things are not covered you also

reduce you try to reduce the premium the more security layers you have the smaller your premium is because your risk is lower in conclusion just wanted to say that defense in depth is something that you should use i like to do this parallel with avoiding covey and the best practices and development the development best practices or like wearing a mask and social distancing right they're effective but can do more automated scanning and testing is like pcr testing audits are like the vaccine they're like 90

effective monitoring and incident response is like temperature measurement or other stuff and insurance is like if everyone everything fails i got sick you need some treatment that's it from my side thank you for your attention any questions or do we have time oh awesome hey uh in the beginning you're going over risk levels from high to low and informational how do you quantify

risk like what do you consider high versus low a medium like what's the categories in this yeah yeah that's a very good question oh thank you so basically we have human readable definitions about like in our audit report about what means to be what it means to be high and it generally involves something which is exploitable by an external party with no privileges and can have a high financial or reputational damage the medium one has lower damage the lower one is probably hard to exploit maybe by privileged users informational ones are very hard to exploit okay these are like they're easy to understand by

non-technical users right and and i would say like the community is pretty technical but there are definitely non-technical users who need these kind of descriptions from a technical perspective when we internally decide we also look at impact and likelihood right there's this matrix which has impact likelihood and you basically multiply those things right to decide what the severity is does that answer a question awesome any other questions don't be shy awesome if you want to read more about pawn stamp and what we're doing please visit our blogs we do a lot of

cool stuff oh there's another question awesome yeah okay um or do you unload like your the audit within quantum i mean what's the process yeah if you can say sure sure so what we do is um it's similar to what you've seen here right this this line here you're talking about just this phase or because like we we first always uh scope it we assign auditors which are best suited for that particular tech

stack and project so like if it's an nft project we have some people who have a lot of experience and passion for nfts if it's a d5 project depends if it's a yield aggregator we take the guys who've done a lot of earl aggregators put them on that right we try to combine that to make this diversity like have a have a quant trader have a security expert have a very good developer because that's how we're going to maximize the the efficiency the things which were found and we assigned these guys if if we need more than like sometimes the project is big right we need more than three auditors right the default is free we need more we're assigning more we're planning for it in advance we're planning like probably a month in advance

when it's going to be done what's the deadline we communicate it to the customer if they agree that's great when we start we do all those things which are mentioned there we ask for code walkthroughs we try to understand how the project works using the documentation or the walkthrough and then we like once we understood it we dig into the code so we don't we don't really jump into the code without understanding what what the thing is supposed to do right and many times we talk a lot with our customers via those shared communication channels so we ask them questions like did you guys intend to do this here because it's not documented right those kind of

questions we we ask them a lot we have long conversations maybe we have more have another meeting if necessary to to the code walk through and at the end everyone does it independently we don't we don't have a pool where everyone puts their notes every auditor does it independently we're highly distributed so we're not like necessarily people in the same office at the end when the internal deadline is there the audit lead collects all the notes comments if anything is unclear and then puts everything into an audit report that audit report is reviewed by all the other auditors and there's discussions on how severe is that issue is it really high severity or is it medium right we change these

things according to these discussions and we might also add or remove issues if we think like we miss something or if there's a false positive okay um last question uh and what is the process to the auditing process i mean you print print the code and try with like i can't hear you okay um it's better what's uh auditing process in fact do you print the code and what do you mean like for each other we have a pen yeah for each audito do you have like um a kind of framework and that every year does follow oh yeah yeah and

so we have an internal um checklist of things that you shouldn't forget to check right but we often go off script so each auditor goes through the code line by line we don't run the tools at the beginning only the audit lead runs tools at the end when everything has been compiled because we don't want to get biased okay so we go through the code line by line and we look at the checklist remember things that we need to check but like i said more often than not we just like each project is very specific in its own way right so maybe there's things on that checklist which like there's things missing right so we sort of like go

down the rabbit hole and find very specific bugs to those projects which are like according to their business logic or or something like that does that make sense yeah yeah totally thank you thank you awesome thank you for your attention have a great conference you