Optimizing Fuzzer Efficiency: A Benchmark for Smart Contract Invariant Tests

Transcript

[Applause] oh that's fine so hi everyone my name is Antonio vigano I'm an independent security researcher and I'm currently working on Audits and tooling one of these tools is for the FYI it's a automated fuzzer on the cloud so today I'm going to show a comparative analysis of fuzzers effectiveness and yeah so for this talk I'm just going through a brief introduction of how a fuzzer work and how to measure an Effectiveness the

performance officer what would be the methodology for that and some key takeaways um yeah so first of all uh yeah like in from from a high level overview a fuzzer we'll just throw some random inputs into your code and try to break it but going a step further we can think of it as like uh you start with some random value and maybe a corpus of previous transactions maybe previous input that you have sent um then until you you are satisfied maybe you have break some invariants maybe you have crashed your program

um you will pick one of those observations you will mutate them and uh if they are interesting you kind of uh keep add them to the list and you do that until you are satisfied so maybe reach a timeout and maybe you'll find a bug so um yeah naively we can already think that all fuzzers kind of are equal because they will just throw random input but that might not be the case since of these functions and this kind of bold uh functions here they might influence on on the performance of or effectiveness of a fuzzer so uh yeah so like measuring uh how kind of

how well a fuzzer can can can be at finding bugs is challenging so first of all because it's uh inherently random um and and also because the performance can vary a lot depending on the run so you might find the bug or bugs in the beginning of your campaign or maybe like after many hours um also because like it's different it's difficult to compare different fuzzers because how do you know that you've found a book um and finally because uh yeah so there are different ways of measuring uh like performance so some solutions that are

proposed here uh they come actually from a paper shared by some folks from 12 bits so like the the methodology suggests uh not using the average like average time to find anything because of the randomness uh you might have very different results from run to run so it should definitely compare medians and and standard deviation across different runs um trying different seeds so depending on the seed value you can have different results and also depending on the test environment so maybe you are running on a multi-core instance or maybe uh on like just a single worker you can have different results

um also it's important to first over long hours like not over only like a few runs so that you can have a very wide range of of values tested um yeah you should definitely try to look for unique bugs so not double counting the bugs are the inputs that will make your your your code crash basically and uh and also using a statistical test uh so that you can like make sure that your observation is accurate from a statistical point of view um yeah so here uh the the um the research that that I that I've

done is basically comparing Echidna the latest version and Foundry um using uh on uni swap V2 so basically I uh we injected different bugs on uniswap like 12 different bugs and we measure the time to find invariants that would break under these bugs um and then we run them for 30 different seed values and we set a one day timeout 24 hour timeout and finally we compare the median of those um of those uh times to break those invariants and we use this uh statistical test to assess which median

is lower than the other under uh specific uh like confidence level and here is an example more in depth of what that looks like so basically um on the left hand side is an instance of the mutated code so basically when you're providing liquidity to UNI swap you need to make sure that you are providing the same value of tokens of both tokens so um on uni swap you take the minimum uh value of both tokens that you are providing liquidity to make sure that you incentivize LPS

to not imbalance the pool so a mutated code like with an injected Bud would be taking the maximum value instead of the minimum value so this is like an example of a manually injected bug and on the right hand side is like what we are doing for our Foundry tests and Echidna test so basically we have a ad liquidity first test and then we check if any of the uni swap V2 environments break so for example one invariant is that whenever you add liquidity you increase k um and also uh like uh adding liquidity increases the total supply of LP tokens

and many many more ordering variants so we have 20 22 invariants and if any any of those break uh then kind of there's a bug and here are the results from the study so just uh take some time to uh like uh understand the the the the chart here but basically on the vertical axis we have um uh the time to uh find uh to break any of those environments for the mutated code um and and on the horizontal axis we have all of these 12 this different scenarios so each one is a different bug injected there so uh it's interesting like from just

from uh first uh Insight is that the results vary a lot so some bugs are much harder to find than others uh for example like bugs number six seven eight nine they are kind of very uh easy to find and others not that much uh the ones with the like hires uh with the bigger bigger bars and and also the dispersion is not the same across different bugs so for some tests uh you might have a higher variance than for for the others um and we can also see that the performance may not be the same for for any of those uh fuzzers

um and yeah so here are uh some key uh takeaways so basically um the performance can vary substantially from run to run so if you are using a fuzzer it will pick like a random seed by default and it will use that seed so depending on the seed you should find your your should break your environment faster or not so up to 100 like 1000 times faster like that can mean either find a bug in one minute or one day um so you should definitely perform many runs and let your fuzzle run for many hours and also like Foundry will break environments faster in some tests a

kidney other tests in other tests it's not conclusive so you should uh you should definitely try using more than one fuzzer it's not that hard to do it you should just like change a bit your your code there is an example on on the Ripple that I shared and yeah and that that's about it so if you have any questions about uh fuzzing or or Security in general uh please reach out to me thank you so much Antonio