The Art of Manually Guided Fuzzing - DeFi Security Summit 2025

Transcript

I'm auditing and fuzzing basically since uh 2021. Uh I'm also teaching at uh Czech Technical University uh and uh we are internally developing a tool called wake with which uh we will be working today and uh uh it's different from the current tools in mostly in that that you are fuzzing in with Python language solidity code so you are not using the same language and it has some advantages over the other tools when uh you are testing with a different language.

Um ah yeah and also I accidentally named the whole so that's not so manually guided fuzzing. So what is mgf about? uh it's about invariance execution flows uh mostly we will uh see later what is uh the fast state uh machine but uh the key point is you probably know what is invariant right it's some uh piece of code that uh should hold all the time uh let's say uh you are um uh you are checking the invariant if uh total supply is still

equal to the total supply in the code or something like that and flow is uh something what uh tells you how to interact with the code. So we are giving flows to the fuzzer and we are checking invariance with that. So simple flow here uh we are we have some sender some recipient and some amount. First thing we will mint some ERC20 and then uh we transfer token. Okay. So V framework it's Python based it's open source. Uh I hope you will install it and follow me uh with the uh

demo. And uh one great thing on it is uh it has autogenerating typing stops. That mean that uh when you are writing uh your tests you have a like a very rich type hinting. So uh it helps you uh to write test more easily. It also helps LM to understand uh what uh it should autocomplete and something like that. So uh it's very helpful. It has also forking support. So if you have uh some uh for example let's say new contract what you are developing and you want to connect it uh to rest of the architecture that is already deployed then you can simply fork the mainet at specify block or other chain uh and uh

use uh locally test uh with uh these mainet contracts uh over your uh new develop contract. So uh uh it's very handy and with the uh newest version of VG it's also extremely fast. So uh you can uh for example f only on uh like uh fork contracts. So you can for example deploy your uh new protocol that you were testing locally but but you can pass the addresses uh to your test and you can like fuzz it on chain. Uh okay. And great features are test reproducibility. Uh when when you are fuzzing uh it uh always starts with some

random seat and uh uh that random seat is displayed to you. So you can reproduce the fuzzing sequence to get back uh uh back to the same buck what you like encountered during your testing. And also there is a feature for shrinking. So you can easily uh reintroduce the bug with uh the saved sequence. And the latest version uh if you will do uh pip install eat it will install version 420 I think something like that.

uh but uh if uh there is also uh release candidate uh I will show you you the command then uh and uh that way you can install the newest version. Uh for today's demo it will be working with both versions. Uh I will be using the newer one because uh it has a rebrand core in Rust. So it's very quick and more comfortable to use. So uh so the first test state machine uh we are basically writing some uh Python class and filling some sections in it as you can see on the stamp on the right.

So the fastest starting with pre-sequence where in the pre-sequence you usually want to deploy some contracts or do some initial steps uh that will help you uh to like better uh cover uh like help help the fuzzer a lot with uh next coverage. Uh then uh you have flows and those flows are uh executed with some weights. We will get back to it later. uh you can set weights for each flow how much it should exe uh how much times it should execute. Uh there is several other helpers but the main point of this is that uh you are executing flows and after each flow you execute all the invariants and uh this

way you are just going for example here you have uh 10 sequences uh and 100 flows that means it will uh run first sequence and uh execute pre-sequence 100 flows and uh 200 invarants in that case and then it will start from the scratch and uh get to another sequence and uh again run 100 flows. So this is very important uh like uh number to balance. Sometimes it's more viable to start from the scratch after some uh number of flow because it will help you to uh like better cover

the possible space where you can find bugs. But sometimes you want to like execute lot of flows uh uh in a row. So uh it's good to play with it uh uh make one number bigger than another and then switch it for example and try how it will behave. Yeah, best practices are basically that you are creating flow per each transaction. That means that uh when you will be uh testing it then uh you will execute all transactions randomly based on the weights uh distribution what you selected. But sometimes you can also create more flows uh which will help you uh to reach uh some state that won't be

usually reachable by usual fuzzer. uh then uh the important thing that uh you are trying all the time like optimize the fuzzing by combining stateful and stateless fuzzing. So you some things are more viable for you to check with assertions let's say events and errors and uh other things like that are based on some state variables and so on. Uh these are better to check with invariance. Yeah.

And uh this one is also important uh view functions only use for invariance. So uh we will be practicing here like uh uh differential fast testing. That means that we will be basically creating model uh in python of uh the solidity codebase and we will be testing over it. Uh this helps us like uh to discover very neat bugs and uh for that you have to always check against your storage in your model against your expectation and not uh uh from the view functions from uh uh directly from uh the contract storage.

So these view functions are uh usually used only in invariance where you are checking if actually the thing what is happening in contract is same as your expectation in your python model. Yeah. And uh one uh like uh very small thing which uh we will not probably encounter today but if you will be using set then I recommend using ordered set because uh it's otherwise messes a little bit uh uh the uh uh random seat generation. So uh you would you will not have a reproducible fast test.

Uh you see that there on that slides there there is always that QR code. So uh when someone comes uh later to this so he can uh just join to it. Uh but there is the link to the GitHub what we will be uh doing together. Okay. So uh we will go quickly through the structure uh of the flow. So first thing uh is decorator for flow which uh tells fuzzer that this function is a flow and um uh then we are preparing some random inputs. So uh this is a very important

part. uh we can uh like uh choose uh some uh more realistic numbers uh to help it to not just uh uh for example be generating some numbers that are close to you max so we can like change the probability and choose uh some corner values some uh something in the middle something more realistic so we can play with it uh the other thing part is pre-execution uh so that's uh the thing what you are prepared preparing before you are executing the transaction alone. Uh so uh in this case it's just uh that you are minting some token and approving it.

But um and it's extremely helpful because uh a lot of fuzzers are actually just waiting when approve is called and then they can uh call the transaction itself. But uh since you are guiding fuzzer here that means that you can do it. So you will have a like 100% success rate. Uh and when the transaction is uh for example uh more complicated you can for example do some uh uh signing here uh some uh payload creation and so on. So this is also important part. This is like where uh the auditor or engineer is just work uh thinking about it very deeply to like uh uh prepare uh random inputs that will be given to the fuzzer.

But um uh also uh to try to uh like uh help to catch all the possible ways of executing the the transaction and uh there is the execution itself. So uh we will be also using today very helpful thing which is uh may revert to wrapper and uh uh it is helpful in it that uh uh the uh the transaction can revert or it doesn't need to uh it maybe can sound weird but uh uh it's very useful since you can uh catch the exception and then you can like uh enumerate underate all the scenarios and

uh like nice fillet handle uh the uh return values uh or rewarding values from uh the transaction. Yeah. And uh this is the part when you're checking errors then you are checking events in transaction and the last thing is that you are updating the Python state. So the invariants uh will be succeeding or not based on the on that uh if there is some bug. Okay. So this is from the presentation.

So we can jump to the example. So who will be following today? What? No. If you have working Python environment that then it is like very quick to set it up. Uh great. Um is it somehow readable? Yeah.

Okay. Uh so we have there like some simple token world. Uh we can see that there are defined some errors. Uh there's some immutable var for for token then uh some state variables some events then in constructor we are specifying the token and some minimal deposit and maximum deposit amounts. Then there is function for deposit for withdraw for emergency withdraw and some setter and view function.

Cool. So uh if you will check the repository there are also already solutions but uh we will start with this one. Okay. Okay. So there's the template what we will be filling and uh so let's start from the first line. So we are importing some basic uh things that are helpful uh for Python testing. What is great since you

are working with Python you can import anything. You can for example import some uh uh graphical libraries and you can write unit test that will also draw something. It can draw some curves or help you understand tokconomics of a token. So you can do anything there. Uh for example during uh some audit I created a fuzzing campaign uh where uh in the end I uh wipe coded a dashboard where I just exported uh all the data from the fuzzing to CSV and that CSV uh went to the wipe coded uh uh web server and uh I could just check on my phone in the kitchen how my uh fastest is being

executed on local network. you can do like a lot of things with it. It can uh uh it's great that you can like use uh uh all the usual stuff from the Python and leverage it uh on the testing solidity. Then there is uh import from vague uh and from vague fuzzing module. Then we have there the important thing which is pi types and uh those are that uh bindings to solidity that helps us uh to get uh uh type hinting and then we have their logger because it's uh more effective and uh uh easier to use uh logger than just printing

something in fastest. So you will be using logger. You can uh like uh place the logger.infos infosh everywhere in your facet. Let's let's say after each flow execution and so on and uh when you run it then you can observe the logs that there is happening something. But uh if you want to have a long running uh task uh then you can just uh change the uh level for the logging and it will be without output. So it's very comfortable uh and I recommend using uh logger for it.

Yeah. Uh so and uh there is uh the class which inherits uh from fastest that means that uh it will have a properties of uh our fuzzer and uh there is the pre-sequence function where we'll be doing deployments flow and some invariant. Then we we are connecting their chain uh uh envel or whatever you will be using and uh then we run the test uh in this way specifying sequences count flow count is everything uh understandable right now. Great.

So uh the most important thing probably that will help you to save a lot of lines of code is like designing uh the fast test. Uh so what uh like state variables you will need uh how you will represent them and so on. Uh so when you have some mapping you probably want to create some dictionary or some set or something like that. And uh for these uh uh state variables you will also need to hold the state in Python. So we can start uh so we will definitely need some token. That's correct.

And uh yeah uh we will need uh uh uh to represent the vault. Uh in this particular case, we probably don't need to use more WS for testing. But when you have more complex protocol, then you probably want more instances or if it is crosschain, you uh want to also have some more complex uh variables uh for representing it uh on several chains. Uh so in this case it will be W in then we can have uh some owner. Okay. Uh and uh this seems this could be everything we need.

Cool. So uh now we will go to the psequence. In persquence it is important to initialize all the variables that uh we have defined in the class. And uh it's again hinting me something but it seems it could be useful. So uh I will just replace this with instead of random account I will use their chain account zero. So it will be the first account uh default dictionary for token balances.

Um I don't have this thing. Ah yeah I have it. Cool. And we are deploying some mock. Uh this is how the pi types file looks like. And uh we can see it is this file in helpers. Okay. And if we over uh deploy function we can see the type hinting. So uh when you are typing deploy it helps helps you say what are the constructor parameters and uh then we have there minimal deposit amount. So yeah it can be

something like this for this purpose and the maximum deposit amount must be uh bigger than the minimal. So uh to make it uh sense for us. So uh yeah we can love it and there is deployment of the world. So if we will try to run it uh uh when you uh enter the project this already has existing uh v configuration file. So you can just call uh wake up pi types and it will create you that python

bindings uh which will be used for executing in the fast test and then we can call the fast test. So we can see it passed uh for that 100 flows uh because we are not doing anything but uh we can see that uh the deployment was successful. Great. Um I will place breakpoint here and uh I will show you probably one of the greatest things on uh like testing with Python is that uh you can during that

fuzzing you can place break points or se on several places and uh use it like for debugging. So uh uh now it is in pre-sequence. So it means that uh the fastest will always stop uh at the beginning of uh uh the fuzzing uh sequence. And uh if I'll execute it then I will get uh uh like a um interactive console where I can interact with the uh uh with the state uh of during the fuzzing which is great because uh you can for example uh create some fuzzing campaign which is running let's say you

will set it will stop after uh some time. So you will let it run let's say two hours then you will enter it and you can create there some entropy you can check the state or you can call some special functions and then you will just hit continue and uh uh it will finish since we had there uh just uh one sequence. So it was it finished immediately but uh uh if we have two sequences then it uh entered uh and in the second sequence it stopped here. Uh so when we are uh in the debugger we can uh explore uh what uh what we have like so uh what is cell

token we can see that uh uh there is uh this address we can explore uh what are uh the set values and uh we can call functions on the vault contract. from some reason there is some sometimes slow uh in I uh in debugger that time hinting but there there you can see everything what you can call on it so so we can call for example this thing and this

case we could for example create it will be like totally useful environment but uh we could check that uh the uh address what we have in the storage is same as uh the one which we are accessing with view function. Okay. So uh this is the breakpoint thing and now we can start preparing flow. Any questions until now?

Okay. So, so let's start with the deposit function. So, we have created wall. So, we we will just access it uh in the flow. But we will start with preparing some random input. So we will need uh to craft some amount which makes sense to do something like this but I will get to it later. So let's say we are sending some this amount and uh

we will need uh what's there we will need some tokens right so we will use a cheat code mint ERC20 where which uh takes some parameters and uh uh it's uh the contract uh ERC20 what we are going to mint then who will be the uh recipient and the amount. So we also need uh some recipient uh which will be the sender uh which will be the caller of the deposit function and um I don't want this

and we will need definitely to approve the token. Yeah. Uh each transaction has a special keyword uh from underscore which uh uh with which you are telling uh who is initiating the transaction. Okay. So this way we have a pre-execution phase ready and uh we can just uh uh call the deposit function. So we will go to self.world wa and uh we will call the deposit function. Here you can also see that uh nice type hinting.

So uh it's very nice um deposit and uh the amount and we want it from the sender. Okay, we can try to execute it. And uh we have some bug. So uh we were showing a break point before that uh we can uh uh when we have some bug it uh it's uh useful to look at the seat what we had and uh save it before uh we will be fixing some other bugs. So uh we will

use the seed with this parameter and this way we can ensure that we will always encounter the same bug at the same stage of the fuzzing. Uh so that's that's a good practice and the other thing is when uh this thing happens uh we can uh pass uh dashd which uh will automatically attach the debugger when it fails. So uh uh we can see that uh we are in uh uh this test case uh but uh the important thing uh I'm talking about debugger but it's still like python debugger it's not debugger in the solidity but uh it it still can help you like uh lot of with the state

exploration and some uh bug debugging. So uh we can see that uh this function failed uh and uh uh so we need to like somehow extract error from it and that's uh where the may uh handler is useful. So uh in this case uh like we can either let's let's do something like this.

If E value is not none, break point and uh this way we if it fails then uh the E value contains something and we can see that there is some exception wrapper. If you will check what is the value then we can see that uh uh we are triggering the below minimal deposit and if we go further we can go to the transaction itself and we can trigger the cold race. So in this case it's the the gold race is very

simple but uh uh I think this is one of the most powerful things for debugging because uh when uh you can easily catch errors classify them and uh also access their their call traces everything programmatically. Uh so it makes the testing like much more powerful and uh here we can see that uh it's this value and uh uh so we will just since we are in debugger we will check what is our minimal deposit amount. We can see is this. So uh uh we can see the problem is here.

So we will need to adjust uh uh not yeah and now comes like a very important decision uh do we want uh uh to generate the uh random amount from uh the uh where the lower bound will be the minimal deposit amount with which this uh bug will be mitigated or we want to uh like parse the error error uh and check it uh uh if the error is triggered.

Well, this this is like uh uh still uh this is dilemma what you will be encountering when you are writing fastest all the time and uh I think uh the best uh uh way how to uh do that is uh to somehow uh try to audit this little part and to unit test it. And think about it if you are 100% sure that you will not get any additional value from testing it uh on a reverting scenario then it will be better to uh generate in this case uh the numbers from the minimal deposit amount. If you think that you you are not 100% sure and uh you can benefit uh from uh getting

the reverting scenario uh then uh it's better to also check for the reverting scenario. Also what I'm usually doing that I'm playing with the probability. So uh I'm usually trying in some uh scenarios let's say revert just 10% and make 90% non-reverting because if you will not be handling these cases you the fuzzer will be wasting time in reverting scenarios and it won't be interesting. So in this case we will just generate uh uh the random in between the correct bounds because uh the contract is simple and we can uh verify uh the uh the out of bounds uh errors on ourselves pretty

confidently. Yeah. So now let's try to run it. And it finished. So we will just clean it a little bit. A little bit. Yeah. So we can also add some logging here. Let's say this. When when we run it, we can see that they are changing some addresses and uh they are depositing some amounts.

Okay. Everything clear? No questions. Okay. So, uh we have uh kind of some flow that is uh doing some deposit and uh if we will have uh more flows here, we can for example set weight. So initially all uh the flows have weight set to 100. So if I will have a one uh flow without this argument and one flow will have this that will mean that uh this flow will will be triggered and that 10 percentage of probability over

the other one. So uh this is also very important because uh you usually want to uh call some critical uh functions on the contract that will be like mostly used that those will have a big flow but uh you also want to call some setters from admin which will change some configuration value and so on but you don't want to change it uh too frequently because otherwise you will again will be in uh un uninterestic branches. is uh in your fuzzing. So uh you can uh have like let's say 200 2,000 for uh some flow and you will just take 5% probability that there will be

something else. So it is also very useful. The other useful thing for flows is that you can define some preconditions where uh if uh this uh lambda function is uh uh uh not correct or the statement what you have there there uh then uh the flow will not be triggered at all. So it will not be reducing the number of flows from the wall campaign. So that's also like a very important thing and very helpful. Yeah. So uh so we are executing something and now we can al for example write some invariant we are getting just less time

and uh what could we test um I don't know for example total supply so if something is deposited Um so we can track Yeah. Yeah. There is total deposits, right? So we can track uh uh the total deposits if they are correct correctly increasing.

So we have already some value for it, right? Uh no. So we will create it and uh I think we can also add these ones without this. Yep. So in the end of the transaction we are changing the state. Uh we are always using uh our values from Python. We are not using uh uh uh like uh view functions from contracts because we we always want to like work with our

expectations and not uh uh with the state. We are just comparing to the state if our expectations are correct and um uh so we are just taking the amount which is their used and uh there we can make assertion if uh our total deposits are same as uh the getter from the contract. Um yeah and we have to also initialize it here.

Yep. So at this time we have now one flow and invariant which are checking if total deposits are increased. So we can create another flow that is for withdraw. Um so what we have here some amount random in so uh we will be withdrawing some random amount from zero to the amount that the user actually has and we will call the withdraw and uh see what will happen.

Uhhuh. So here we can see that we are passing zero amount and it's not allowed. So we will just increase it for one. Uh like I think most of the bugs with fuzzing I was disco discovering just by writing the fast test. Since you are rewriting uh the solidity logic into that Python model, you will just verify a lot more things uh then you will verify just with a manual review. You are just checking like you are trying to hit like the 100% coverage. So you are trying to execute each piece of code uh with this fuzzing campaign. So we are

checking for everything and thanks to that you will just encounter uh some uh best scenario just by writing it and uh not just by running uh it for for a long time. Okay. So we can see we have there some another issue and uh that's because uh uh we are passing badly values here. So we have to have uh also some check where uh

where if there we pick some random account that doesn't have any funds so we will not be failing on this uh random integer generation. Uh so if self want something like this so this might work. Yeah. And now we can see that uh uh we hit some assertion error. And uh if we check it with debugger we can see that we failed on our invariant for total deposits.

So uh that's because we are not uh like updating the state for the total deposits. Once we will do it, we'll be good. Okay. So everyone after today should know what is flow and what is invariant. Uh does we accomplish that? Yeah.

Cool. So there is much more things uh what we could do uh and what I could show you but uh uh we are just losing time. One of the things uh is for example that uh if we if we if we would like to fork uh we don't need to uh like uh deploy their mock token but uh we can for example just use interface of it uh because uh it has transforming functions and so on.

So we will just use it as interface and we will pass there some real address um uh which is for example let's say USDC and uh when we have some others there uh it means we will need to uh like um fork the main end because uh on the address there is not nothing uh yet.

So we can just uh uh connect to some RPC let's say from public node and if I have connection then we can see that it's connecting to the RPC node so it is little bit slower then when it will establish the connection and uh you can see it's much slower because there is very shitty connection Sorry. But uh uh this is how it slows your fastest if you if you have a bad connection and you are forking. But now what we are doing is that instead of some uh mock uh we are just actually

testing this contract with with the real USDC and uh uh that's great because uh when um you have such codebase you can then for example uh create uh some uh uh list which will contain uh weird ERC tokens uh let's say USDT uh or others and you can test the code base with uh uh by changing all these tokens and uh if it uh breaks somewhere like uh it was very common that uh I've written some uh test cases and uh then I just replaced uh this address for USDT and uh it seemed that it was uh like immediately obvious that uh the codebase does not

support that even to that they were planning it. So uh this is very very useful thing. Okay. And uh what we could do more? Is there something what you would like to know? like we we can proceed with uh uh like enhancing more this uh this fastest or I can show for example some solution on a problem what uh you will have well um we

can for example check this solution and uh lookuh what's what's there so uh nice thing what we were mentioning is checking events. Uh it's like um it doesn't seem uh much important from the first place but it can uh save a lot of things. Uh so we can uh like access the transaction events and then we are checking uh uh the events if uh uh they are triggered and with correct values.

So it can be seen here and uh there it is for the other flow and yeah uh as I said you usually want to uh create flow per each function. So the next steps will be to create uh emergency withdraw flow. The other uh will be setter for these deposit limits.

And the other thing uh will be that we can perform just some transfer uh which we called like increase the entropy and uh create such environments. Okay. So if there are not any questions, so I think uh that's it. And uh I hope I uh somehow showed you uh how you can like fastest in Python and what are the like main benefits of it. Thank you.