Fuzzing and Punching by Clément Lakhal

Transcript

Hi everyone. So I think maybe we can we can start now. We're ready to start. Okay, perfect. So hi everyone. I hope you're having a a great time today at the conference. Uh yeah, that you're enjoying Dubai. My name is Clement and I'm a solidity developer and today we are we are going to be talking about how we can uh punch contracts in the face, how we can break contracts, how we can test them and maybe if we are lucky how we can uh catch some bugs. So we are going to start with the the basics and when we are thinking

about smart contract development I guess there is like this picture uh this flowart flowchart that we can picture in our head uh I think most of the good developers have this flowchart coming all this uh life cycle for a smart contract obviously you can skip some steps but uh only if you're pure for DJ and you want to to ship united untested code which I wouldn't recommend to be honest but yeah basically the idea is that first you will be specking your smart contract you will be yeah just defining the purpose of the contract once you are ready with the specs then

you can start the development phase and then during the development phase you can be right at the start or once you reach like a specific milestone. When you're happy with the state of your contract, you can start writing tests. Maybe it can be by the end of the development cycle. Everyone has different thoughts about it, but anyway, you should be testing what you are doing. At some point, maybe no one is perfect. Bugs can be in there.

Obviously, if you found bugs, you will want to fix them. So, you can go back to the development phase. you will fix the bugs. Testing again and at some point you will be ready and then you can deploy or at least go to the auditing phase which should be what most people should do. But yeah, this is more or less what's happening. But you will tell me, okay, we have maybe bugs. How do we know if we have bugs? Uh how we can test our contracts? So that's obviously the main topic here. And when it comes to smart contract development and testing, um there are a few different techniques

that we can use. Uh some of them are very basic, some of them are a bit more complex and we're going to talk about them right now. So this is a very simple contract. This is just like I'm sure many of you are familiar with it. When you're making a new foundry project, this is like the example contract. It's very simple. You're just like a you have like one variable called number to store a value and then you have increment. So it can be used to increment the value and then set number. So you can just set the number. So very very simple. How can we test this contracts? Well, the first thing that you can use, the first

technique is unit testing. So basically unit testing is very very simple. You have inputs that you will provide to the test and there is like outputs and outcome that you are expecting. So in that case we want to test the increment function. What are we going to do? We're just going to call it. It's very simple. And we know that the increment function will increment the value. it will just like add one to the value. In that case, we start with zero. So if we increment our number, it will be one. So unit tests are very basic. They're very

simple. They don't do much. uh they just do actually the scope you are giving to them and they are not going to be like um trying to go f further and also the state is not persistent which mean that if you run this test it will be always the the same state at the end because it's the same state at the start which is why in that case the value will be always one because we start from zero. Then we have something called stateless fuzzing also called just like fuzzing in a more uh general way. So in that case fuzz tests

are a bit more advanced they are a bit more complex because they will also start with some inputs but in that case they are not expecting inputs from you. they are just going to use random values. So what's happening is that here we want to try we want to test our other function called set number. Set number requires an input uh you 256 in that case and well we want to try out this function. We want to know if by providing a variable to set number it will store it in the storage. Obviously there is like a huge amount of u

different um variables that we can pass to this function. So we are not going to do it by hand otherwise it will take us like 10 years but we are just going to give the uh the possibility here in this case to foundry to use like a random value. So it would be x and foundry will know that uh it can just like generate a specific value for x can be 0 1 2 3 or whatever like a huge number and it will feed it to the set number function. So a few things that we have to keep in mind here. I've been talking about random values. It's true but it's more like uh something you still have control over because you can still say I want random

values but I don't want zero. I want random values but I want above I don't know like 1 million something like that you still have um control over that and even if this test would be running many times this is what what is called runs in foundry or first testing in general maybe you can set up like 32 times so this test will be running 32 times but the state won't be persistent every time will start from the base uh state so what you defined in the the the setup function and then we have stateless fuzzing aka invariant testing. So in that case things are a bit different

because we are not providing exactly an input we are not providing exactly an output or an outcome but rather we want to test we want to check an invariant. So I guess the next question is what is an invariant? Well, if we just take the definition invariant is a property which remains unchanged when a specified transformation is applied. So this might be a bit obscure but basically just that you have a statement and this statement must be true must stay true even if many things are going to happen.

Um so what is going to happen from like a technical point of view is that you will write your invariant. So for example this is true and it should be always true and then in our case foundry for example just going to fuzz the hell out of the contract. is just going to call a bunch of random functions and the goal Foundry its goal would be just to break the invariant by trying many different path many different function calls to see if there's a way to break this invariance. So I'm just going to give you like some real world examples of invariance. So something very simple with the for example ERC20 tokens the

total supply must be the sum of all holder balances. So if we have a thousand holders and they have all one token then the total supply is maybe 1,000. This is something very simple but this is an invariant. Obviously in that case this seems obvious simple and it's quite easy to to test but we can go to more complex things. If we take the case of the wrapped ETH contract, then we have the E balance held in the wrapped ETH contract must be equal or greater than the total supply of wrapped ETH.

This means that we can be sure that if we have wrapped ETH, we can always redeem, we can always unwrap and get some ETH. So this is for the solvability of wrapped E. Once again, this is invariant that is quite uh straightforward to test but still quite interesting to set up. And then we have XY equals K. So you know this for example from uh unis swap pools. Uh in that case they are using that in their pools for the swaps to validate uh if a swap can be done or not. And just that the reserve of token tokens X uh times the reserve of token Y equals their

invariant. So we can sum up this uh this way. Basically we have unit and first test. So they are kind of similar in a sense and they're very good to test very specific things. For example, you are going to write some functions for a library that is computing the square roots of something and you know that for example you have a bunch of values uh that will you will feed on one end and on the other hand you know that the outcome the the return values should be this a b c d etc. So unit test first test they are very good at testing very specific things and invariant test they are more used for global global testing

let's say you have like a brand new lending protocol or you have like a an AMM so you might want to use uh invariant testing to check if some properties would be always true uh can be also done for something simpler like as I said like 20 but this is the idea here in that case you might want to use invariant testing and yeah for for example you have a new AMM and you want to be sure that if a 100 users come and they are going to buy sell by sell many many many times well things will be fine in your contract nothing is going to break uh I don't know like uh some value is not going to overflow or someone is

not going to be able to redeem the liquidities they provided things like that and you can do that with invariant testing with the other tests like unit test it would be way more complicated to to simulate specific scenarios like this. So now you might be wondering okay this is cool but how can I actually like write invariant tests? Well, the good news is that we have many many tools. Uh we have the amazing Akidna. So the logo is not very good quality, but uh the tool is very powerful. It's from Trail of Bits. They made this tool like long time ago, maybe more than five years I

think. So it's kind of an OG tool is using Slicer. Another tool they made uh to do like static analysis and Akina is very powerful. They use it for most of their audits and they've been able to use it to find bugs in protocols uh such as balancer, unis swap, even in die. So very very powerful tool. You might the the learning curve might be a bit steep but a very nice tool. Uh trade of bits also released meditusa so I haven't tried it yet but basically I have some notes here it's written in go based on guess you can do parallel fuzzing smart mutational value generation whatever

that is I'm not sure to be honest onchain fuzzing enhanced debugging capabilities so looks very good I haven't tried it yet but uh it's on my to-do list then you have halmos which is quite new also maybe it was released I think like one one year ago. Um in this case this one is more for formal verification and it's also using like a symbol testing testing sorry. So in that case symbol testing is a bit different because it will use like math formulas. So the goal is more that you have your invariant that can be translated to math basically to formula

and then what's happening is that using that formula you can convert it back to code and find a way to break your contract through all this mathematical uh representation of your invariant and your protocol. And then obviously I feel like the most obvious obvious uh choice nowadays is Foundry because Foundry has so many things so many tools. They have like a very powerful testing framework uh which is also including what the invariant testing it used to be good but a bit basic but now they have very good features. I don't know if uh if it's like as good as a kidnap, but they have many things super interesting like uh

shrunk replay stuff like that. Basically, once the foundry has found a way to break your contract, it will try to uh shrink the whole pass. So, how many functions were called to break your contract? Maybe at first it was 100 then foundry will replay this attack. this not attack but replace this scenario and it will try to find the shortest way to break your contract. So it's extremely powerful to debug stuff. So let's get into business I should say and we are going to see how we can actually set up some invariant tests uh on a more like low level. So

basically we have like three different types that we can use. Uh the first one is the direct assertion. So it's very basic. You just have I don't like two values in your protocol and you want to compare them. You want to be sure that this assertion this invariant is always true. For example, in that case we have the token the total assets should be greater than or equal to the total supply. So you can just write this and then Foundry will like call a bunch of functions from your contract and at the end maybe it will call like a thousand functions and at the end it will check if this is still holding. So very simple. The second one is the ghost

variable uh technique. So I like that one because that one basically is having a way to track um variables in your test files in your test contracts and then you will compare them to the actual values inside of your protocol. So a very simple example would be we are going to do like a transfers in an ERC20 contract. So what's going to happen is that obviously the balance of a holder will be increasing or decreasing in the ERC20 to um token contract but I can also track that in my test contract and what's happening is that I will have two values uh one that

will be in the test file and the other one in the actual token contract and at the end of the the run then this invariant can also be checked. The last one is deoptimizing. Uh this one is pretty fun actually. Let's say you wrote like a very very very good and very gas efficient very performant very optimized algorithm that you are using to distribute rewards in real time to many users like I don't know for yield farming or something like that.

How can you be sure that uh well your formula is correct or how can you be sure that your implementation is correct? Then you can do the naive implementation technique. Basically you're going to write a very very simple very naive implementation of your code inside of your test file. You don't care about gas. You don't care about anything. You can use for loops, whatever you want because you're just like testing. So it's completely fine. And what's going to happen in that case is that you will be um comparing once again the value that is uh that was computed inside of your test file

against the value that was computed inside of your protocol. So this can be used let's say for example you have like an AMM and people are like providing liquidity and you want to be sure that they are earning fees and they should be earning fees this way. So maybe you have a very optimized um algorithm inside of the pool contract. But in your test you can just for example have a very naive implementation that will use like mappings uh for loop stuff like that. So it's like a interesting way to compare things. So an example. So this one is taken taken from the wrapped e uh example invariant testing repo from

horse facts. And basically we have like here two very simple invariants. So the first one is conservation of e. So basically we are just checking that uh the supply of e is just like equals to the total supply inside of the wrapped e contract and uh there is another one that is using the what I've been talking about the ghost variables in in that case we are just checking that uh the balance of e is like the sum of what was deposited by the users minus what was withdrawn by the users. So you can see like invariants are quite simple. They are

just like things that you want to check in your contract to be sure that the contract is working the expected way. But the hardest part is now and is how can we check the actual invariant like how do I need to write like how can I write a test for this? Well, you have this first uh how can I say it's not even a technique. It's just like the the open open technique open way. You just say okay I have this invariance and then foundry you just figure it out. The likeliness to find something here like a bug I mean is very low because foundry is just going to call random contracts. It's just going to

call random functions with random values and you won't be sure exactly what it's doing under the hood. I mean, you can still track it, but you cannot really control it. And the thing is that Foundry doesn't know like uh if you want to swap something. It doesn't know like exactly which uh I don't like amounts should provide. It doesn't have like maybe the right token balance, things like that. So we have to go uh to we have to use sorry another method called the the handler method. So basically it's very simple we are using like three elements the handler the targets and the invariance. So you guys

already know what is the what are the invariance for example as I said total supply of wrapped east or total supply of a token and then we have the targets. Obviously this is very basic. This is very simple targets or just our contracts. We want to try our protocol. So we are going to say to Foundry for example, you are only going to call our contract. This is like the the ERC20 contract or whatever or our AMM. And we can also go a bit further and we can say you want only to call this function or that

function. And then we have the handler. And the handler basically is kind of a helper contracts. We're going to see that in a minute. And the helper the handler sorry is helping foundry uh into like calling the actual functions. As I said, if you are writing making like a brand new AMM, then Foundry doesn't know how to make like a swap. So you will have to help him out here. So what is going to happen in that case using the handler method that we have foundry and foundry is going to call the method here uh from our handler and what is what are we going to

do is that we are going to say the targets are actually the functions from the handlers. So Foundry is going to call these functions and then our handler is going to call the different uh contracts from our protocol. So it's going to do that thousand time for example and then it's just going to check the invariant at the end. So from like a very uh code perspective, how does it look? Well, you can have for example this handler. This is also taken from the horsefax uh repo.

you have some cost variables. For example, you want to be sure that when people are depositing or withdrawing, you want to track how much uh was deposited or withdrawn from um the wrapped ETH contract. So, you can have like ghost variables there. Um, and what is going to happen also in that case, it's a it's a bit advanced topic, but you want to be sure that the colors are not like all the time the same address because it wouldn't make sense. For example, if you're making an AMM, you know that many many wallets will interact with your contract. So, what is going to happen

here? We are going to use like a setup that is doing like uh what is called actors and basically we can create uh or use actors. Let's say for example uh Foundry is going to make a new call then we can say okay for this new call you want to call the deposit function. Okay. So let's create a new actor. Maybe we we are going to give to that actor like one east and then this address can deposit or try to deposit one east. But it's also the case for other functions. For example, we can say okay we want to withdraw. So you have like different scenarios. For example, you can say this

address wants to withdraw. Maybe it's an old uh actor. So he's already holding wrapped e. So he will be able to withdraw but maybe he's not holding anything. So maybe we want to check that that if an address that is not holding wrapped is he able to withdraw uh e at the end it's it's not possible. So this is also interesting because with invariant testing we have different ways to set up things and you can have invarants that should fail or at least revert or some that should not revert.

For example, if you're making uh an AMM or even in that case do you want to make the test fail if someone is trying to withdraw more money that they deposited? Maybe not. But you just want to be sure that at least the transaction is reverting but the test is passing and then these are the actions that are defined into our handler. So basically you have the for example deposit function. So this is also a function that is in the wrapped e contract or at least an action that you are supposed to do. And what is going to happen here is that you can see like the

you win 256 amount. So we can also provide like a random amount here obviously and you can see like the create actor modifier. So this means that we want to create a new actor and we want to deal we want to give him some ETH so they can deposit some ETH inside of the contract. And for example we are also like uh increasing the ghost variable deposit sum. So in the handler everything will be like all the logic you want to check like the ghost variables the actors management will happen in this contract. And if you look at the withdraw function uh what's going

to happen here that we still have the random amount parameter but we also have the actor seed. It's just a way for us to say okay let's just use like a random uh actor from all the actors we created and then once again we're just like updating the ghost variables and obviously we are calling the actual wrapped e contract in deposit we were depositing and then in withdraw we are withdrawing. So once you have the handler done then you can go back to the the setup. So I showed you this earlier.

You have like in the actual test file you have like your invariance but then you might want to update at some point uh sorry it's here. You might want at some point to update the the whole like test file. And in this case we just want to say okay so in the setup function so we are starting with like a clean state. We deploy the new wrapped contract. We deploy the new handler. We give the address of the wrapped contract to our handler. And then what is going to happen is that we have this weird array of bytes for uh it's like function selectors.

And this is telling Foundry we want to call all these functions from our handler contracts. And at the end the last two functions are actually quite interesting. You have like target selector and this is well what I just described. We say only call this functions from the handler and then we can add the target contract and say only call this contract. If you don't do this, maybe there's a chance that Foundry will try to call another random contract just to see if the invariant is still standing. Obviously, will be standing because your contract won't be interact with. And so, this is another screenshot

from another project I'm working on. But basically, once you've been like running your invariant, you can see like we have the here number of runs. So we call the fun the invariant test call invariant uh tracks total token sold and proceeds etc whatever it's called. We have 32 runs which means that the whole thing was uh has been running like 32 times and there is like uh 16 uh 16k calls. So this means like 16k functions were call randomly and there is the also the amount of function calls that reverted. In that case we don't

really care about uh reverting because what we wanted to check was uh actually like um that the in that case that we were tracking properly the amount of tokens that were sold. So someone buying, we don't really care if it's failing. Um, and you can see like this report generated by Foundry. You can track the amount of calls that were successful, that reverted, etc. So yeah, I guess this was uh the introduction, but if you want to go further, I recommend you like a few different uh very interesting resources.

the there is this uh article by rare skills pretty pretty interesting then obviously you can check the go Patrick Collins uh he made like many many videos on YouTube about testing invariant testing many interestings like that another goat horse facts uh he made a cool presentation at ETC so the video is on YouTube you can check it out and then there is also that one video from trail of bits it's actually not one video but I think 10. So you have like 10 hours of uh tutorial how to use a kidna. It's pretty dense but very very interesting.

Good examples uh Sablier V2 flow Doppler my own contract and unis swap before uh akidna wrote I mean the trade of bits used akidina to write very good um invariant test for v4 and something very very interesting that you can also read uh this is the list of all the bugs that were caught using akidna so this is this list is obviously by trail of bits but they they have like uh 20 reports and uh yeah you can read out read this out it's uh it's very interesting and this was the last slide I forgot to make a slide to thank you guys for listening to this talk but I'm

done thank you very [Applause] much any questions all right thank you guys have a good