AI Fuzzing: From Stochastic Chaos to Directed Precision.

Transcript

To get back to that. Ah okay. Hello, hello. Yep, yep. I've been this good. Yeah, I've been like thanks for coming and that we actually have water here. I will start from like some introduction of myself. So, the talk is about like the AI fuzzing and I'll quickly go through like what is fuzzing and all these like AI things. Like the major things is that it's not we are not applying the from the perspective of like really building a fuzzer rather than like using the existing fuzzer. So, that's the things that I will go go across and yeah, let's start with some introduction of myself. So, I I'm I'm I'm Andy, the founder of Memoria. So, the whole Memoria is basically dedicating for building a automatic fuzzer for for band pen testing and the automatic value extraction. And my

background basically I start from like a teaching in the academia that teaching in economics on automatization apply econometric and I moved to machine learning research. It's still in economics doing all those kind of random forest classification and I moved to the crypto on the AMM mechanism design. The major things why I'm here that like why we are here that we care about security. Security is the bottleneck of like making people why not coming into the crypto space. So, the next things is like oh, this talk is actually like we do collaborate with University of Tokyo and this is like a working paper that I would like to present and of course like the the people a lot of extension. This is like brief level of the idea that I want to talk about. Yeah, and yeah, first of all fuzzing and AI. So, should

we fuzz everything or should we just like use AI LM to to replace the whole security thing. So, in the end of course the answer is like mix of both. But how do we mix of both? For this talk is more about like how do we use the AI to help the fuzzer? At the end of the talk I will share just like maybe two two two cent of like how a fuzzer can actually help AI on security as well. So, before the before actually understand what is fuzzing, we need to understand what kind of things result in the value extraction the exploit. What is the nature of exploit? In the end it just like functions and parameter. There's like a sequence of function. You do like add collateral, liquidate, remove and then you run some input parameter. So, functions and parameter equal to value

extraction and take the money out. So, this is like the fundamental of why we are doing like security pen testing. So, what's the problem? Why is this quite easy but what's the problem? How do we actually get into this kind of solution? The major problem is that like for every single function there's like so many path that we can try first. That is like oh, this is like amount out amount in to data and then all these is actually having so many different revert path. All of all these like stochastic path, we need to find a way that actually result the target execution. So, this is the major problem of a fuzzer and then the major problem that we are we are encountering. So, for instance like now of course like well, we you we swap through fund and and then we have like

if I ask you that like you need to input this parameter, how do you do it? I mean asking me I I also don't know how to do it. So, that's the problem of like why is like getting into function getting to the target branch is quite difficult. So, how do then we actually understand deeper of like why a fuzzer is not sufficient these day is because like this is a this is a one of the one of the one of the hack that back in 2024 44 mil got hacked and the first function that is about like create lock campaign. Within this function you can see like these are the branch that we can actually get in. They have like 300 and 300 538 branches and all of these branches that we need to get into the

valid branch. This is like quite difficult. For a fuzzer they usually just like doing random mutation. They do by type but still to get into the particular branch it's just tough. So, this is like the stochastic nature. The next thing is about like the okay, am I good? I should be good. Yeah. [laughter] The next thing is about like the the external call and then like in our our smart contract system comparing to traditional security, the major things that we have is that we are interact with like external call so many different address even single protocol. They would design their mechanism in a way that having so many different address. So, last last year we got the balancer hack. Balancer they design in a way that the full contract they will keep the money. They external call for the for the pool contract for

calculation. So, in that sense like for the automatic fuzzing or automatic checking, we need to we need to able we are we have to be able to like identify the relationship between the the the the the core contract and then the poorly external contract. This is another things that is complexity. So, it's like the batch swap they have like talk is so many different input parameter that we need to do and then within the batch swap we need to find what kind of input we should play around in order to in order to change the parameter for the external call which is like another really sophistication that here. So, the third things that is quite similar to the balancer case but sometimes when we

implement the smart contract we take like really weird input parameter that So, amount those kind of things is actually quite trivial. The difficult one is thinking about like the byte type core data. You need to construct a core data to call a function. This is crazy. And and then this is like the things that a fuzzer or like a normal human or like the standard technique that is not really able to get into the this kind of function call. It's just tough. Yeah. So, we yeah, so this is the batch swap similar that they have like the dynamic type dynamic tuple. Those kind of things is also like having um having messed up the storage slot management as well. So, then it's actually thinking about the solution then I I try to present some solution to propose the solution here. So, we will

talk about from stochastic inefficiency. So, this is like the so many 500 branch and then we can actually take out for the revert unrelated branch and then we reduce the path that we have. So, from that many to less and all of these many then how do we we need to build a fuzzer to actually get into the path that what we want. So, this is like from my my thesis about like the stochastic chaos to directed precision that we just need to get into particular branch to yield some potential exploit. So, this is like how do we do it? But how? Yeah.

How do we do it? So, the way that we design the the mechanism is that we do like three layer. The first layer is actually thinking about the AI LM which is our core talk about here that I will dive in each each of it. The second layer is like using the fuzzer the fundamental fuzzer that basically building to get into the end of the branch and then we also build something called basically GPU execution the EVM execution on the GPU but which is not about the talk here. So, dive into the LM component. So, we have like the taint analysis, external call compressed data type and more important is actually thinking about like the revert type. So, how when we actually do an execution we encounter a revert, how do we how do we actually use the LM to guide us for the

next step? So, the first part is actually taint and and taint analysis. It's quite a lot of people doing it. It's basically the static analyzer trial of this build slither that back in time before the LM people used like static analyzer to know where is the vulnerability. So, in the end it's like some architecture that you play around and then you you build chunking semantic semantic code segment for the LM to understand better. For instance like you can group build a graph on the system architecture that like the the the core contract that is related to different maybe interface related to event, maybe to to other items to teach the LM to understand the relationship across the contract cuz otherwise like if we just put in to the

source code to the LM it's just tough to to understand better. So, this is like one of the things that we need and when we actually thinking about the result. So, this is like one of the very famous other very liquid flow and the things that what we are wanting to actually get is that we want to see the line of code what where is the problem and then the potential problem. So, uh make sure that here I I want to emphasize that I don't mind about like the false positive one. Even though this may not be the critical but because like we are applying some fuzzer. So, after we identify the core region that we should investigate, we can use a fuzzer to investigate to validate whether it's true or not. So, that's why it's like having these to guide where we should go is a good way

even if it's false positive we should still search it. Yeah, even if it's a suspected problem. The part B, let me see. The part B, yeah. Good. So, the part B is actually thinking about the external call. The question to be asked is that the caller and callee relationship. So, how do we manipulate the caller to get into change the external call relationship? So, for instance like there's a lot of things about price manipulation. The balancer case is actually an external call manipulation that you need to call from the full contract which is like the the caller will call the full contract but like how what they manipulate? They actually manipulating the external call through the external call.

So, that's why it's like we need to link between the caller and callee concept. And how do we do it? Pretty much like we can also ask the LM to actually link it between these two. We just need to build a better call trace in the execution to know it all. By understanding the call trace then we can actually let the LM get the dependency graph between the caller and callee. Yeah. So, so basically like there's like some approach that getting the caller function, getting callee function and then link between like the address program and then trying to generate some callee input flow for the LM. Basically we ask them to do this instruction then they will suggest that oh, so this is like Uni V4 contract that

they the money money V2 got exploit and then the V4 contract they external call to the money V2 then they would identify that like the caller contract is a swap function and then what is the external call function is that the before small, but how do we manipulate it? They will suggest that where is the quality input to to be able to get into the branch. Yep. The third thing is that we are thinking about is that like oh, we were talking about like call data type, those kind of like super weird type. We can actually like let the LM to understand better. So, Moon hacker actually like is another case that got hacked. And as long as like the hack is that like as long as you put a call data that after you put the after they decode abi.decode, the whole

call data to be M token, then the M token is the attacker address, then you get the validation. But the major major blockage here is that the call data is just difficult to construct to actually exploit. So, we can actually ask the LM to reverse engineering that oh, because like understanding this logic, they are doing decoding. That means that oh, maybe we can do some encoding to reverse engineer the the call data, too. To actually like generate some potential searching opportunity. So, this is like some other things. The major the final major things that we do it more dynamically to interact with the fuzzer is that the revert-based AI fuzzer thing. So, this is like the the whole engine that we can see that like the

fuzzing engine that they are doing execution, then they when we do the EVM execution to encounter revert. And for the revert, we need to see the the whole execution trace, and then we convert this execution trace to let the LM understand that oh, where do we have the revert problem? And once we know the revert problem, we can actually ask the LM to do the part B and C or suggest for better result. Because like I know this is the revert message. Maybe they are talking about oh, your balance is sufficient insufficient. Your balance is greater than greater have to be greater than 100. Greater than amount of input.

Of course, like we know it as a human. But for a fuzzer or like a program analysis, it's just tough. So, that's why it's like for the semantic things, we can actually pass it to the LM to do so. So, one thing is that like I encountered back in time like I was like getting the execute execution trace, then if we actually like getting the raw trace from the EVM or using the M field debug trace that we will get something like this that they will tell you the stack, they will tell you the op code and the program counter all these things. Yeah, this is like not a chance. I don't even understand it how the LM understand it. The LM don't understand this way. So, what we need to do is actually convert that call trace into some more semantic trace that having all

that we have like the the depth on this contract and then the call on these and then the whole call trace logic to let the LM understand better what happened. And once we have it, we can actually ask the LM to generate oh, what is the error message? We see the error message here. And then because of the error message, they will suggest what to do next. So, this is like basically the whole from revert and then and then like we query the whole engine because like the fuzzing process, it doesn't work, we call the revert, we let the LM to actually help us to go to the next phase. One interesting thing is that what we implement is that like after the revert, there sometimes it's not about your parameter input is wrong.

Cuz like sometimes it's about you haven't called some other functions first. So, they may actually able to understand that oh, so so for instance like this is a swap function. Um if I directly call a unit swap we for bunny hop V2 external call with new hop V2 function that swap function, they will actually need to call some other function first. So, you may you may have to do initialize the contract, you need to modify the liquidity to actually do something first. So, that's why it's like the LM can able to suggest these kind of things. Yeah, to help the fuzzer. So, in the end, it's just like how do we get into end of the branch?

And then the fuzzer itself like how do we do it? Do we really understand the S low S low operation to to understand like each This is like our control flow graph. Yeah, how I never mentioned. This is like the whole control flow graph. And then for the basic block, we need to get into some particular op code. We understand the jump I and then we understand the jump I operation to get into a deeper branch. So, this is like I think the time is quite good. And some Yeah, so basically I will slow down a little bit. The discussion today I talk about like what's the problem of the challenge of the fuzzer. And then how the AI actually can help. And then I will talk about different component that we can help. And the major things that I want to like just quickly share about it

is that how the fuzzer can actually help AI. We were talking about AI is like is a is really really involving for security. But how do we use the fuzzer to help the AI? In the end, what I see a a fuzzer or like AI AI that they are lacking is actually these three things that a lot of people emphasize about data, but I personally think data is like the is important, but like comparing to the context that we should give the prompt and then the context, that's is like the most important one. We also need to use the tooling. Now it's like we are entering in an era of like the agentic era, which is having reasoning instead of pure one-shot engineering. So, we need to give them the tool to have the reasoning, collect the information. You can see like the

agent they they will grab it, they will call different tool get the information and that help the AI to actually do it. So, yeah, I mean I hopefully should be good on time. And yeah, I think thank you. Thank you very much. May the force be with you. Right now, we can take some questions. Does anybody have Oh, I'm I'm interested. I have 2 minutes. I'm on time. Yeah, you are on time. We have 5 minutes left. I thought I talked I talked a lot. No, no, you are good. Yeah yeah.

So, thank you. Yeah, thank you. Thank you. Yeah, I talked really fast. Thank you. Thank you. Now, I should go slower. Thank you. No. You're good.