Leveraging AI for Automated Vulnerability Detection in Zero-Knowledge Proof Systems

Transcript

[Music] One, two, and two. All right. Hi everyone. I think we can get started now. Thanks for coming and uh see me talking about AIS. I wish I had an AI come to here today to talk about it, but we're not here yet. The slide was made by an AI though, so uh we we have that. If you don't know me, I'm CryptoDavidw on Twitter. If you're not following me, you should follow me so I can get more followers. Uh I wrote I wrote this book called Real World Cryptography. You should also buy it so I can pay for my rent. And these days I work at ZK security uh where we do u audits and dev and research. And so if you're

interested in these kind of things for ZK and MPC, FHE uh these days TES and stuff uh then um reach out to us and we'll we'll work with you. Cool. Anyway, I don't have too much time, but I wanted to put things in perspective. So, we started the company two years ago, almost two years ago. Um, and then it was it was almost when Chad GPT was released, the first version. So, we were starting to see AIS and we were like, okay, these AIs are they seem pretty useful, but but not yet. They're they're kind of hallucinating a lot. Um, we're not really using them yet. Uh, and so we we're not thinking too much about it. Fast forward to today, a lot have happened has happened. So 01 came out.

Uh 03 and 04 came out. We all know that AI has made crazy progresses. Um if you're dev or if you're doing research or if you're an auditor, you're probably using AI in your day-to-day job. If you're not, you're probably missing out. And uh today we can said we can say that AI has uh impacted the way we worked uh a lot. And so yeah, I started thinking, "Oh my god, I we created this company two years ago. Are we going to get replaced?" And thankfully, there's this guy that came in a leather jacket and he said, "It's not AI that's going to take your job.

It's actually the person who uses AI that's going to take your job." So that that sounded comforting. I was like, "All right, I just got to use AI." And so uh yeah and so I I started working on this tool and the first target that I used it used the tool for or I started experimenting with was circum. I don't know if you can read that but this is what circum looks like. It's a programming language for zk circuits. And so I thought that was a good target because we're auditing a lot of these people are using that to write zero knowledge proofs. uh a lot of that on Ethereum. Actually, most of the Ethereum zero edge proof application use circum

on the L1 at least that we know of. And so we we were uh thinking these kind of things are pretty low level. It's pretty hard. There's a lot of cryptography involved in what what is written in these circuits. So if we can do a good job there, it's pretty easy to generalize. So that's how we started. Uh just want to mention that code came out uh the last version of code came out with this report and they they sort of uh showed that web web apps are pretty easy to solve let's say like if you have like challenges and bugs in web apps but for crypto it's not it's not that easy. So cryptography is still it's kind of tricky to find bugs in cryptography.

So anyway I started in a very naive way. I put I wrote this prompt. This is actually the first prompt I used. I I managed to find it. So I I started you know you are one of the best security consultant on planet earth like like it was kind of a naive I actually we still don't know if this kind of prompt works better but not really some people say you shouldn't be uh polite and you should be kind of aggressive with the with the the the agent or the eye and so I I don't know people are doing different type of things but anyway this is how we started. Um, if you don't know, all all of these colors are the tokens. So, tokens are the atom of AIS.

Basically, you're paying per token. That's the currency. So, the more tokens, the more you're paying and the output usually, uh, is even more expensive. So, the tokens that the AI generates, uh, you pay for that. So, I started very naively. I put everything in the prompt, right? I put that I put that I put all the code. I said, "Okay, look for a bug." And this work kind of okay a lot of the times uh the prompt. So these are the numbers today for the how many tokens you can have in the prompt and the output.

I just uh looked at the state-of-the-art like closed source models but everybody has at least 200k tokens these days. So it's it's a lot of tokens that you can put in there. Circumcircuits tend to be relatively small compared to bigger applications. Um and then G Gemini and Google are really on the forefront there with like a model that has 2 million tokens. So they really have u large context windows but so um in our so sometimes we would reach the limit of the context window and that was that wasn't so great especially for larger projects when we started generalizing that to more than circum so um when we started using large context window we sort of solved that

sometimes sometimes we still reach the limits but that solved a lot of the the issues as models got better but we also realized that the more stuff is in context the harder um not the harder but the more you put in context the worse the results like basically there was too much stuff and I I guess you know it's always hard to understand exactly how these agents work but the idea is that if there's too much stuff it's hard for the to focus on what's important so a lot of the work that we've done like very early on we realized that and a lot of the work that that was done was to try and remove everything that was not uh relevant and we're a lot of the work is May I would say maybe 50% of the work is trying to remove what's what's not

relevant and also um if you remove too much we I'll talk more about that later but um you also don't get good results so part of the work was also finding out what's do you have enough relevant stuff uh and this could be enough relevant code do you have enough implementations of dependencies and so on do you have enough documentation do you have does the language model understands circum and I'll talk more about that later. Uh but yeah, that a lot of the work is is doing that and and then at the end you're like, okay, is this enough to find bugs, right? Like the the we're doing all these things. We're optimizing things like is this going to be good enough for us to find bugs? That's the big

questions we're we're going to try to answer. So, okay, so we have a codebase and the these are the problems we're trying to solve. If we don't want to put all of the stuff in the prompt, what what do we do? The first thing we thought about was fine-tuning. So you're taking a model and you continue to train it and you train it on the codebase that you have. The problem with finetuning is that you have to do that every time you start with a new model. If you want to test different models, you have to do that with different models. It's costly. Uh you need a lot of data to make it work.

U so it doesn't work well all of the time. Uh and when we were working on this tool, there's new models that would come up all the time. So so to us, it didn't really make sense to fine-tune anything. We just wanted to use the best models uh figure out the low hanging fruits figure out how we can optimize as much as we can and at the end once we have a good product maybe we can um or or something that's very useful maybe we can look at fine-tuning to even further improve the results. Okay, so now finetuning the next thing we looked at is to uh to use kind of a rag approach.

Uh if you don't know rag that's um retrieval augmented generation and so usually that means you you're storing a bunch of your documents in a vector database and then you're doing uh this kind of simulator similarity searches. So this is converting everything into vectors and figuring out which vectors are closer to each other. I'm not going to talk about how it works, but the idea would be to take the codebase and create like chunks of of the different functions and and constants and like whatever in the codebase and then store that in a vector database and then we can query that that vector database uh when we're trying to analyze one function or something like that.

So uh this didn't give us I mean we we sort of started with that and then we realized wait we actually have all the information we need like if we want to to audit like um like a template we can already run circum we can compile it and we can sort of get the dependency graph and get all the all the implementations that we want. So quickly I realized that I didn't really need rag or at least it wasn't super useful at this point and I quickly moved to static analysis and fork circum and uh generate a dep dependency graph and and got all of that in a in a tool.

So uh by the way rack is still useful. We're just not using it at this point. But if you think of of applications like deep wiki uh that we use a lot that that allows you to ask questions to a codebase I believe it all works thanks to rag. So this is still something that we might use later but uh yeah anyway I'll talk about rag later as well but not in the context of uh the code itself. Finally uh especially when we didn't look at circum we needed a way to to let an agent sort of roam around in the codebase and it's pretty easy. You just need to give it give it like you know cats ls crap these kind of tools and then you have an agent that can just browse the codebase.

Uh oh yeah what does it look like? I guess you give some instructions uh and some user prompts like you know look at this function in circum analyze it look for bugs and what the agent is just a loop it just requests uh some function then we produce the result and we give it to the agent and then it might ask another function and then we give the result right so these functions are like ls right it lists all the d all the files in a folder or it reads a file and so on and then at the end it gives you a structured output And this sort of hints to the agent how do you what what are you supposed to produce? So usually we ask for like a severity a description of the impact if he finds a bug uh reproduce you know

like kind of like a report that you would read from a security audit. Okay. So I guess uh these are sort of all the the techniques that that we're using to to have an agent sort of go through the codebase and understand what it's supposed to to audit. I'm going to I'm going to talk now how do we reduce the context window like some of the work that we've done. Um I guess this one is is more of a technicality but MCP was kind of trendy and so it was uh we quickly moved the static analysis behind an MCP uh server.

If you don't know MCP, it doesn't matter too much. It's just a way to have like a remote uh server that gives access to tools to an agent. Um and and then oh okay and then we really wanted to look at these traces and what was happening. Uh and so we we created this UI to help us and and here I'm just showing you that so you can see what it looks like. You have uh some stuff that was happening before. Here the agent asks for the implementation of some function and then we answer with the function implementation. you're not seeing that but we also say this function calls these other functions and it's called by these guys but we we give all that data to the to

the uh and yeah just to mention it as for a lot of the stuff that we're trying to solve when we started working on this tool not a lot of things were existing before but as we worked on that tool um open source frameworks started offering the tools that we actually implemented so like not so long After that, OpenAI like released this trace uh tool which is pretty useful to see what's happening um with the agent. So you can see the agent is just like listing directories lsl lsls cat it's reading something here is doing a document documentation search. So this is why we use rag. We if if there's a lot of pdf in the project

we we'll use rag here. Uh anyway, I don't know if this is super useful, but there's so many frameworks and uh we didn't really know what which one to use. If you're working AI, you might know a lot of them. What we ended up with is um it's it kind of sucks to use these high level frameworks that have a lot of abstractions and that break all the time. So, we're doing we're getting closer to the low level. We're trying to be as lowle as possible. So either we DIY like we do things ourselves or if we I mean we ended up doing a lot of things with the OpenAI um SDK just because OpenAI the the the models that they were giving us were giving us the best results and so we we kind of like went with them. This might

you know bite us uh later but we'll see. And sometimes we using other frameworks like we use lama index for example for rag and and we use frameworks just because they offer some functionality that's useful to us. Uh but all of these frameworks are trying to lock you in. They have their own abstractions. Um uh and uh yeah they're not always pleasant to use. Yeah. Not not really interesting. Another thing I can mention is that we're really try again there's this theme of always trying to reduce the context window what's in the the prompt.

We realize that when an agent roams around the codebase it creates a lot of tokens that are not necessarily you know relevant. It reads files it realizes they're not relevant. It reads other files and so on. So we we moved we do that for a lot of tools. We move these tools behind another agent. So now an agent basically can ask questions about the codebase to other agents and these agents will use these tools and uh and answer. I know we have a summarized answer that is only relevant for what we're looking for. Uh I don't know I have a diagram for that but it's pretty much going to repeat what I'm saying. If you had a bunch of ls cat ls cat like whatever instead of doing that uh we just asked a question and we got a

result. So that's much more compact, less tokens. And uh behind the hood, under the hood, what's happening is that there's an actual another agent that's doing that that search. And and here, this is where we could reintroduce rag at some point if we wanted. Uh rag would pretty much be like kind of a grip if we had a vector database. But yeah, many techniques like that to basically reduce the what we have in scope. Uh and yeah so so it just mentions for rag we also moved it behind an agent to make it more relevant.

Cool. And so so I guess at this point what we ended up with is uh what we call a mix of agents probably because there's a mix of experts in in LMS and uh mix of agents really if you think of agents as like work like human workers they behave very very similarly to human workers ex except that you have as many of them as you want they don't sleep they don't go to the toilet they don't eat and uh you can specialize them and so what we have is sort of something that looks like a factory floor where you have managers, you have orchestrators, you have uh people that just look for bugs. And so um what we have today sort of looks like that where we have an orchestrator that

will crawl the codebase create an overview of the codebase that's going to be useful for the security agents and it will uh then find more areas to focus on and then we have security security agents that will focus on specific parts of the codebase and then they themselves can use other agents to ask questions about the codebase um ask questions about CIRCOM uh some metadata with the MCP and ask researcher agents about documents if there's uh PDFs and more documents in the that are relevant and uh we're we're sort of trending towards what we call micro agents where instead of having a security agent try

to look for bugs they will actually run other agents and so we'll have specialized agents that look only for specific bugs. Um so uh yeah this and I I suspect that basically anybody who's creating tools in that space will have a design that will look like that and the only difference will be how many of these agents specialized agents they have how much prompt engineering they've done um how much human assistance they have like how much extra stuff they collected manually to improve the results uh and basically all of that is the secret source of uh someone trying to create something like that.

Cool. I I guess I I can give you also uh I I wanted to talk about the models and their knowledge of cirereum. We quickly realized that models don't understand anything about circum here. We use distilled models. So like we were trying to be cheap and use small version uh of deepseek. I think it's 7 billion parameters here and doesn't understand circum obviously. We asked uh distilled version of kodlama gave us some uh I don't know what that is maybe it's some very log I I don't I cannot recognize that language we asked uh distilled version of mistrol it gave us some rust um and star coder was the best one uh it said I don't know and then he asked me do you know circum but at least it was honest you know

so so okay we're like okay these models they really don't understand anything so we we need to get a sense of what they So we quickly created this evaluation framework and we asked specific questions about circum and the questions were like do you know how the assert function works I didn't write the question here but maybe I should for example do you know how the assert function works is it dangerous if you know circum you know that it's dangerous because it doesn't create asserts assertions it doesn't uh create constraints and so uh then we have another agent that will judge the first agent that we're trying to to um to uh qualify and but that other agent knows the answer and so they can tell us if it's a bad answer or a good answer. By

the way, that technique is called LLM as a judge. And so we quickly realized that even 03 mini nowadays we use 04 mini or 03 but at the time of that screenshot we have 03 mini. They don't really know what what they're talking about even with high reasoning. Um, and also we realized that if you run it several times, sometimes they will get the answer right, sometimes they will get it wrong. And that's bas because they're, you know, they're not deterministic. The temperature, um, well, the temperature is supposed to tell you how deterministic they are. Um, and, uh, it seems like they always answer differently. So, you can sort of like run it 100 times and sort of get an idea of how many times they get the answer right. But to us, we're like, okay, we're not going to play around. We know

that they don't understand things. we're just going to give them a lot of guidance. Um, so nowadays we give we used to give them guidance on basically the these things like is assert a dangerous function to use and so on. But nowadays we we realize that they still don't understand much about circum even though they're very good at working with circum um so now we give the whole documentation of circum and we can use an agent to even like compress it to something more concise. Uh so yeah so so that's sort of the lesson is like if if you put too much stuff in the context window you don't have great answers but if you put too little you also don't have great answers. So yeah we here we put a lot of guidance uh but also the

documentation and so on. If we had the time, what we would probably do is fine-tune um these models to make them understand circum. And actually, there's some work with move, the move language, where they fine-tune models uh to see if they work um to make them work with uh move. And they got pretty good results. Um I don't need to talk too much about that. These are reasoning models. Uh if you don't know reasoning models, they're just they're just very good. They're extremely good at answering questions because they will use these uh they they will try to like add tokens to sort of think out loud.

And by thinking out loud, my my I guess don't look at this too much, but by thinking out loud, my my my understanding is that they bring context tokens that are relevant by themselves from their training data. So it's kind of like rag except that the the data is not external. it's in the training data and so they're bringing tokens from there. I don't know if that makes sense but uh I'm I'm sure that makes sense for some people that have worked with the okay so uh here we can talk about the results.

So uh we we did this sort of UI to to help us uh think think through uh the these results. It doesn't matter too much but it's not really a product. It's not really a a thing that uh you can use or anything like It's more like an experimentation framework that we use internally to to to see if it's useful. Uh it was designed for us like auditors and uh maybe one day if it's very useful for us we'll we'll uh let developers use it and as I said it was initially designed for circum but now it's it's a generic framework that works with any kind of codebases and it's it's I think that's a key point. It's also human assisted. So we spend a lot of time figuring out what to

bring in context. So like like any documentation that can be helpful, documents, uh specifications and so on. That can really improve the results. We spend a lot of time looking at traces to see where agents uh don't you know it up and and and don't really understand what they're doing. And unfortunately the most of the most of the time that we spend is is spent interpreting the results. And that's that's what I'm going to talk about next. And uh wow, there's a spider right next to me. I don't if you guys can see that. I'm going to get away from that. Um so the prim the big problem with us was that we had to um well, okay, I'm not going to talk about false positives yet.

I have two bullet points. The results is that we actually found legitimate minor bugs. So it's very good at finding off by ones and and bugs that are very common in all sorts of languages, not just low-level circuits. We also found legitimate critical bugs and so that's where it gets interesting. Um I guess I say more on that soon. Oh more that soon because we actually cannot talk about it yet. Uh we actually run that on some open source projects and and found some interesting things but hopefully we can talk about it in the next days.

Um but but we also run that on on tools after we did the audits and we we found some really good bugs as well. Um, I guess I'm going to talk about that now. So, from the good bugs that we found, do we had some false negatives? Um, oh, wait. What am I talking about here? Oh, yeah. So, so, okay, false negatives. So, so we get some some some bugs, but they're not bugs, right? And so here the the ice makes wrong assumptions like sometimes it don't doesn't look at a function uh or it or it pretends that this function it knows the function and there's a return value that we're not looking at and so so we fix most of these with prompt engineering but we

still have a lot of false negatives and I believe the best way forward now is to um have an agent that can create proof of concepts and if we don't have a proof of concept then uh sort of like scratch We've experimented with that. It's kind of hard like it if it's like low-level cryptography, it can it gets kind of lost and it cannot produce a proof of concept, but in some settings it works really well. So, we're still experimenting with that. I think this is the most promising approach. Um, and yeah, I I think we can remove a bunch of false negatives even if we're not very sure. I don't think anybody expects this tool to be exhaustive. Uh if we really want to make it useful for other people than just us, I think we

just need to remove as many false negatives as we can. Yeah, lots of it. Uh yeah, that's what I said. So it takes it has a lot of friction right now. It takes a lot of time to try triage even though we find bugs. It's not something we use on audits currently. Um but we if we have a free week after an audit, we'll use it to see what's what's uh to to keep working on that tool. Uh, but I believe it's not it's at the we're starting to use it in interesting ways, but it's at the point where it's kind of a trade-off. Uh, it's not clear if it adds friction and uh it's it's very helpful or it's not helpful. We're we're at that limit.

Um yeah. So, as I said, AI is very good at finding bugs if we are good at finding bugs. If humans are good at finding something, there's a lot of documentation about it. it tends to be very good at finding them. It turns out um for things that are you know like sometimes we try to match the spec with the code is very tedious to do that as a human. The AI is very good at doing that. So there's stuff that is very good at doing. Uh and sometimes it was pointing in the right direction and if you investigated like it was it was a wrong bug. It was not a bug but if you investigated there was a bug behind it or he made a wrong assumption and there was actually a bug. Um there was

actually something where it was like he made a wrong assumption because the documentation was saying that a large number was not factorable but actually the large number was factorable uh if it was a smooth number and and it had a lot of small factors and so like he actually found bugs that we had found but it didn't think they were bugs because of other reasons. uh or sometimes it was like no this is a minor bug but that was actually a critical bug like we had like some uh some interesting things like that but yeah like at the end of the day it still requires a lot of expertise to to go through these uh these reports I don't know where the spider went I don't know if it's on me but someone let me know so so I don't think these these kind of tools are going to

replace us anytime soon because some of the bugs that um we find either ourselves as as auditors or with the tool are really tricky to understand and so I I think you you're still going to need uh especially if you don't have a lot of expertise in house uh you're still going to need people to go through and triage and so on. Okay, so this is interesting. I think we have pretty interesting insights since we worked a lot on on these kind of things and we're continuously working on that about the future of these kind of tools.

Um, so as I said, AI is already improving our job. Like we're using AI to learn about the codebase. Uh, we're using AI like with some agents on on like how to, you know, when you look kind of like developers are are using AI, right? Like if you're trying to understand a codebase or if you're trying to understand where a function is being called now, uh, you can you can sort of ask an agent to do that and they'll do it for you. That it's very useful or even to to read ducks or or these kind of things. It's it can be very useful. Um as I said I don't think it's going to replace us anytime soon like the the kind of bugs that we find are very low level and and involves a lot of math and these kind of things and uh for for

number of bugs that we find like the AI cannot find them and so now we're sort of trying to figure out if we with specialized agents we can find these bugs. So that's uh the next time you see me talking about that hopefully I'll be talking about that. Uh but I think for low stake audits I think AI and AI tooling is probably going to replace a lot of consulting firms. So web apps is a is an easy one. Right now there's a lot of work in consulting security around uh the security of web apps. I think a lot of that is going to die or these teams are going to shrink. Um smart contract is a is an interesting one. Also Netherm just released a tool uh to to look for bugs in smart contracts. I think for like a

lot of these small shops that don't have too much expertise are going to die. I think a lot of the the markets there and these these big consultancies are going to shrink in size. Um I mean smart contracts are not necessarily low stake because there might be a lot of money behind it but the bugs are u usually pretty easy to find like bugs are are very well documented and so on. Uh yeah that's that's what I said. Bug bounties is an interesting one also I wanted to mention uh from from what I heard there's more and more reports on bug bounties platform that are basically just agents reporting things and some people are actually using the bug bounty

platforms to uh rein do uh reinforcement uh learning and so they just throw whatever and then they see if it goes through uh the the filtering stage there but the bug bounties are not using AI agents to filter uh bounty And so it's kind of like, you know, agents talking to agents now. And so you you might wonder like why wouldn't, you know, if if you have a bug bounty today, why wouldn't you use use an agent inhouse to filter people reporting these? U my guess is that most bug bounties right now are just going to become SAS with like basically agents behind uh doing the filtering for you and and everybody will have their secret source to filter through uh through uh reports.

Another interesting thing to mention is that there's there's a lot more ethics uh talk these days. So I'm used to ethics in physics uh you know after the bomb uh was released there was a lot of talk about ethics in physics. Uh so that's pretty old and uh in cryptography there's some talks about ethics also um notably with rugaway uh and these kind of DJB and these kind of people and now we're saying all the AI papers that I'm reading about finding bugs with AI have an ethic section now uh and it's actually quite scary when you think of of uh finding bug with AI. I think there's going to be a lot of you know we're making finding bugs very accessible for people. So script kitties

are going to use these tools and and potentially exploit exploit smart contracts, but at the same time developers have access to these tools and maybe they'll make things more secure. So it's very interesting to to see what's going to happen. Okay, so uh I went fast, but I think I'm I'm on time, so that's good. Uh again, I'm CryptoDavid W on Twitter, and uh if you want to talk to me, I'm I also work at ZK Security. Uh part of the team is here as well, so you can ask them questions as well. Uh yeah, thanks. Do I have time for questions or Yeah, question here.

Thanks. Um have you tried to play with like Gemini CLI or cloud code and so what are the results and how how different is it from the tool that you show? Uh I mean we we we work directly with with the models. uh usually these uh actually I don't know too much about Gemini CLI but I I imagine it's it's like um OpenAI codeex or these kind of tools or um yeah so I mean at the end of the day we're we're basically rebuilding these tools like talking to the models directly and when I was when we were testing things OpenAI was sort of the best uh we're getting the best results there so we kind of kept at it but it's very easy to change models so basically there's nothing that

prevents us from like testing another model, see if seeing if it works better and just moving to the other model. Um, yeah, other question maybe here. Thank you for the talk. Um so my impression is that when the codebase is really big it will be really hard for the models to have all the context as you said. So did you have I don't know an example of a big code base that was really hard to audit with this

agents. Yeah, I can give examples uh per se, but we looked at some pretty big codebase and that's I mean that's why the architecture is basically one orchestrator browsing the codebase creating a summary of like useful facts and uh spinning up new agents that only focus on one part of the codebase. And then with that technique I talked about where they can just ask questions to another agent to look at the codebase, we're we're making them really focused and then we're making their um the prompt very small by using other agents that are that will do searches on the codebase for them or in documents for them and and so on. So that that's probably the best answer I can give. And this works pretty well or at least we

find stuff uh this way. It seems to be useful. I don't know how exhaustive it is. uh obviously, but um I mean I'm convinced that as things are going to get better and better, it's these tools are going to be even more useful. Yeah. Cool. I think this is it. Thank thanks so much again and uh yeah, thanks very [Music] house.