Benchmarking AI for Smart Contract Security: What Actually Works - josef gattermayer

Transcript

Okay, so thank you everyone for coming to this talk. So, my name is Joseph. I'm leading a key blockchain security and also I am assistant professor at the Czech Technical University in Prague. And today I will speak about some AI stuff as pretty much around this panel. Um because times have changed.

This month April 26th that was the Armageddon. It was the second worst month in DeFi losses ever. If we don't count the Bybit hack, it would be the first month, the worst month. But the problem is this was not a single hack. This this was like a constant month that every day a protocol was hacked. And I would say it was kind of like failure of the all security in DeFi. This should not have happened.

What was the cause? Of course, probably AI. Black hats get familiar with AI sooner than the white hats. And if we look at some numbers, we can use even bench. Uh if we ask the AI models to uh identify possible attack vectors, um the overall average of detection is something like 45%.

But if we know that there is a vulnerability and ask AI to write an exploit for that, the success ratio is about 72%, which means it's easier to exploit issues than to discover them. So, it's a huge huge advantage for the black hats. And uh what the rest of the industry that still is here has to do is of course to catch up. And I'm happy that all my previous or the previous speakers at this panel were addressing this topic. So, basically the security guys now know that they have to work with AI.

How does the security landscape look right now? I would say there are about four tiers of tools. Uh so, the first uh level is a plain models. It's basically uh back in the day called script kiddies, you know, who just take uh chat GPT and write, "Please hack this protocol." The sad news is it's quite good. Then the upgrade to this is provide some additional context using, for example, skills. So, you can have LM with skills.

Here, one of the industry, I would say standard skills are from Trailbits. Uh one of the mainly mostly visible on Twitter skills are from Pashov. They're open source, everyone can use it. You can upgrade your LM using these skills. Then there's like the old school batch. The these are the tools that were there even before. deterministic scanners because the others LMs are non-deterministic. So, we still have static analysis. We still have fuzzing. And then there is a I would say new category of tools that is uh now reacting on the situation, and it's a multi-tool multi-agent tools. And that's probably where the defense is. And uh to name some of the tools, Nethermind has

their audit audit agent. We have Ackee. We have our Wake Arena. The question is uh how does it work and how can we know what works? It's not that easy. That's the problem. But first uh let's look like what are the obstacles of uh AI doing the perfect auditor's job? W- Why AI didn't take over all the auditor's job yet? The first thing is the context size.

if we have some frontier models with 200k or even 1 million tokens context, still like a bigger DeFi protocol which has a lot of dependencies doesn't fit into the context. Which means the AI doesn't see it all. for this, of course, there are approaches how to address that. One is the native orchestrator, which basically is for example cloud code. You know, cloud cloud code if you uh type there, I don't know, do security scan of this repository, internally is slicing that into the context, you know, running multiple agents in parallel, and this is the deterministic or this is the native

orchestration framework. but still it's kind of random. You don't know what will it do, which we can see on the numbers that sometimes uh it works better, sometimes it works Uh so the other approach is having something deterministic. We call it deterministic orchestrator, which is basically like old-school program that is running the prompts, but uh the program is uh driving it every time the same way, so we don't depend on the built-in uh reasoning frameworks from the model providers.

The second problem is that uh we still don't fit into the context, and uh finding bugs in DeFi is about a lot of integrations. So, for example, we have cross-chain calls, and in a typical cross-chain call, uh the context of the two different, you know, changes also pretty big. And uh plain LL model or the plain element model with the default orchestrator doesn't discover these like distant relationships. For this we need some extra tools. And the tools can be the tools that we already had, which is the static analysis. So if the static analysis using some graph theory or something like this provides additional, let's say, hints to the LLMs, it works much better.

the third problem and probably the biggest one is triage, which is basically to find out if the finding uh is valid or is false positive. Uh because on average about 80% what comes from the AI, if you type, you know, "Find me bugs, make no mistakes." 80% of that is false positive, it's a mistake, you know. And that's the reason why all the bug bounty platforms are just like flooded with security reports. Immunify is banning about 20% of accounts because they are producing AI slop garbage, you know, and who is to read it. So that's a real problem that right now probably there are true positives among these findings, but there is a too much that no one wants to read it, you know. So

this the hardest problem to solve. Uh I would divide the systems that are solving this into three different categories. So the first one is the plain model. It's "Find all bugs, make no mistakes." It's this. Uh if you upgrade it with skills, uh we have we have the agent thinking reasoning loop, the same as the base model, but up to that we we add the skill file skill file, which could be some kind of like methodology hints, specifics about the protocol. and the third layer, which is what we do in Wake Arena or probably others are doing it as well, is that we add to these two things also some information that is deterministic,

which comes from MCP servers. and what we do, we provide we provide static analysis results, we provide the call graph, we provide data data dependency graph. And also what we have, we are not relying on cloud code as a reasoning framework. We have our own in-house deterministic orchestrator that is built specifically for security. So, does it work? We need to see some data.

But it's not that easy because of course when we want to benchmark multiple auditing tools, they have to be benchmarking on the same data set and there is no data set like for security or some academics try to define some data sets but of course the problem is when you create a data set, you can fine-tune on the data set which means it doesn't show then in the future that it will work. So, everyone is benchmarking against something else. Which means if one benchmark tells, you know, we discovered 98% of the issues, it doesn't tell it's comparable with 50% from the other benchmark. So, it's kind of mess of course.

What exists on the market as a benchmark data sets is EVM bench from Open AI / Paradigm but we didn't like it much because it's there is a lot of old contracts in that so it's basically like solving something that was already solved. Then another benchmark was presented by Entropic. We also didn't find it like very pre-native. There is some academic stuff and the last benchmark is from Zilliq which is basically from Code4rena and it's a it's a data set of 94 vulnerabilities that were discovered across 14 real protocols and it's quite recent. So, that's the reason why we like this data set and

we have built our measurements on this data set. Of course the data set is public, which means everyone can replicate the measurements. It's it's the core of the idea of having a data set. And how does it look? So, uh it's public, fixed, real bounties discovered by humans. We know that the humans across all the platforms that uh this benchmark or this data set was present discovered there 94 critical or high severity issues. So, this is the uh recall uh 100% is 94. Uh it's built on 14 real protocols.

And important thing during the measurement is that we have to disable our searches because the models are clever and when we when we wrote to GPT, you know, uh find all bugs, make no mistakes, uh it was cheating. It was trying to find audit reports, you know, online. So, we had to cut off the internet so they are not cheating. And uh so, this is core of our benchmark. What we tested, uh we took six systems. every time when a new model is released, we test it internally. So, uh we tested from GPT uh 4. Now, the this data I will present is GPT 5.2 5.2 and GPT 5.5.

Important is to have the parameter extra high reasoning, which means we are trying to get the best out of the model. We also uh of course benchmark Opus 4.7 on max reasoning. Then for reference, we took the best of uh skills uh and we were running that on uh the strongest model, which is Opus 4.7 with max reasoning. And then we took two versions of our uh Vicuna. One is older, the second is newer, we call it upgraded. And this is what we tested. So, how does it look?

This is the result. Uh so, the data set, which had 94 findings, in this data set basically uh with Wake Arena, we are able to get to 66 findings out of 94. So, we have a recall. Recall is the numbers of issues discovered out of all issues. So, we have recall 70%. The older version had something like 67. then the best performing uh LLM model in the plain mode uh was Opus uh 4.7 with max reasoning. It discovered uh 48 issues out of the 94.

Uh with partial skills, it discovered one issue more. Which I think uh is nice, but I was surprised that it's not that much. And then uh then is data from uh Zilliq from their scanner. To be honest, this is their data on this data set, which uh which is from the uh end of the last year. They didn't um update the data, so probably they are now somewhere else, but this is where they were at the end of the year. And you can see that uh GPT 5.2 is much worse. To GPT 5.5, I can tell that it also depends on the task. Uh for example, Opus is very bad at Uniswap mathematics. Uh GPT 5.5

is very good at it. So, this is also some part of our know-how. What we do is that we are uh prompting different models for different tasks. That's why we have the better results. If you look just at the winners uh to recap, so uh so the best performing plain model is about 50% recall. we are able to get about 20% on top of that. Uh of course, the question is when Mitos arrives, nobody knows how will it perform.

Uh but of course, what we are trying as auditors to do is always to max out most possible outcomes from the plain models. And if we'll be able to keep uh the advantage of about 20% on the most up-to-date model, I think this is fine. There is also another view on that, which is cost. prompt on the best performing plain model cost on average $2 per one vulnerability discovered. So, basically uh if some of you are here uh white hats who are uh who are on Code Arena, so this is quite a bad news for you that for for high or critical uh the

price now is $2. Um if we try to get more of this, you know, the efficiency goes down. So, of course, with our frameworks and reasonings, we are somewhere around $5 per vulnerability. And if you look at the Bash of skills, basically, we are at $11. So, for for like one more finding uh out of 97 findings, we paid like triple the price for four times more the price.

So, it seems like reasoning frameworks of the models is so good that providing it with like general skills that are not tailor-made for protocol makes no sense. Just the skills. There's also a view on the single projects that are inside of the data set. So, you can see basically this is the Code Arena. The second thing is the best performing uh best performing mm model.

Uh it's quite consistent, so there's no like uh the data were gained across all the projects. It's not like that we gained the findings in one project. What about the false positives? That's important question. The answer is we don't know. Uh the problem of all these data sets or all the results everyone is publishing with the with the AI security tools is that they don't publish the false positive rate. So, I cannot compare to this. I don't know what's what is their false positive rate.

What we can tell about our system we are trying to tune it that way that the false positive rate is below 50%. Uh we could have more findings like high recall, but at a cost of more false positives. And for us the the the like hard stop is at 50%, which means once we hit uh more than 50% false positives, we uh are able to sacrifice some of the true true positive findings because then no one would probably read the report or trust the issues.

The question is, as I said, the data set is quite recent. But, of course, the newest versions of the models could have been trained on the data set, which of course will uh show better results, but it will not tell that in the future findings it will perform the same way, you know. So, we did one more small benchmark, which we don't publish or didn't publish, will not publish. and we just looked at some uh audits that basically are not public. So, we looked at five audits that we performed in a key within the last quarter of 25.

And these audits didn't have published reports. So, none of the models could have learned on them. And uh we just took uh these audits. They were performed by the best security engineers of our team. And we compared it with the models. So, try to guess. It's kind of brutal. The humans uh within a key, which I think is quite a top tier auditing company, discovered 31 findings, which is the baseline. Uh GPT 5.2 discovered 29 findings.

Our uh Wake Arena discovered 35 findings. So, basically uh the tool outperformed humans here. And this of course means that something is wrong or at least for me it was a wake-up call. That the current security is broken. We we cannot continue as we as we were working before. Uh so, from now on basically the auditors they need to be creating uh their tools. Because auditor without a tool is outperformed by AI.

That's it, but it's how it works. The good news is that if you or if the auditors create something meaningful, it's still possible to outperform the base models, which is nice. So, basically now now the skill is in this. But it's not it's not only that bad. If we look at the findings, there are still some findings that the AI is struggling to discover. And mainly these are logic issues. So, if I was the auditor right now, I would be focusing on finding these because these are the ones that AI misses.

And it's uh economic exploits. It's uh protocol invariants. Uh for example, when we ask AI to create invariants for a protocol, it usually creates garbage. So, so the AI they don't it doesn't have the idea, you know, what could go wrong. So, in this the human thinking is still much better to to bring some idea. So, we are now focusing on finding invariants, not on testing them. And of course AI still sucks in the very complicated cross protocol bugs. And as we can see now during the hack season, the governments, admins, and stuff like in the infrastructure outside the contracts, it's still something that AI

is missing. And that's something that we know that we have to include in our audit methodology to basically look around, not just look at the contracts. Yeah, so what are the takeaways? Security be built on multiple layers as always, and now the layers look like there should be some cheap scan for the CICD. There should be some like very robust, the best AI tool possible, you know, for the pre-deployment checks.

There still should be the humans because even the best AI catches 70% of the issues, at least in the data set, not in our internal benchmark. So, there is still space for human work. And then in the end, it's of course still the bug bounties. So, yeah, this is the best we can do. Build a security from multiple layers. just we have to focus on building better AI systems than the hackers do. And what is the system? So, of course, what I would like to everyone in the industry who's building some tool, you know, please do benchmarks, do benchmarks on open data

sets so we can compare, you know, inspired, get inspired by each other to see which uh which basically systems are good marketing and which AI auditors are good tools. And yeah, that's pretty much all. So, thank you for the attention. We have still 3 minutes, so maybe >> Thank you very much. So, that was very interesting. Do we have questions from the audience? Yes, several. Maybe I'll just give you my microphone.

>> Uh did you try like a hill climb approach up uh in like or are you experimenting with that? You know, like something like auto search where the model makes up hypotheses, tries a lot of parallel experiments, and >> So so I I there's like a echo. I don't don't hear you. >> Hill climb up uh like auto research by Karpathy, something like that approach for the audits. Uh >> What is our approach for the audits? >> Yeah, I mean, are you trying something >> Yeah. >> on those uh lines as well? >> Like this is now part of our audit methodology. So basically, mhm in our manual audit reviews, we know that this uh AI tooling is a necessary step there.

So so that's that's the takeaway we we we made in uh late '25 that this needs to be part of every auditor's methodology. There's another question here. >> Hello there. Uh thank you for the talk. You've mentioned that this April was a terrible month because of the hacks, and you've connected it to AI. But what's your take on the fact that most of the hacks that happened in April were actually not smart contracts bugs or whatever that was eventually discovered by AI, but moreover configuration flows and off-chain issues. So what's your take on that, and do you have any ideas on how

AI and auditors can also help with that rather than just the smart contracts code? >> Yeah, thank you. It's of course like my assumption that this was thanks to AI because like before Opus 4.7, there was no bloodbath like this, you know, in DeFi. Uh but it's a speculation. Uh but as you said exactly, uh it shows some kind of like majority that the smart contracts are now not the weakest security part in the systems. The weakest part is elsewhere, which of course means that auditors, AI tools need to look also elsewhere. So, that's the reason why uh bridge configurations, oracle configurations,

everything like this needs to be part of the audit, not just the smart contracts. >> All right. Thank you very much, Josef. >> Thank you. >> Thanks for for being here. >> You too. >> The next talk will be the first talk of our second track, sustainability and speculative futures.