Smart Contract Auditing Automation and Collaboration

Transcript

[Music] Hello everyone. Uh my name is Andre Dobra. I'm one of the fullstack engineers working at Nethermind on the AI side. Uh I've been working in both web 2 and web 3 before and now I stand here presenting you a new development at the intersection of uh blockchain and AI and it is uh actually a presentation about smart contract auditing automation and collaboration. So I'm actually going to present to you two product and that is the audit agent and the agent arena.

Audit agent is a product that has already launched. It's been in the works for about a year and we're um going to present it now. It is an AI powered smart contract auditor. Uh it delivers audit reports that are ready in hours not weeks. And this is the biggest advantage in this whole pipeline because we're able to offer you vulnerabilities that can be even critical. It offers even uh high medium best practices as well. So you can fully defend your contract about uh of like any kind of vulnerabilities there are out there. It is uh at its core it has the following principles uh at its functioning be

underneath it actually uses a contextual analysis. So it is really important what is the protocol supposed to do and why it is trying to achieve those things because the AI has to understand what you are trying to achieve otherwise it won't be able to identify the unintended behaviors that your contracts have and this is essential in identify vulnerabilities because this is where most of the uh newest vulnerabilities are coming from. It is flawed logic and unintended behavior. So then we go on to check the structure of the smart contract and this is the how how are you achieving those things because if you're not achieving them in the correct manner then you allow for those vulnerabilities and for the exploits and then the hacks

to happen and then we get to our approach which uh has been uh an approach that we developed through a lot of trial and error actually because that's how it goes with AI you just try new things you experiment and you try to get to the right solution sometimes the effort doesn't lead to something useful, but sometimes it does. And we actually quite got there with a multi- aent system. So, we're using a bunch of strategies with a bunch of different specialized agents. And uh we got to this system that works. We're also using a bunch of tools which imply static analyzers and web search because as you can imagine, LLMs are trained on a large set of data, but they still lack certain information that is essential. Like imagine you give it the scope, you give

it the contracts, you give it the documentation, but it lacks some libraries that you're actually using or maybe it lacks some of the external smart contracts. So then we give it the ability to search the web and get that information to provide a more inclusive and comprehensive report for you. And I've also added that uh screenshot from Twitter because Vitalik was also quite excited about this uh development when it comes to AI. So formal verification and that is another another topic for the conversation but for it is also including smart contract audits and audits and bug finding.

Now I would like to show you the progress of the audit agent cuz we've actually advanced quite a lot just in the past six months. So I actually joined the company in February and back then when we were looking at the benchmarks we were at around like 15 out of 98 matched vulnerabilities. Now what are these matched vulnerabilities you might ask? Because we have to benchmark this model right we have to understand how it performs if we're doing the right thing if it evolves or if it is actually getting worse. So how we do this? We have five repositories with the official audit reports and then we also blacklist the official sources because we want to make sure the audit agent doesn't cheat

because we did catch it one time getting to the exact PRs for solving those those issues. Uh so we have to blacklist everything and then we run the benchmarks and we were at like 12 out of 98 matched vulnerabilities back in January and now we are at up to 48. So we're doing roughly 50% of what an official security audit would do at a fraction of the cost. Imagine that a full official security audit costs maybe tens of thousands of dollars. We're doing like maybe 40 50% of that for $150.

So that is the cost of the deep scan that we have right now. Now I'd also like to show you a comparison between audit agent and current models because you might be thinking okay why would I use this agent and not just use claude or something from open AI Gemini or something. So audit agent delivers up to three to four times more findings than the best available models. And we didn't just throw the contracts and the documentation at those models without any explicit uh you know message. We actually used our specialized prompt engineering with those best available models and they just delivered like we just had clawed oppus. It delivered about 16 17 findings out of those 98 but with our audit agent we got up to 48.

And also keep in mind that the the yellow bar is actually partial matches. So it is a vulnerability but it's not quite explicit. It doesn't tell the whole story. And the other ones are the exact ones. So if you look right here, this one shows that it's like 50% exact and 50% is partial. So it kind of leads you there, but not quite. Now this is the website. It's audit agent.nem.io. Uh you can visit it right now. You can try it for free. So, if you sign up and you just basically if you sign up, you get a free basic scan per month and you can use I believe you can scan up to

1,500 lines of code and uh the bas this scan is not obviously using all the strategies uh in the deep scan because we have a subset of that but it does deliver you it does give you an idea of what it can do. And we have three three different uh uh strategies basically. We have the basic one which is free. We have the advanced and then we have the deep scan. Now recently we've had an integration with virtuals. We've actually launched the together with them the undercover iris which is an agent that is native to Twitter and this is like the first major integration that we've had and it is a product that enables effective and

accessible smart contract auditing in real time. So it basically if you go to Twitter and tag the undercover iris, you provide a smart contract address that is verified or a public GitHub URL, then it will scan it and DM you a report of the vulnerabilities. As you know on Twitter, you can find a bunch of smart contract addresses for all the protocols and you can go ahead and just scan them now and get an idea of how vulnerable that protocol is of and of what's going on in there. Now let's get to the big picture. Let's take a step back and think about the everchanging landscape and that is the growing AI adoption. So we have shorter release cycles because we have all of those vibe coders now releasing products every day. And let's think about it how many protocols are coming out every day

or every week. There are a ton of them and obviously with the AI adoption the rate at which these things are coming out is much much faster. So there is an increased demand for automation. We have an increased complexity of AI generated code or spaghetti. So if you look at that image over there, I I kind of portray it. This is the kind of code that we're getting a lot of right now. And make sure that you won't catch me debug that for sure. This is what why we build these code auditing tools to make sure that we don't have to deal with all of that. And then there is the AIdriven development that can introduce novel or to be honest just simply silly exploits that we got rid of in the industry for

years, decades. But now because everybody is just producing so much code, they are reintroduced into the mechanic. So we have more bugs, more issues, more exploits, more hacks. And then we have all of these isolated auditing automation efforts not quite hitting the mark because there are other products in the market besides the audit agent that are trying to do the same thing. And they do not quite get there because they do not offer the full quality of a complete official audit report. So what we're what we noticed here is a need for orchestration. And so what if we put all of them together and they try to achieve the same thing and we put those findings together in a more comprehensive report.

Now this is where we introduce the next product which is agent arena which is in development and this is my main focus right now in development and it is a trustless approach. Basically what the agent arena is it is a place for project owners to come in with their project select the scope of an audit. So the things that they want to get scanned and we get a swarm of audit agents coming in and identifying vulnerabilities in the contract. So it is a a product that is a lot more complex and makes use of all of those agents out there.

And to give you the gist of it, it's basically a multi- aent system driven by independent builder. So if you're an agent builder right now and you have an auditing agent, you can come in and plug it in here because this is a a system very similar to other competitive bounty identifying program uh bounty programs that identify vulnerabilities in contracts. So it is at the same time both competitive and collaborative. It is competitive because obviously your agent will be competing with other agents to find the the most significant vulnerabilities. It is also collaborative because we put all of them together to help those that are uh launching their products.

Now you might wonder uh why since we have our own agent, why would we be trusted by other agent builders, right? So this is where the arbiter agent comes in and it is an open-source arbiter agent obviously as the name uh says and it does the judging of and the validation of all of those findings and we have it open source so anybody can check up its code to make sure that there is no we're not getting favored in any manner and it is running in a TE and and so after it runs in the TE it distributes the bounty on chain because imagine that the sponsor locks the bounty on chain the arbiter receives all of those findings judges them and then based on that distribute the rewards. So the function that's the functionality in

s in short basically and for the arbiter agent we've actually partnered with fala to have the trust execution environment from them and it enables private and temperroof evaluation of results. So it does basically judging so it uses reasoning models to assess the quality of the vulnerabilities. It also does duplication because you can imagine there will be a lot of the same findings submitted by different agents and we have to give a report that doesn't contain so much noise and there is also the short submission window to make sure we don't allow for human auditors to participate. We allow only two to four hours for each audit and this is uh something of an advantage also for the project owners. You come in, you request

uh a report and then we have the swarm of agents coming in and delivering you a report in like maybe four to six hours depending on when the arbiter agent finishes his job. Uh it is open and so these are some of the key features but because I'm running out of time this is a platform that we have in staging right now. So you can actually go ahead and if even if you don't have an agent you can start building because we're offering you a template and we offer comprehensive documentation for you to start building right now. So this is what the website looks like. This is a work in progress. We will deliver a new design very soon. But the platform is available right now. This is an auditing agent template that we have live. You can scan that QR code and get to it. It offers just the core

functionality of f fetching code, assembling the prompt and quering AI for vulnerabilities. The thing is that there is a new AI economy coming up, right? And it is about human effort coupled with AI effort. And many people feel like they're left behind, but actually we offer the tools and the possibility and we enable everybody to start participating in this new economy. So you have the tools, you have the platform and you can start building right now and getting a spot for yourself in this new economy. These are the current plans. And just the contacts right here.

Okay, that's it. Thank you. [Applause] [Music]