AI Is Becoming a Superhuman Smart Contract Auditor | Alpin Yukseloglu

Transcript

Bank of Nation. We are here with Alpen Yukalolu. He is an investment and research partner at Paradigm. Also the co-author of a p paper titled EVM bench, an open benchmark for smart contract security agents written in collaboration with open AI uh to measure the ability of AI agents to just detect or patch or exploit smart contract vulnerabilities. We're going to talk about the way that AI and AI capabilities are going to impact our crypto ecosystem. our smart contracts. Alpin, welcome to Bank list. >> Hi, thanks for having me. I want to start off the question with a very big qu this podcast with a very big question. How at risk are we >> from AI? How large of a threat does AI smart contract capabilities pose to our industry? Yeah, I mean in the long term

it's it's now increasingly clear that AI is going to be extremely extremely good for crypto because uh especially on the security front because um we're going to get to a world where uh where because everything is much more secure the ceiling on the industry is much higher. So our partner Matt talks about how uh if you have a grocery store that's run by mom and pop because they can't see everything in the store there's a limit to how big they can get. But the moment you add security cameras in so security has this effect of increasing uh the capacity the carrying capacity of of an industry um I think in the short term it's up to us uh because the models are getting extremely good like strikingly

good when we started working on EVM bench which is a benchmark that consists entirely of fund draining critical bugs um around 6 months ago the models were able to find less than 20% of the bugs around 12 to 13%. And just over the course of while we were working on the benchmark, this number went up to over 50%. And in between when I drafted the launch tweet and when I had to actually hit send uh with the release of G uh 5.3 codeex uh it jumped up to over 70%. So these things are just growing at a blistering pace and it's very important that we uh position the industry in a way that we can defensively uh protect

against attacks. But in the long term I think it massively increases the carrying capacity of crypto. >> Yeah. I think what you're saying is in the long term we get something approaching perfect security. Yeah. Right now we do not have perfect security. Let me ask you the same question but a little bit differently. say only bad actors, only black hat hacked actors have access to AI capabilities. In that context, how at risk is our industry? Like how exploitable are our smart contracts given the increase in AI capabilities? >> Yeah, I mean I think um it's really hard to say when we approach super intelligence levels. I do think until we hit the like right now the models are quite good but they're not better than

the best human auditors. So we already have existed in crypto under this threat model of extremely intelligent uh adversarial actors that are constantly trying to break all of our software that with all the money in it. Uh so in that sense like crypto is already quite hardened. Uh but it's it's just really hard to know when we talk about sort of a technology inflecting into super intelligence. This is very similar to how um in coding uh capabilities were increasing mostly linearly over the last several years and in December last year they crossed some threshold where they were better than sort of the median engineer and a lot of stuff clicked for everyone and it started becoming this

aha moment and this sort of oh crap moment and I think something very similar will probably happen with security where right now it's like increasing pretty rapidly at a at still at a linear clip uh but it's not as good as the best human auditors yet. So, we don't feel it yet. It hasn't actually broken any of our assumptions. But once we hit in maybe 6 to 8 months, I I'm pretty confident at this point by the end of the year, a superhuman AI auditor, uh this will just completely break all of our assumptions and we'll need to go back and and make sure that we're hardening all of the contracts that are housing the what nearly hundred billion dollars of assets in crypto.

Galaxy operates where digital assets and next generation infrastructure come together, serving institutions end to end. On the market side, Galaxy is a leading institutional platform providing access to spot, derivatives, structured products, defy lending, investment banking, and financing. With more than 1,600 trading counterparties, Galaxy helps institutions navigate every phase of the market cycle. The platform also supports long-term allocators through actively managed strategies and institutional grade staking and blockchain infrastructure. That scale is real. Galaxy has over 12 billion in assets on the platform and averaged a $1.8 billion loan book in late 2025, reflecting deep trust across the ecosystem. Beyond digital assets, Galaxy is also building infrastructure for an AI powered future. Its Helios data center campus is purpose-built for AI

and high performance computing. With more than 1.6 gawatt of approved power capacity, making it one of the largest sites of its kind from global markets to AI ready data centers, Galaxy is serving the digital asset ecosystem end to end. Explore Galaxy at galaxy.com/banklist or click the link in the show notes. Euphoria brings onetap trading to the palm of your hand. Built on Mega ETH, Euphoria takes real-time price charts and projects it over a grid of squares. You tap the squares that you think the price will enter in just 5 to 30 seconds in the future. If the price goes into that quadrant, you can pocket anywhere between two and 100x your trade. No other application helps you trade faster and with more leverage on market driving events like FOMC meetings, presidential speeches, or global macro events. Thanks

to Mega ETH's real-time blockchain, Euphoria is the way to get real-time price interactions with the market. On Euphoria, you'll be able to compete with friends using Euphoria's real-time social trading experience, allowing you to go head-to-head with your friends. A great party trick if you project the app on a TV. It'll be like the Mario Party of derivatives. To trade on Euphoria, people can deposit stable coins from any chain or do direct fiat transfers, and everything gets converted into Mega's native stable coin, USDM, in the background. Check it out at euphoria.inance and download the app or find it in Telegram as a mini app.

In 2024, emerging markets generated over $115 billion in annual yield for investors with yields ranging between 10 to 40%. These are some of the highest, most persistent yields on Earth. The problem, DeFi can't access them. Bricks changes this. Built on mega ETH, bricks takes emerging market money markets and solvent carry and turns them into composable primitives you can access straight from your wallet. While DeFi investors earn 3 to 6% on stable coins and T bills, institutions have been harvesting 10 to 50% yields backed by sovereign monetary policy. BRICS connects these worlds with institutional grade tokenization, local banking rails, compliance across jurisdictions, and real-time stable coin settlement. BRICS does the heavy lifting so DeFi can

finally access real collateral and structured products on top of real world yield. Even the best carry trades can be within reach. Brrics brings DeFi's promise to the emerging world and brings emerging market yield to your wallet. Let the yield flow with bricks. >> Alpin, when we if we zoom out here though and we think about um AI intelligence and its security capabilities and its um bug detection capabilities kind of going exponential and we think about a super intelligent AI. I don't even know how to think about security in general because it can envision scenarios beyond human comprehension. For instance, what if it

thinks up a way to crack some of our cryptography with some math that we didn't even know existed, >> right? Like I I heard actually Justin Drake on a podcast recently talk about this. It's not just the threat of quantum computers which is kind of a real known threat and some of our you know um encryption algorithms uh are under threat due to quantum computers. But if we have a super intelligent AI, I mean who knows what it could have the ability to actually hack and uh decrypt.

I mean, I guess my question is when it comes to super intelligent AI, is security just like not even a thing we can prepare for? I mean, how do we even think about it? >> I I mean, I think security So, the way I would think about it is that I think right now this frontier is very illeible and if you try to do this at the li in the limit thinking, you end up leading to very odd places that that may be uh you know, very psychosis inducing. I think one of the skills of right like I think I think one of the >> right I think I think the capacity to face the singularity and stay sane is a very important skill to develop. Uh and I think this is um you know the best we

can do right now is that we can get ourselves into the frontier into this sort of experimentally bound future where we're running the experiments ourselves and be ready when those inflections happen to be able to react. Um because I think the the model of like there are only bad people in the world and and they're going to have access to to this technology and they're going to break all of our systems. I think this is what leads to the sort of psychosis around like are we all just completely screwed? But that's not what's going to happen, right? We're all going to be in there together and is when we're in this frontier and as we have access to these uh these frontier models, the the sort of agent harnesses around them that are able to run these exploits that are able

to for example find undiscovered math or that are able to uh break existing cryptography. uh there will be both sides of it and right now it's not clear whether this is going to be an offense or defense favoring technology. Uh I will say that there are still fundamental constraints in the world like you can't break laws of physics. Uh there's there's a there are systems that are chaotic like for example the threebody problem where you know the fact like the fact that even if you have super intelligence you can't predict too many more ticks ahead because it's just a fundamentally chaotic system. So so I do think that there's this um you know the world is a complicated place and and there there are still physical laws and

constraints that will catch these things. uh and uh and practically the best we can do is we we have to push that frontier together instead of instead of letting it just sort of happen to us. I think that's a fair point there. There are physical constraints here and super intelligent AI does not just mean like it could appear as a god to uh humans but it does not give it you know godlike capabilities to break physical laws of of the universe. Certainly. But actually, I want to dig into this because I got the sense that maybe you have some insight here because I've been uh doing a lot of staring into the abyss of the singularity and trying to stay sane and I I don't quite have a knack for it. Like sometimes I feel like um I'm staring and I am feeling myself go a

little insane. Like I just don't I don't feel settled about it. >> Is there some wisdom you can share? just like a pattern or something that like I I was starting to get out of what you just said. Well, maybe the key is to kind of take it a step at a time >> and don't think about the far future and the limit. Just maybe think about the next day, the next month, the next year. >> What what's how do you stay sane when you when you stare at the singularity? >> I I mean, I think the core the core point is agency, right? So, Peter Teal has this uh framing where um acceptance and denial, most people relate to them as opposites, but in many ways they're the same thing because both of them uh imply that you're sort of everything is

out of your control. If you're fully accepting that something's going to happen, then then you're not doing anything about it. And if you're fully denying that something's going to happen, you're also not doing anything about it. In that sense, I think both the doomers and the the accelerationists are both wrong. uh and that the real answer is uh is that we have agency over these outcomes and that you yourself can bend the arc of the future. Uh and I think there's you there's a lot of comfort in that, a lot of stability in that because if you believe that you have agency over the outcome uh then um then somehow like you're still in control. Now I will say that uh the current frontier and like I guess you can argue the frontier has always been experimentally bound which means we

don't know like we can't sit down and theorize about what is going to happen. Uh generally the way that we're going to figure out how things are going to happen is by is by uh by experimenting and then by seeing the results and you know these AI models and also the current frontier of technology is grown. It's not manufactured. it's much more organically evolved in a in a way that like we can't today really predict. Uh so even just the the like armchair pro uh trying to theorize about what's going to happen, it'll drive you insane because you can't know. It's it's sort of a it's not bound by your ability to to algorithmically figure out the future. So you have to get in the in the

trenches and try things. Yeah, there's some stoic acceptance, the fact that um you can't know everything and so you just have to let that go. And then there's also a belief in agency and I guess an applied belief. I mean, yeah. >> Yeah. How much of how much of agency is just sort of a a blind faith versus maybe like a practical faith of doing, how do for someone listening in general that uh feels themsel looking at the singularity and it feels frozen, I guess, by by like just like not knowing what to do and doesn't feel like they have a lot of control and agency. Is that something that they can develop or is your advice? Oh, you just have to

have faith. You just have to um believe that you have more agency and that will become self-fulfilling and you you will have more agency. >> I I I think so faith faith is uh faith is good but it's not a particularly agency inducing headsp space to be in. Uh I mean when we for example started having the thought that agents could get extremely good at exploiting smart contracts which we're obviously heavily exposed to uh you know about 8 months ago we could have just sat down and been like holy crap like are we screwed are we not screwed let's like and like right and and and then there it turned out like there's another path which is well you can go figure out to what extent these things are at risk and and then also start making making uh headway into

the the labs that are actually pushing this frontier and maybe start getting getting crypto integrated into them so that we can get into a position where uh where you know as the fog of war clears and as we start to see for example now that that there are defensive measures that we can take uh that we're in a position to actually exercise exercise those those paths um and and I think I think there's something about like uh the the dumerism and also the um the the general like in the limit thinking about about the singularity and about super intelligence that is just like captures people uh because it's like you can sit down you can think about it for a long time and and it'll make you feel very

strong emotions but at the end of the day like that's not going to be the thing that is actually like you can go build things you can go work with the people who are pushing the frontier you can get in the trenches and you can start contributing uh and and the information you gain from that is going to be much more grounded than whatever one can come up with in in their head. And this is I mean uh I think many of us in crypto are used to this because a lot of crypto was like this for most of its history. It was extremely illeible. It was very hard to pin down exactly what the use case was going to be. And now we're starting to see okay we have this store of value use case. We had stable coins that are compounding at this

monstrous rate year-over-year. uh the industry kind of gave birth to the whole uh market of prediction markets which which is sort of adjacent to crypto now but it's just compounding at this insane rate. Uh and you know 5 years ago it would have been very hard to say this is exactly what's going to happen and it took some level of some combination of faith and some uh some combination of agency to to actually go and build those things. Um and I think so culturally this has been a very big uh anchor point for Paradigm because our entire firm is built around uh around building and researching alongside the investment and and so if you talk to anyone on the paradigm team or in our orbit like the the sense of groundedness you get is is

anchored in the fact that we're actually we have contact consistent contact with the frontier and uh and and there's there's sort of there's comfort in that there's stability in that and there's agency that comes from that. I think it is worth uh understanding and reflecting on like the the only reason why the singularity staring into the void is intimidating is because what it's what all of these technologies are doing are providing everyone else with agency to produce the singularity in the first place. And so if that singularity is intimidating to you, uh, you know, grab a mop, you know, get do do something like there's work to be done. And I think, you know, the best path forward is is through there. Like if if you are

intimidated by everyone else having agency because of these tools, you can have agency yourself. >> Well, it's a it's a polarizing technology, right? It's it's like the the the threshold of agency you need to be able to execute on something is going down. So, so like if you're kind of on the fence, you get snapped to zero or to you can do a lot of things >> or to one. Yeah. >> And and I think in that sense like uh if one has the intuitive sense about themselves that they might get they might be on the side of that fence where they get squeezed to zero that can be extremely fearinducing. And actually the solution to that is to uh is to be much higher openness right adopt the technology much faster uh be much more fluid uh about about um about changing

and adapting to to the environment u there's um I think uh mount on our team had mentioned at some point about how um there are times when speed is more important than cohesion and I think the current environment we're in because the frontier is so unknown and there's and so unknowable to some extent Um there's there's moving fast and adapting fast uh has a premium over being able to sit down and figure out exactly what's going to happen and put the pin in the right place. And this is somewhat paradoxical because the more agency you have uh the more it may seem like it matters that you do the game selection right and you sort of pick the right game to play etc.

But practically and empirically what's happening is that actually it's better to move fast and ship the thing within 24 hours of inception than it is to sit for two weeks and try to figure out the exact right way to construct it and then try to ship it then. Um and and I think that probably goes all the way down to like many parts of of one's life where where we're in an era of speed over cohesion. >> Yeah. Yeah. We are in the just do things era. I wasn't al expecting this to be such a philosophical episode to open this episode up, but uh now I think we can kind of like corral ourselves and point our agency towards the topic at hand, which is >> what happens when people feel high agency with AI towards the security of our smart contracts.

>> Is it worth talking about like what kind of contracts are like most at risk or least at risk? Is there some sort of like category or knowledge landscape that we can understand that when AIs have have very high smart contract agency that we should be paying attention to certain kinds of smart contracts over others? Is there a conversation there? >> I I think um I think not in terms of market. I think that's really hard to say like you know a DEX contract versus a lending market. I think uh you know simple contracts that have been around for a long time uh I think are probably better in a better position than for example like the 200th contract deployed on Binance Smart Chain or something like that where you know there's a in the in

the past there's been a sheltering that's happened from uh being in a small market. So if you if you were deploying something uh where the most amount of money that one could make if they fully exploited it was sort of in the order of low thousands of dollars then you were sheltered by the fact that they're just much bigger fish uh and and the bad actors and even the good actors or the people who are trading etc like are just are just not you're not in their overd window. But as the models get better and because the cost of inference is so much lower than the cost of an extremely talented security researcher uh that long tail might get shaken out very quickly. So so I think most at risk is probably small cap uh or you know low TVL protocols in longtail chains uh that

are that are still built on uh well understood uh stacks like the EVM and solidity. Uh and then and then I think there's a there's just sort of an unknowable uh security risk for the for the major contracts uh the the OG DeFi contracts that are currently battle tested but are still very complicated and uh and you know we'll see over over the coming uh year or two uh to what extent those those contracts are actually exposed right so the OG contracts that have had a ton of lindy and a ton of value locked over time that have been tested by the market are like safer in the near term But nonetheless, the people managing those contracts will still need to have agency to be on the defensive to make sure that they are

winning the arms race against the offensive types. >> Well, well, the the prize is much larger for exploiting those contracts. So, so I think it has this like, you know, there will probably be this canary in the coal mine effect where there there will be smaller or or protocols that are less secure but have a lot of assets in them fall first. uh and and I think there you know I you know we'll have to look out for the first exploit that happens that is almost entirely from AI and then from there uh it'll be the race will be on to start start taking the defensive measures necessary right and then like the long tale as you said uh the long tale of contracts testing and prod will no longer be a thing because when the cost to exploit a $1,000 contract is like you know $10 to $50 of tokens then

th those contracts just simply won't exist. Somebody will write a bot that says uh hey Claude openclaw uh go hack me some some contracts and then that thing will actually have the capacity to do that because the people that you know didn't really think too hard about their security because they weren't didn't need to think too hard about their security those people will have uh will not have a good day. Yeah, I I think this is a general trend that that the long tail will get collected by people who can use AI well. Like for example, you can look at something like prediction markets where there are markets where if you trade them to perfection, the most amount of money you can make is maybe $50 to $100. And like it's not worth it for Jane Street to put a quant on those markets because it's

too expensive for them. Uh and it's bound by the cost of intelligence and the cost of attention. But if you can trade those markets nearperfectly for 10 cents of inference, then you'll do it. And in aggregate maybe that long tail is pretty valuable. So, so right now we're in a world where the long tail is sheltered by the fact that it's small. Uh and as AI gets better in all of these different domains, we should just be assuming that all of that's going to get collected by people who are able to use these tools. >> Let's talk about EVM bench. Uh this is the the paper the tool. You guys call it a tool. Is that right?

>> Yeah, it's it's a benchmark and also an agent harness. Uh so we maybe had two releases in conjunction. The first evaluates the ability of an agent to um to exploit smart contracts and then the second one is is sort of an agent harness that is like you know uh similar to an auditing agent. Uh so it it can actually find the bugs. Um and obviously the agent harness that we released is sort of not at the frontier of capabilities because we don't want it to be used for for black hats. But we have a we have a UI that uh that you can upload any smart contract into that will do sort of a uh a baseline check for bugs.

>> Can you define harness that I I'm pretty sure that's a technical term uh that I think coders will be aware of but I'm not. >> Yeah. So the the core idea is the model labs will release these LLMs, right? So you'll have GPT 5.3 etc. Uh and you know you can do the baseline test of like just just prompt the model just ask chat JPT hey is there a bug in this contract? And then you uh in addition to that like you can like let's say that gets you to to x% on the benchmark. You can add a bunch of scaffolding around the model that says hey like for example here is an EVM that you can test against. You can deploy a contract and actually run an exploit and see if you're able to drain the money. Uh and it turns out

that if you give agents these tools this sort of scaffolding that that they can sit in they perform much better. So the harness is like similar to uh basically it holds the agent uh the the model and it gives it superpowers that are specialized to the task. Uh now the the interesting thing on the current arc of AI is that most of these tools that we add in fall like flake off with time because as the model gets better it just absorbs the harness. the the core example being how at the beginning of Tesla's fully self-driving um development cardi talks about uh the majority of the code was uh hardcoded and uh and handwritten and very quickly it started ramping up to now I think

over 50% of it is actually just the model uh there's no like they had they removed all the C++ code that that was like saying if if x then y and the model just figures out a way to do it uh so right now we're you know the agent harness quote unquote in the if x then y like hard- coding hard-coded tools that we're adding in to give it these capabilities. Uh but u but probably in the fullness of time it'll it'll get absorbed by the agent >> as it gets. >> I see. I see. So like the harness is kind of like a bootloader to get it started but then eventually data will take over data and experience will take over the actual internal like operations of the machine.

>> Yeah. Exactly. Yeah. Uh I mean right now the harness is super valuable in like very counterintuitive ways. Like for example, it turns out that just giving an agent the ability like an environment to test against even if it barely uses it leads the agent to think for longer and for to try harder and and thus get better results. So it's like there's still so many lowhanging there's still so much low hanging fruit because the agents themselves are not fully well parameterized and calibrated yet. Uh but yeah, I mean definitely we'll get to a to a point in time when uh you know the next version of codeex or or opus will be able to just spin up an EVM on its own and we won't need our harness for it.

>> Okay. So what does the tool actually do? Is the tool the thing like the agent doing the exploiting or doing the patching or is it just the benchmarking? Like talk to me about the actual utility here. the core the core release the core thing that we want to get out in the world is the benchmark. It's how good are the models at at exploiting smart contracts. Uh there are three components to the benchmark. The first is uh the ability to detect bugs. The second is the ability to patch bugs. And then the third which is sort of the most interesting and novel contributions the ability to exploit bugs which which is um you know one of the biggest problems with previous attempts at having uh

having security related for example auditing agents has been this problem around false positives. So the agent comes to you and says, "I found 50 bugs in the contract." And maybe one of those 50 is an actual bug, but it just is so time inensive for you to go through and figure out which ones are real that it's not better than a human auditor. Um, and what we did in in the sort of uh in the exploit component of of the benchmark is uh we leaned on the fact that crypto is verifiable. Uh, and we used this production grade EVM environment where we load in a bunch of chain state and we uh we set up a bug environment and let the agent try to exploit it. Uh, we

leaned on this to lower the false positive rate down to basically zero. So it got to a point where if the agent tells you that it found a bug, it literally has a proof of concept that it can exploit against it can run against a production grade EVM environment. uh and um and drain money from a contract. Uh and this this is sort of the core breakthrough of the of the paper is that uh is that there's a verifiable environment that actually uh leads to a very low false positive rate. >> Uh that's the actual benchmark is like you guys have established like you guys can actually measure the thing effectively.

>> Yeah. Exactly. Because otherwise if if someone says, "Oh, we found all of these bugs and we got 90% of this benchmark." you don't know what it means because you have no way of knowing if half of those are real or fake, right? >> So, the verifiability uh was ended up being very important. I think this is one of the reasons why um models are going to get extremely good at crypto very fast because the basically you can slice the future related to AI into two categories. One is the verifiable stuff and the other is the unverifiable stuff.

And the verifiable stuff is very easy for the models to learn because they have a very clear training signal and they know exactly when they got it right and they can just keep running at that and and improve and and climb that hill. Whereas unverifiable stuff it's like you don't know there's no tests. You don't know if you got it right or wrong. So it's like is are you good at writing a poem? Is your joke funny? Like these are very difficult uh for the models to get good at. And you know, if you were to just take the whole universe of code and look at which pocket was the most verifiable, you probably would end up with a pocket that's almost entirely crypto, right? The whole substrate is based on the concept of being verifiable. Which means that with very little data even though we there there

isn't that much in the in the form of uh contracts there isn't that much in the form of like no crypto people are in these labs generally or very few uh the models have gotten extremely good because it's so verifiable. Um and and also just as the models get better uh like for example Gemini famously learned an entire language just in context. So as the models get better uh you you know the amount of data you need might might be uh might be lower. Uh so I think the general trend and trajectory of you know these models are going to get extremely good at crypto extremely fast. I think I think we can bet on that.

>> So when the EVM bench paper says something like top models are going from 20% to over 70% exploit rate you know something like the newest JPT codecs. Uh what does that mean? 20 to 70% you like 70% of all smart contracts it comes in contract with it can exploit like what what does that mean practically? >> Yeah practically it means we collected uh all of the historical um fund draining critical bugs from open audit contests like code arena. So the set of so so the bugs that are found are not like oh you know you you had the small issue where where maybe like someone could have uh frozen the contract for a day or something like

that. It's much more you could have strained money from this contract if you found this bug. And the 20% initially or less than 20% initially was meant that if you took a frontier model and you put in front of it uh all of the hardest um all of the hardest audit problems after it knowledge cut off it would not be able to find the vast majority of them. And by the end of the benchmark it meant the over 70% means that um you know if you just reran code arena >> uh and instead of GPT4 it had GPT 5.3 codeex >> 5.3 codecs would have found over 70% of the bugs the critical fundraining bugs

that human auditors found. >> Um so throughout throughout his throughout our history of human audits and finding bugs it would have found 70% of those >> of the critical ones. um with with some constraints like for example we we didn't go all the way back in history we started at uh past the knowledge cutoff of because we we want to avoid contamination so this kind of gets you a rough benchmark of like basically chat GPT five 5.3 codeex is like 70% as good as all of the human auditors out there something like that >> something like that although I these things are highly nonlinear like for example there are dumb bugs that can lead to losing all the money in the contract but actually not that hard to

find. Um, so so that's why I think I think the uh the there there's more in the paper that is notable about the fact that these aren't there wasn't just like one trick that the model figured out and like got to 70%. It wasn't like all re-entry, right? This is a very diverse set of bugs that it was able to find. Um but yes uh it's like fundamentally the the models are getting uh you know very close to being as good as the best human auditors. >> One thing that's fascinating about having a benchmark is then like it seems like all the Frontier Labs love to compete for benchmarks winning benchmarks right humanity's last exam.

In fact they almost like game some of the training towards these benchmarks. And so if you have an attractive benchmark that kind of propagates even like socially to all of the frontier labs, >> then it seems like it provides some sort of >> social incentive to have them uh you perform and train in order to um you know uh compete against one another to exceed each other on those benchmarks. Is that kind of the flywheel you feel like has been set in motion with EVM bench? I think I think it's I think it's an important uh I mean maybe the zoomed out point is that uh crypto in its history has been very stigmatized and very illeible to the AI labs. So the fact that there hasn't already been a

massive push for crypto-related evaluations is kind of absurd because the labs currently today are entirely bottlenecked on uh evaluations that are verifiable and economically important. and crypto ticks both of those boxes. So, uh, so I think it took Paradigm pushing a little bit of our weight around to to get this through into the labs. And I think yes, like our our firm hope is that this will uh start the flywheel of uh of labs paying more attention to this technology and we're going to continue doing work on this front as well. Do you have an explanation as to why that it's been so slow? It's also been perplexing to me because it's all open source. It's all out there. It's all like and

>> we already got all the data. >> Yeah. I mean um we talked to recently and his crashy his explanation was >> there's a lot of liability when it comes to um you finance and crypto and having AI models trained on those data sets, right? It's like what if an AI model does exploit a bug? Whose fault is that? So maybe there's some risk associated with it. >> Um there's also you just mentioned kind of a stigma. I think you know memecoin casino speculation.

Um certainly Peter from OpenClaw. He had a negative experience that he associates with with crypto culture with bunch of people calling themselves part of the crypto industry try to frontr run him and and develop memecoins and he just consider considers it shady. like what are the reasons why the frontier labs have been so slow to train on this incredibly rich data set that as you said it's it's it's perfect for training because all of it can be verified. My sense is that it's almost entirely a social thing. Uh I mean in in my peer group crypto is the biggest industry that has remained the most contrarian.

Uh and I think part of that is because it's very reputationally volatile. Um and part of it is because uh there's this dynamic where the best people in the industry the gap between the best people in the industry and the median person in the industry is much larger uh than anywhere else. So if you for example don't have exposure to the highquality pocket of crypto then all you see are the scams and it's it's like that can distort your view such that you you just completely dismiss the industry. And historically there's been a lot of alpha in that right and I think a lot of us have benefit from the fact that that there's significant reputationally uh reputational volatility and if you aren't as sensitive to that as a person in terms

of your temperament uh you can do very well in crypto but I think I think it's a social thing and I think it's just this legibility point about about there just hasn't been a uh a brand that can bridge the crypto and the AI worlds. I think it's like if something touches crypto, it it sort of in the AI world historically has been tarnished. Uh and as a result, people just have tried to try to avoid it altogether. And this is sort of um this was created an opening for something like uh for example something like EVMbench uh to get built and shipped inside OpenAI uh without any sort of comp. I think like all of the major model labs are going to be running on this benchmark and probably any

future versions of it without significant competition inside the labs. Like there aren't like 50 30 to 50 crypto-related benchmarks or or training environments that people are shipping. Uh in some sense it's actually agencyinducing for us because because they'll just defer to to the to the crypto industry uh to just figure out what's valuable for them. Uh but I think it's fundamentally a social issue and it's it's sort of uh tied to all of these dynamics around like you know you see someone who gets extremely wealthy who you don't think should get extremely wealthy like maybe it was like there's a lot of volatility in the industry and like there's some person who you don't respect who you who made a lot of money like these are all kinds of things that that go on in the minds of like the AI

researchers at these labs that lead them to think that the whole industry is a scam. Um and you know obviously there's uh this has made it an incredible environment to be investing in crypto because it's just not in the overton window of anyone in the valley. Um but uh but I think I think that's the core dynamic. >> Interesting. Interesting. We know um bots these AI LLMs they are very good at writing code. It's like the first thing that they got good at. Uh is that the same is it also true with writing EVM code. Is there a gap there between writing the rest of the world's code and EVM specifically?

>> Yeah, I mean historically there has been and part of the reason you know one component of what motivated us to start this work was uh the realization like man these models are so good at Python and so bad at solidity and so bad at for example like Solana related code honestly anything that touches crypto. Uh, and um, part of my expectation at the time was that we were going to have to go like p crowdsource a bunch of data from the industry and like spoon feed it into the labs to get them the models really good at this. But it turned out that because the substrate is so verifiable and also because there's sort of generality in these models, um, they ended up getting quite good much faster

than we expected with much less input than than we expected. So there's this dynamic where if you uh teach a model a poem in English and then uh you know biology in Spanish, it figures out how to write a poem in Spanish even though you never describe how to write poetry in Spanish the model. And I think that kind of dynamic is happening here as well where where it's sort of like quoteunquote learning the language of crypto without without as much direct training data. Um and also just because I like very uh hard to like under underrate the verifiability of the thing, right? Um most software is is hard to verify. You need human labelers to go in and check is this thing

correct? Is it running? Kind of the only threshold of verifiability you have is does the program compile and does it pass the tests? But the tests need to be sort of written by a human, right? You don't have this notion of like we have a bunch of state. We can make assertions about the state. We can, for example, send a model on a new contract on a new EVM that it's never seen before and make assertions about whether it's able to quote unquote drain money from it. Like those concepts all would have otherwise needed to be hard-coded into the program. But because it's crypto, uh we and there's so many standards. Um it's verifiable and the models are just ramping up really quickly in capabilities.

>> Okay. So do you think that this right now is the time that the the the floodgates of um crypto data to train these models that that is now starting to open going to be opened and if so what types of capabilities do you expect future models to kind of drop? I mean, yeah. >> Will they have skills and um I guess personas, connectors directly for for crypto within some of the the core LLMs or what types of developments are you looking forward to? There's a reason we started with security and not sort of general programming capabilities. It's because it has this very nice shape of it's extremely economically valuable.

It's extremely sort of intelligencebound, right? It's like you can't it's intelligencebound and it's very easily verifiable. So we know when an exploit has happened. Um so security capabilities I expect will develop very quickly and and then we've talked about sort of all the implications of that. um other crypto-related capabilities. I think um for example things in the domain of mechanism design or around market market related films like what is the mechanism for an exchange uh how how do if you have a market of agents right what is the best way through which they should coordinate with each other um these are I think open fertile soil and then of course you can go down to the

protocol layer right you can say well how does a model land a transaction on the Ethereum blockchain um how uh like there's a security side at the at the Ethereum client level, right? Or at the protocol client level where like sure maybe there are hundred billion dollars of assets sitting in open source smart contracts, but there's way more than that in ETH and soul market cap that can be exploited if you're able to find critical vulnerabilities in Gath, RET, etc. So, so I think going down into the protocol layer is going to be important.

um model capabilities around uh me and and sort of extractive uh extractive um tactics. I think that will have the same effect as you know longtail hacks, right? There's a bunch of stuff on chain that you can just collect if you if you're able to do the endto-end process of figure out alpha in the market, construct a trade and underwrite it and then submit the transaction and land it on chain reliably. These are all things that actually the models are not that good at right now. Uh but they will get good at really quickly.

>> And all of that does seem long-term good for crypto. However, the the here and the now and and the short term like when we read in the evan bench um paper that top models are going from to over 70% exploit rate. I'm not sure whether to like I mean feel good about that or bad about that because um it it sort of depends I what the intent is and who is harnessing it. So if it's uh white hat that's great. I mean that uh improves our ability to find bugs and exploits before attackers do. However, if it's black hat, that's not great. Uh because,

you know, it improves their ability to find these before we do. So, it depends. I guess with with this tool set, it depends who is using the tools and the intent behind them as to whether it is short to medium-term bearish or bullish. I guess one thing it does is it does seem to inject some variance and uncertainty into the market. I'm feeling that everywhere with AI right now. It's just like it could be really good, it could be kind of bad, but one thing is it's not going to be as boring. It's going to be >> highly variant outcomes.

>> I guess when you think about the capability that is being unlocked here uh and the ability for LLMs to detect exploits, is that good or is that bad for crypto in the short to medium term? >> Yeah. Well, I mean, you mentioned that in the long term, cryptos positively levered to almost all of these developments and as models get extremely good at security, this will raise the ceiling for the whole industry because and that's because it's like going to be a survival of the fittest thing, right?

Because all the weak stuff gets just >> exploited. It's up to us. I think it's up to us like what the path there is. Um, us being the industry. Uh, it's it may be survival of the fittest. it may be that that we get uh uh figure out a way to have the defense get ahead. Um and I think but the core point is that the amount of assets that can be sustained on these networks is proportional to how secure they are and in the long term there's this benefit where as security improves uh more assets will be able to securely stay on chain. Now, in the short term, I think I think this is one of those things where uh it's in our hands, right? I think it's bound by the industry's agency on

the best way to handle this. We don't know uh like if we just let the clock play forward, there's a lot of uncertainty. Um we don't know exactly who whether the attackers, you know, the black hats will get capabilities before the white hats do. Um but we also are active participants in this market and we can we can bend the arc of this such that for example we make sure that if there are frontier models or unreleased models or there are uh there are uh new developments in security uh relating to AI that we that we get this into the top protocols. Uh, you know, one one version of the world that you can imagine in the short to medium term is that you always have every single contract being scanned

by both adversarial actors and defensive actors 24/7. And when there's a bug that surfaced, uh, you know, whoever catches it first sort of will will react accordingly. And then in that world, it is just kind of more of a race between between the good guys and the bad guys. uh and and I think we have a lot of uh a pretty great hand in terms of in terms of making sure that the good guys have uh have the lead in that race at the end of this once all of the contract like the weak contracts have been exploited or we've beefed up um security enough such that they're not exploitable I guess that gives us an incredibly hardened uh financial system for the

world something that's ultra >> there's almost like a it's close to perfect Right. It's like how many how many nines? I mean, it's maybe like four nines, five nines. >> Um, and it almost creates kind of a a barbell model of a security for for the world for financial assets. Like >> the most secure financial assets will probably be in this dark forest environment like onchain. How do we know that? Because >> um the world has thrown everything it can at it, including our most intelligent LLMs, and it's still there. It's still standing, right? Hasn't been exploited. That'll be one side of the barbell. The other side, honestly, is things that are completely outside of the digital world altogether. You know,

like a clump of like a a bar of gold or something that >> digital gold, >> right? And then everything in the middle will be pretty exploitable, pretty insecure. I wonder if that's what is on the horizon with with our world. >> Yeah. Until the LMS figure out how to synthesize gold. Right. It's right and send the robots after the gold. >> Right. Uh yeah, I think I think that I guess that makes sense and I think that that barbell view of the world uh might be how things play out. Uh I guess the way that I relate to crypto as an industry and also as a technology is is that if you start from the first principles uh vantage point of let's say you want to do payments at the speed of light,

right? Like I want to send you uh Ryan money from America to Europe or some other part of the world. um and I sent it as fast as I sent an email. And the problem that you have there is that you don't know if I also sent that money to David, right? You have this double spend problem. And this was what Bitcoin solved. And it got the time for that transaction down to about an hour. And since then, we have had successive developments that have increased both the the speed of these transactions and the expressivity of these transactions. And I think that I think that in that worldview of this is like this is not just some path dependent thing that happened that the crypto industry emerged the way it did. Uh and that actually this is if you were to play it

forward from first principles that this is how it has to be. uh you end up in this in this conclusion where you know if you have agents that want to move at the speed of the internet uh and the current banking system was you know created before cars were invented uh that that these uh that those agents are going to uh discover uh the uh crypto rails as as the right way to transact. Um, and you know, I think there was a concern maybe like 6 to 8 months ago uh definitely on uh uh for me where it was not clear if the agents would get good enough at crypto-related software for example for them to be able to discover the current rails and maybe they'd have to reinvent them from scratch. But over

the last 6 to 8 months as we've been working on this work with OpenAI, it's become increasingly clear that uh that one they're extremely strong network effects inside of crypto and that and two uh these agents are able to just learn like they just want to learn these verifiable things and crypto is very high on that list. So uh so at this point I think you know I've become extremely extremely bullish on uh on crypto as a substrate for for these agents. Um, and I think it's sort of an open game. Who in crypto is going to win that? But, but the shape of the technology fits it perfectly.

>> The EVM is by far the most common programming language programming environment in in crypto. Solidity being the most common language. And then there's like some longtail languages like Cardano's like Haskell for example. Uh, and as we know AI loves data. The more data and AI can get on the better better it can be. Uh what do you think this network effects around environments how does that play into AI? Like is the EVM going to be the favorite environment for AIS to work in? >> Uh does Salana's uh what's what's Salana?

>> SVM the SVM. The SVM does the Salana's SVM also, you know, cross the threshold or maybe the threshold thing is a is a false illustration. What do you think? >> I think it's it's actually very hard to say. Uh and currently the ball is in the air. We don't know. Um part of the reason why I mean I can point to some of the bottlenecks we had while we were developing VMBBEN where we also have actually started work on uh on a Solana related component of it. But for example one challenge was that actually it it requires a lot of human talent like human talent to be able to go in and and construct these evals. uh and that was just much easier for us to come by for solidity and these things are really

hard to build right these are pretty heavy infrastructure. So um so even small additions of friction uh lead to you know when you need to cut scope in some capacity uh you kind of have to cut it in in the direction of the stuff that you can do more easily first. Uh that being said, I think the point about um the fact that uh you know, one crypto's verifiability is a huge edge and two uh these models are becoming less and less data hungry when it comes to learning new programming languages. I think it may end up being more even of a playing field than it might seem. Um, and I think, you know, at least we h have uh an intent and interest in actually going

across ecosystems and down the stack to the protocol layer and and making this sort of more expansive of a crypto flagship uh benchmark. Um, but yeah, right now there are obviously network effects around the EVM. Uh, right. One other example of a counterintuitive reason why uh why someone like Solana might be able to catch up is that u first at first glance it may seem really bad that most of the contracts there are closed source but if you take the world view of actually if it's open source it gets in the training set uh then the closed source benchmarks and training and contracts for training might actually be more valuable for for a model's development because it's not in

the train now For example, like we have uh these sort of what are called canary tags in various parts of our evaluation uh that filter them out of the training process of most models. So you know there are tricks you can do but still it's possible that these the bugs in EVM bench over time leak into the pre-training of the models whereas if it were closed source it would not leak in at all. Uh so so my expectation is actually that you know there will be some asymmetry at first uh but actually the models will get really good at all of it uh and and there will be sort of a merit-based uh uh sort of who you know who the best will rise to the top. Are you sitting on USDT or stable coins

after taking profits and wondering where to deploy next? What if you could access stocks, gold and ETFs without ever leaving crypto? That's what tokenized stocks on BitGet unlock. Traditional markets still run unlimited hours, but capital is moving onchain 24/7. On bit, you can trade tokenized stocks and ETFs 24/7 with up to 100x leverage, all settled directly in USDT or USDC. No brokerage accounts, no off-ramps, no platform switching. Bit has already processed over $18 billion in tokenized stock trading volume with most of that happening in the past month alone. The platform now captures close to 90% of Onondo's tokenized stock spot market share. As gold and silver hit record highs, onchain trading followed. Over the past 2 weeks, volume in SVLO on tied

to silver and IAU on linked to gold surged on Bit. This is Bit's universal exchange vision and action. Crypto equities and real world assets in one place built with crypto native speed and flexibility. If you want to trade stocks the way you trade crypto, explore tokenized equities on Bit. Learn more by clicking the links in the show notes. This is not investment advice. Few people in crypto put real skin in the game when they make public top or bottom calls. The D5 report is one of them. The week before the October 10th flash crash, Michael from the DeFi Report emailed his entire newsletter saying he's going aggressively risk off and sold the majority of his book from crypto into cash. This is when ETH was about $4,000 and Bitcoin was 110.

Michael runs the DeFi Report, an industry-leading research platform built on data cycle awareness, risk management, transparency, and most importantly, skin in the game. We like Michael at Banklist. We like his analysis and that's why you hear him on the Banklist podcast about once a month. And the DeFi Report is giving Bankless listeners one free month of access to the DeFi Report. So if you're looking for some sharp datadriven analysis to make better informed decisions around your portfolio, you can learn why and how Michael called the top and what he's doing next. All in the DeFi Report Pro. Check it out. There is a link in the show notes. Ethereum's aspiration uh is to formally verify its entire endtoend tech stack in the fullness of time. you know, we first we have to get like the beam chain. We have to do all all the the hard force to get there, but

ultimately we want to do a formal verification of the entire Ethereum tech stack. Uh AI based formal verification is that is that a real thing? How does AI capabilities work its way into the conversation of formal verification? >> Yeah. Well, I mean I think it is I think it is a real thing. Uh last year we invested in a company called Harmonic which is a foundation math model um co-founded by uh Vlad Tanov of Robin Hood and Tutor. Uh I think part of their thesis and I think part of where the world is clearly going is that um is that there's more software that is being generated than can be possibly reviewed by humans. uh and formal verification is one way to quickly check whether a

component of software is actually doing what it says it's doing. Uh and then obviously in the context context of security um it it can especially if the spec is written correctly it can be uh it can be a step function change. Now it's not a silver bullet in the sense that you still have to write the spec for the formal verification. So there's still surface for bugs to get in there, but you know, you can make the case that actually the surface for bugs in writing a formal verification spec might be lower than writing the code to start with. And uh and definitely with time, I think uh all of the best um all the best models, all of the best uh software uh will probably end up being formally verified. Uh and and if you take the

vantage point of an agent and you have two options to choose from, one of them is formally verified and one of them is not. the formally verified one might just gain preference just because it has all these nice properties. >> There's a section in the paper of the EVM bench paper called future directions. What does EVM bench v2 look like? How does this uh you've called it a project? How does this project grow from here? >> The top level goal that we have is to help the model labs develop the crypto capabilities of their models. And I think that security is one component of that and maybe an increasingly urgent component of that. But there uh there's so much that EVM bench does not touch.

So there are other ecosystems and stacks. There is a protocol layer which we talked about uh where maybe arguably it's more important from a security standpoint that the Ethereum protocol is secure rather than any specific solidity contract. Um there there are out of protocol components like how do you land a transaction on chain? How do you deal with the mempool? How do you deal with sort of the nondeterministic parts of crypto? Um and uh and then obviously there's there are components that are even farther on the verifiable and intelligencebound trajectory like for example around cryptography and around zero knowledge proofs etc. And I think all of these are uh extremely fertile soil for future work. Uh so we're

currently uh open to uh trying to source collaborators for uh for future versions of of EVM bench and and uh you know we're obviously working on uh on uh next steps uh for it ourselves. Uh and I think that this this direction of like we finally have a foot in the door into the model labs for uh for getting crypto capabilities into the frontier models and I think that we should we should leverage that as an industry and we should try to uh try to get these models as good as at crypto as we possibly can.

>> Elin, you're you're very smart uh as evidenced by this paper and in this conversation. You have a high degree of of agency. Obviously, you're definitely on the frontier. You've chosen to stay in crypto and not kind of uh leave and go to AI and you seem incredibly bullish at crypto. Even bullish that it's that it's contrarian at this moment in time. Why crypto for you personally? I've personally never had um hard lines around industries in my mind. Uh I think you know we talk about like I work in X because like to make what we're doing legible to other people but I don't think that's the right way to relate to it. Um I've spent all all of this time in crypto because it's been one it's

been extremely intellectually interesting and two it has this it's just as I mentioned it's remained extremely contrarian among my smartest friends uh in a way in ways where I can put my finger on exactly what they're missing. I think that's sort of that's kind of the best that that one can ask for. Um I think that you know we talked about how uh crypto is positively levered to the security developments in AI but you know you can make the case that it's positively levered to most of the developments in the world right now. Like for example as uh as the creation of new uh as new uh creation of new goods and intelligence etc becomes commoditized scarce assets become more valuable. uh as geopolitical instability uh ensues uh systems that are extra

sovereign right outside of outside of any jurisdiction that that are kind of the equivalent of endto-end encryption uh for finance uh those have more space to thrive uh and I think that you know I grew up in Turkey um most of my family is still there uh I think that people who grew up in America do not have and in general in sort of a stable world world do not have the uh the sense for what can happen uh as the world destabilizes and I think that uh you know as many people in uh in the country that I grew up in are starting to on board to crypto rails and sort of using that as a lifeboat I think it's increasingly clear to me that uh that this technology is is on a sort of

compounding trajectory to do really massive things and so the combination of that plus you look around and no one's even talking about it it's just really exciting. >> Yeah. And you do seem convinced that the acceleration of AI is going to benefit crypto that it'll be you know all boats rise together and I think that um well there is some category of software industry that AI doesn't seem to benefit at least in the short run. You know anthropic drops a new security module and all the cyber security stocks drop like 10 to 15% in one day. Um, why are you so convinced that AI's acceleration will be beneficial to crypto? It's not, obviously, nothing is guaranteed right

now. Um, I think if we let everything run its course, it may be bad for crypto. It may be good for crypto. We don't know. I guess the the um the conviction that I have is that if we push things in the direction that we want them to go in that we can make AI be extremely good for crypto. Uh and and also I think there's the component of this where I do strongly believe that if you were to rederive all of this from first principles, you end up in a place that's very similar to where we currently landed with crypto. Uh and um yeah, I think I just think that for all the reasons we've talked about uh that for fundamental reasons, crypto is extremely good for AI and AI is

extremely good for crypto. So I I think that uh I think that you know nothing is guaranteed and we still have to exercise our agency, but uh but I think for all the reasons we've discussed so far, uh it's like pretty clear to me that that these things are going to converge in a positive way. >> Well, let's end a note on high agency and conviction. Uh, Alpin, thank you so much for joining us today. >> Cool. Thanks for having me. >> Got to let you know, bankless listeners, of course, none of this has been financial advice. You could lose what you put in. Hopefully an LLM out there. A white hat is protecting it. We are headed west. This is the frontier. It's not for everyone, but we're glad you're with us on the bankless journey. Thanks a lot.