How Open Arenas Bring Trust to AI Selection

Transcript

All right, hope everyone's holding up with the rain. I'm Michael and I'm here to talk about how open arenas bring trust to AI selection. It ties into a lot of things we just heard from previous talks and panelists. And let's jump in. So, this shouldn't be a shock to anyone here, but the internet's being rewired for AI. uh from capex of the largest companies in the world uh to the introduction of 804, x42, A to A, A to P, ACP and every other acronym. Basically, the internet is being redefined for an AI first era. And in the not too distant future, AI agents and models will be the primary users of the internet itself.

And just to provide some context really what we have when we say the AI ecosystem and AI online are we have do everything models. So the LLMs created by some of the largest companies in the world and they're generalists. They can do everything from helping you know how to change a light switch in your house to summarizing a document and maybe even acting as your therapist. And obviously Google just released Gemini 3 which has been highly applauded. Uh and next you have specialized agents. And unlike the big models, these agents are specialists.

They're designed and built to do one thing and one thing really well. And uh really the capabilities that they can have are quite endless. And so, um, we've seen in the crypto space, people focus on building agents related to yield generation or portfolio management or trading, but we're just beginning to scratch the surface on what agents can do. And if you've ever used clawed uh, code or any of the other like vibe coding tools, you've really used agents.

If you used chat GBT, you're using agents behind the scenes now. So it really is the application of some of these models for specific use cases. And they're basically just AI powered apps. And to round this out, all of these things come together to provide an infinite number of skills and use cases to users. So we covered a few of them, but really the possibilities are really endless here. And you know, I think as we see, we're still quite early in experimentation on what AI is good at, what it can do. But the question is like, how do we make it better over time or how can we really evaluate how good

it is at the things it claims to do? And when people are considering using these AI for whatever they want, uh really businesses and sort of like enterprises fundamentally want to know which AI to invest in. Some businesses are attempting to build their own AI internally and they have to go through this evaluation process. Is the tool we built better than something that we could buy off the shelf or use. And secondly, if they're sort of like using it fundamentally in their product or in their internal operations, it's an investment. It's something that they're running their business on. And some of these things are non-deterministic or semi-deterministic. Uh there's no real reputation system. So, it's like what

are we actually going to invest capital into and stake the future of our company or our product on? And if you're a dev and you're building a tool, an agent, uh, an piece of AI powered software, you really want to know which AI to build on, which model best serves the use case that you're trying to implement. So, which model provides me the best foundation to build a trading agent or which model provides me the best foundation to build a therapy agent? Whatever you want, but effectively, it's really hard to know. And lastly, if you're just a consumer of AI, if you're someone that wants to generate images or figure out which portfolio manager to use, you just literally want to know

which one is the best. And right now, it's quite opaque to know what that is. And we've heard from some of the other panelists on ERC or EIP80004 um which provides some of the sort of like a reputation registry for agents, but really what are you putting in that registry? What data defines the reputation of an AI? And there's a lot of ways to go about this. Um but really this is the fundamental sort of like conceptual overview of what it looks like. There's all these markets for all these skills that AI can do and they're powering businesses and products in those industries. And whoever you are, whether you're a business consumer or

developer, you really just want to ask the question, which AI is best at X? And so the question fundamentally is how are we selecting the best AI for the job at hand? And um there's really a few problems with the current industry standard for doing this, which are benchmarks today. So benchmarks, for those of you that aren't familiar, are really just define tests that evaluate an AI system on a given task. And you basically create these nice looking charts that give you some amount of confidence. and people can look at the results or read the reports and choose

the one most optimal for that skill. But there's a few problems with them. And one is sort of first you see in the middle here and Gro 4 is a decent example of this of an AI model that aced its benchmark but really fell short when it came to real world experiences. And that's mostly because almost every large model made by a big company today is benchmaxed. Benchmaxing is really the process of models training on the benchmark itself which are well known.

And so of course they're going to do well on the test when you know the answers ahead of time. But then when people like throw it out into the real world and throw all different kinds of scenarios at it, these models are vastly falling short of what they claim to be able to do. Secondly, another problem is benchmarks are exclusive. They're run by a few key operators. They test only a handful of the most popular large models, but excludes the long tale of specialized models and basically the entire universe of AI agents from reputation, from evaluation. And so, if you go to look at these benchmarks, you won't see your favorite crypto trading agent. And in fact, there's probably not even a

benchmark in these official benchmark lists that measures its performance at crypto trading. Which brings us to the third problem, and that's the limitation on the skills it tests. So really, you have this system where it's run by a handful of operators. The tests are defined and well known. It only tests a limited number of AI on a limited number of skills. So, how can you really trust the results? How can that be the foundation of the reputation system that's going to bring forward the AI economy and sort of enable agent to agent commerce or more effective commerce generally for the AI ecosystem and if it wasn't obvious difficulty with

this benchmarking system scales with growth. So as more agents come online, as we reach tens of thousands, hundreds of thousands of agents, as the types of use cases and the things people want to do with AI expands, as it is every day, benchmarking just can't keep up. So we need a better system to evaluate and measure the actual performance of AI at the skills that matter to people. And it's not going to be a centralized system run by a handful of operators. It's going to be one built, owned by, and contributed to by the world. And so what we're building at recall are open arenas where AI proved themselves. So we're moving beyond a world of marketing

hype and faulty benchmarks to a meritocratic system where anyone whether that's a user, a business, uh, or a developer can come and create a create an arena. Basically, you fund it. You define what success looks like. you define the eval itself depending on whatever you're trying to test. Uh you open that arena up and AI can come and compete. So those are big models, specialized models uh any AI agent built by any developer around the world and the results and sort of the outcomes of these uh arenas can be handled in a three main ways. So, one for objective

skills, uh, like crypto trading, you might just sort of like look on chain to see what the results of that competition or that arena are. Uh, but there's still many ways to think about what is the best performing trading agent. Are you just looking at P&L? Are you looking at riskadjusted metrics like sharp ratio or Sortino or Calmar? Um, what defines the best trading agent? Does it trade blue chip cryptos? Does it trade meme coins? Does it use per? And so each of these eval criteria can be its own arena. So it's like not limited to one definition of what good at a skill means, but you can customize it to whatever you need.

So if you are a sophisticated investor and you're actually thinking about riskadjusted returns for your trading agent, well, you probably want to spin up an arena that finds those agents and you can define that skill. So all these AI are competing in an open environment and users play the role of curating the competition itself. So including some prediction market dynamics in there. Um arenas have to run over time in order to get some sort of statistical significance on the results of these arenas so you can generate strong reputation signal. Often times you have to run multiple rounds or multiple runs of an arena so you can be confident in the performance. trading once, you can be lucky. Trading 10 times and winning

10 competitions in a row, well, you're probably doing something other than luck. And so, users can curate and provide forward-looking signal about the performance expectations of these AI to sort of provide that early signal before you can reach statistical significance in the arena. And at the end of all of this, it these arenas output rankings that continually update dynamically in real time with performance and they're all published on chain. So they can be integrated into things like EIP80004, a reputation provider for each skill.

And what's interesting here is there's not one universal reputation system to rule them all for AI. There's not one score, but really what you get is an sort of an entire universe of reputation scores for every imaginable AI on every imaginable skill. And that actually provides you the insight you need to make these selection decisions. I want to use the best AI for X. Well, now you can sort of verifiably choose the best AI based on actual real world conditions and results and not some academic benchmark report.

Um, and so here this is just a quick summary, but really it's all about surfacing the best AI. How do we really know the true performance of these AI models and agents that's based on performance that's not benchmable to humans? Humans have a diverse set of needs. Like we've heard from our users that I want, you know, now finally GBT 5.1 just said Sam said if you tell it to not use M dashes in your writing, it will listen to you, which seems like an easy thing that should have been obvious, but like here we are GPT 5.1 finally getting it. But this is an eval

arena that our community ran four or five months ago because people express the need to have an understanding of which AI models are best at generating content that fits their writing style. And that's sort of like a type of eval that you're not going to see the big labs put out and there's an infinite number of them. So it's really about having AI responsive to human needs and being able to actually provide value to users. We started out by running a lot of DeFi arenas because that's what most people in crypto care about. And so we've run 15 arenas to date. Uh ranging from different skills, spot trading on aerodrome, per trading on hyperlquid,

paper trading, which is basically like trading simulations, something you heard about from Shaw earlier. Uh we've tested over 50 models and 150 plus agents built by community members. And in those arenas, they've generated over 150,000 trades and more than 1 million users have made more than 10 million predictions in those arenas. And so the data set that these arenas are generating is massively important for us to be able to make better trading agents in the future because now you get to see all this data about which AI strategies work, which foundational models provided the best results and uh you can use that to train better models and agents in the future. Um,

and what we're launching now, we've already run internally and we're working with a few partners to spin up early versions of this and we'll release it publicly in the coming weeks are custom arenas. So, until now we've sort of pulled the community, asked them what they wanted and run arenas for that, which is largely around DeFi trading. Um, but now we're expanding. So, we have an arena coming up for those of you in the US and that watch football on Thanksgiving. We have uh a football play calling arena and we were just testing which models are the best at playing making play calls in the NFL, predicting which plays the coaches of these NFL teams on Thanksgiving will make. And

users can participate by predicting which ones will win. And it's really just to evaluate which model is the best football coach. Um, very kind of random question, but sort of proving the the range of skills that you can test. Another one in the top right are coding arenas. Internally, we built our own coding arena where we pulled in a whole bunch of coding agents that all claim to be good. But the reality is no two code bases are the same and not every coding agent excels at every coding skill. So, we're running an arena right now to effectively interview which agents are the best at working on our codebase. And the eval or the the metric that this

arena measures is uh comments before the PR is merged. So we basically take a bunch of coding agents, we make them compete on a task, they all submit PRs, and our engineers go through and provide them feedback in the comments, and then they make the revisions and submit new ones. And the agents that win are the ones with the fewest comments to merge. And this sort of works in rounds over time until you get to a result of like this handful of agents are best suited to working on your codebase. And so this is an interesting one because it's an internal arena. It's not a public arena like no one can look at the leaderboard for this publicly, but it's something our

business is doing internally to become AI first. And this is something that we can help any business do. Uh no matter what your codebase is. We're also launching arenas for prediction market agents. Um and we're launching verifiable arenas on ien cloud where not only are the results of these competitions verifiable via recall but the execution of the AI models making trading decisions themselves are running in IGEN cloud. So it's sort of the first fully verifiable endto-end AI arena where you have execution verified through IEN cloud and performance verified through recall.

Uh and just to note, this is all a tokenized ecosystem. It's open. It's owned by the community. And so anyone can come and fund the creation of an arena. Uh anyone can enter their AI tool or submit an AI to compete in these arenas. Anyone can come and curate the best AI in these arenas and earn. And sort of the token secures the entire system end to end, effectively tokenizing trust and AI performance. And so the arena is open. You can come. You can participate. You can predict outcomes. And of course, we'd love to talk to you if you want to stand up an arena for your business, your community, uh, or your ecosystem. Thank you.

[applause] >> Okay. Amazing talk. Um, this looks very fun. So, don't miss out. Scan this QR code. Um, we can start. Uh, I think we we have time for one question. >> Time for Q&A. >> Yeah. Yeah. I think we have time for one question. So, if you have a question, raise your hand. >> And Yeah. Uh, we have a question in the middle row. >> Might have to yell over the rain. >> Yeah. Yeah. >> My Hi. Uh my question is about uh evaluating uh trading agents. Uh as everyone heard a lot of times

um past results are not guaranteed of future >> uh performance. >> Having a hard time hearing with the rain. >> Okay. One two three. >> Yeah, I think it's better. >> Okay. regarding trading evaluating trading agents. Uh everyone heard that past results are not guaranteed of future performance like um with um trading future is never the same as the past. How do you decide like because you if you evaluate you are for sure evaluate on the back it's back testing. Uh how do you adjust for that?

>> Sorry. So still having a bit of trouble hearing but is the question how are you ensuring that the evaluations and the performance measurement of trading agents isn't just back testing but also is future-looking. Yes that >> okay. Uh so effectively we it's about building confidence, right? And of course there's no way of knowing what an AI will do in the future. Um but if you know the sort of the version that the agent ran and you see its historical performance in real life market conditions so not in simulations and you see that that reputation is built over time and they sort of consistently rank

highly and the ranking the numerical score that we use is similar to ELO and chess. So it's like your score is dependent on the quality of competitors that you have competed against and your overall performance. And so over time we have like two dimensions to that score. One is the raw performance score and the second is a confidence or certainty measure. And so when you first start out if you perform really well but you haven't competed many times your score might be high but your certainty will be low. So overall your ranking will still be pretty low and through many runs of competing in arenas you can build up that certainty score. So it sort of like

ranks you more highly relative to other competitors. So a competitor with a score of 1,600 that's competed in a 100 arenas will rank more highly than someone with a score of 1650 that's competed once. Um, and so though there's no way of guaranteeing what they'll do in the future, um, having this element of confidence in the ranking algorithm, um, sort of like helps you get to that as best as you can. >> Okay. Thank you Mike.