AI agents on prediction markets

Transcript

Cool. [Music] So, let's get started then. Hello everyone. I'm Gabriel. I'm with Nosis AI. Um, today I'll be presenting a little bit of an overview about what we've been we've been doing with Nosis AI for the last uh roughly two years, one year and a half. Um not sure if everyone is familiar with what Nosis has been doing but um it's it's been around for roughly 10 years. They started with prediction markets. Uh they launched a couple of infrastructure projects like cow swap most notably safe and now circles which I also touch uh touch upon this presentation. Um and yeah let's get

started then. Cool. Uh here a brief agenda. So I'll start with the trader agent which is an agent operating on prediction markets of course. Um the second topic will be the safe watch which is an agent that is able to secure your transactions on safe. Um the third thing is a kind of economic game around treasure hunting. Um and the fourth thing if if I have some time is is some bonus topics which I prepared. Cool. So I don't think I need a very large introduction to prediction markets after an Clemon presentation but not sure if everyone is is familiar with poly market or other prediction markets.

So just a very brief ultrash short introduction. Um you have you start with an outcome or a question that you want to to um answer or have an estimate for. I can't use the the laser but so in this case will any new airline suspend flights to 80 in this case. Uh and you have two outcomes. So yes and no. Um each outcome has a given probability of happening. Right? So in this case yes is roughly 57% and no is one minus yes. So um 43 roughly um and those those probabilities are also the token prices.

So if you are going to go to this market and uh place a bet on this that means let's say your view on this market is yes then you go ahead and and buy this token this outcome token yes and this means that when this market resolves and you hold yes tokens you each yes token entitles you to one unit of collateral after redeem if your bet was correct and if your bet was wrong you're entitled to nothing. So in this case for example um the price was roughly 57% so you can purchase each yes token to uh at the price of uh 57 cents and if your bet is correct and you redeem that for $1, you had a return of roughly 80% more or less which is uh not bad. Um yeah so what

what I'm trying to say is there's a monetary incentive to update the price. So every time you let's say you have a view on yes and the market currently is pricing this at 57%. You go there and you purchase yes tokens there is an AMM behind it. So the yes token will become more valuable thus moving the the odds towards yes. So it will be the the market has a larger odd of yes happening and that's why you kind of move the market in the direction of the of the estimate that you are seeing thus providing um a profit potential to report the the outcome in the correct direction. Um if that makes sense. Sure.

And on the other side it also may it's also true there is a disincentive to misreport simply because if you're betting in the wrong direction your payout will be negative. So you're incentivized to provide valuable information to the market. Um yeah, public odds then as I said act as a reliable information because parties are incentivized to report correctly. Cool. Um yeah and then uh quoting Vitalik here uh on this topic of info finance. So why are prediction markets interesting and useful? Uh of course gambling is is a is a big one but prediction markets can also be used for info finance where you start from a fact you want to know. So the question will flights to 80 be suspended? Um and then you design a market to op optimally

elicit that information from market participants which are incentivized to provide accurate information. Um excellent. Cool. So coming back to our AI and trader agent um our goal is get the highest possible accuracy on predictual market bets. So that means that for every bet we do, so every direction we pick um our we we get a proportion of those bets that were correct and that's what we define as our accuracy. Let's imagine we have the highest possible accuracy. We have a deli dely oracle for example which is 100% precise. Um what that would be the case uh in that case we would have a deli like behavior of future events. So we would able to

predict future events at 100% accuracy. Of course very new topic but that's what we are working towards increasing our accuracy more and more and as we have high accuracy we can also leverage those trader agents to some other markets for example use prediction markets for decision- making and uh that's of course futarchy which I will touch upon in in a few slides then yeah diving diving a little bit deeper into the agents actions it is comprised of four big steps so the first step is research in that is if the airline was going to suspend flights then that might entail Google searches, tavly searches.

We also of course use different models for this search, perplexity among others and different strategies. But in the end what comes out is a research report. So it has previous flights, are there strikes happening, weather conditions and so on and so forth. Um and with this research we go to step number two which is a probability estimate. So with this probability estimate, we try to interpret that research and spit out a float value which gives the estimate that this event will resolve into the outcome we're asking about. So in this case is a the probability of yes, right?

Because it's a it's a simply a simple binary market. Having this probability of yes and also a measure of confidence, we go to step number three, which is the trade placement. So there it's a matter, it's kind of a betting strategy as we call it. So we then look at market odds. We compare with our probability of yes. We also compare our confidence and we have some logic in place to determine one which direction we are placing a bet on. Is it yes? Is it no or and also how big our trade should be and that's where our confidence comes into play. Of course if you are more confident about the prediction we should bet more and uh less confident bet less. Finally comes to step number four payout redemption.

Um that's simply collecting the payout if our bet was correct or doing nothing if our bet was wrong. Um, and here also a call out to the platforms we we uh operate on which are on the nose chain. Uh, Presio, which is the rebranding of Omen, and of course, SER developed by the by the Claros team. Yeah. Uh, here's some results. Uh, all by the way, all work is open source and transparent. I'm going to present a QR code afterwards with GitHub links, uh, with the blog post we are also publishing, and most importantly, the Doom dashboard to verify what we're doing. Um here's the agents betting accuracy on the y-axis versus time. Um you can see that in the last 30 days the

accuracy has been roughly 75% across all agents and across all categories which is quite decent. Um and then um here's the number of actions per day. So this includes both buy and sell because the agent the way we process it we pro the agent is triggered so to say every two hours and then every two hours it process the markets and sorts and filters them by some conditions um and then uh it places a trade either buy or sell depending on the previous position and for example it's also interesting why the sells are there is because if the agent has placed a bet earlier and his view changed or maybe this position grew too much then there are also actions like take profit

for example that the agent can take so that uh it incurs also a cell. So um those kinds of actions are also um part of the behavior of the agent. Cool. Um now comes the futarchy part. Um so it's it's basically uh governance decisions powered by prediction markets. Uh interesting quote here from professor Hansen. Vote on values but bat on beliefs. Um so doing a a brief introduction again as um probably was presented before um futarch is useful when you're trying to make a decision and want to leverage prediction markets for that. Uh simple example should ethereum implement a pectra upgrade um and so this is you start with a question

and point number two is you want to uh optimize for a metric for a given metric which can be dangerous but in this case let's say there's a simple metric you can have multiple but this is a simple example. So the metric here is the E if price. So you're trying to maximize the E if price for this condition for for this decision and with that you open you create two markets in parallel. So you by this way you have two uh universes that you're also investigating. So universe number one on the top is the you're trying to predict the price of if if the upgrade is implemented. So this is a scalar market which will convert to a given to a given value if the upgrade is is implemented. And conversely on the bottom the there is a similar market to the price of if without upgrade. So

you're trying to quantify the influence on the Ethereum price of the upgrade. So you wait for the markets to resolve uh typically a few days depending on the market and you observe which market has the highest price because that's what you're optimizing for. You want to maximize the price. In this case um the scenario on the top was the winner. So if the highest price is picked and implemented that's the decision. So let's go forward with the up upgrade and the scenar the market on the bottom is just ignored. It's deemed invalid and the money is returned to the users. Um what why that's useful as well is okay we made the decision we implemented the pectra upgrade. But let's assume in an in one year but when this uh price is also observed then it's there is a

comparison being made between the the Ethereum price and the price that the market resolved to and people who placed a bet on the correct direction which means that they they predict the price well they are incentivized and if they predicted the price not very well they also have a lower payout than they would have otherwise. So uh this is also to incentivize correct and and uh good participation uh in broad terms. Cool. Um that was about full turkey and the trader agent. Now comes the part about the safe watch and um this agent that secures our safe.

Um I ass I I assume I don't have to explain what safe is uh to you guys but um after of course after the by bit hack it became obvious that additional measures need to be need to be implemented and need to be available. So we develop an agent which we call the safe watch which can uh act as a co-signer and you can add this agent to your wallet does um does scanning your trades and also moni so monitoring your trades signing and rejecting the the transactions uh in a continuous manner. Um the the idea is it it has different checks. It can uh scan for malicious addresses blacklists scammy addresses scammy tokens and so on. But the most interesting part is

really the LLM analysis. So um it acts as a kind of bank fraud detection. So for I'm going to show some examples uh in a bit. But the idea is um if it identifies some odd behavior when analyzing this uh these kinds of data like previous transactions, previous balances which go entirely into the context then it's able to uh reject said transactions. So it has some more context into uh whether this transaction is a is a fraud and should be rejected or if it's okay and should be approved and executed.

Um yeah, this is the first example. So this is a a non-malicious transaction. I just wanted to point out the read out the LLM part. It just says that it's a swap of 250,000 euro for USDC similar to previous swap. So it means I learned what you did earlier. So uh I know that this is kind of thing that uh is allowed. It's part of your previous behavior. So I'll go the LLM thinks it's okay and goes ahead and signs and executes it. Um another example here it's a malicious transaction. So this was flagged as potentially malicious because there is a large amount being transferred to a new recipient address that I have never never seen before. So this means that LM is a little bit of suspicious because uh

it's a new address uh maybe a new token unusual amount. So in this case it rejects the transaction and also sends a message to the user via the safe APIs telling that hey I rejected that that's uh let's please do not go go through with it and this is just illustrates this agent being a kind of bank fraud system which learns from the previous from the previous transactions and uh tells you uh kind of learns from your patterns your behavior and is also um specific to you to your needs to your behavior to your previous actions. Here's just an example depicting how it looks like if your transaction is rejected. You see here from the safe UI that this transaction was yeah onchain rejection there. One of them uh signed a

rejection order and then it cannot go through. Um and yeah cool uh how can I try it out? So um everything is open source. So I'm going to present some some links afterwards. Um also import important to notice is that the agents permissions are limited. So the agent is not able to propose transaction or create a new transaction but rather it can only sign. So either confirm and then and execute that on chain. So we we were sponsor the gas fees on nois chain because they're so low. Um and also the agent can reject the transactions uh if it's deemed malicious. Um also interesting is that the there are three ways to interact with the agent. First way is simply add the agent that we have running on our own premises as a signer to your safe.

That's one possibility. Or you can also run the agent locally and then add this new this new agent to your safe if you prefer to try it out yourself. The second way is via API. So uh we also have some open AI endpoint open API endpoints that uh anybody can test and a web app which uh also people can test out and it supports Ethereum mainet and also knows this chain as I mentioned. Cool. Uh so this is the third use case. Uh this is the economic game I mentioned earlier. It's called NFT treasure hunt.

Um, so you most of you guys probably heard about the Frasa agent and the the the Frasa game. So there was this agent which which was got quite some attention in end of last year and the idea of the Frasa agent was it had access to a a small bag of money in the beginning and every time you try to send a message to the agent you would pay some money and this the the value or the the cost of sending each message would increase over time. Um, and why would people try to send messages to this agent was because the agent had this treasure which was increasing on with value on time. So over time, so every time you sent a message to it, a portion of it would go to the treasure. So it had more and more money in the bank. And if you were able

to crack Frasa and convince her to send you the money, you got the entire treasure to yourself. So it made sense to it got quite some attention to for so that a lot of people tried it out and some clever uh developer managed to crack Fraser. It used a very complicated prompt with anyway tons of of tricks and he was able to unlock roughly 50k on uh from Frasa basically which got which was quite interesting from a security standpoint and how you code your agent and we tried to do a uh something similar inspired by Frasa but in our case so we kept the treasure idea so we have we have a bunch of xy in the treasure um and instead of having just one phrasa we have five phrases because

to open the treasure one needs three keys out of five. So there are five sellers each with one key. If you get if the one of the buyer gets three keys, it can open the treasure thus having some monetary reward. Um but it's it's it's actually actually tricky because one of the agents so uh seller three for example, it was coded in a way that it's a trickster actually. So we coded it in a way that hey try to trick everyone to send send so that they send you a lot of X die or a lot of E or whatever. but do not send the key under any circumstances. So, it's a trickster, a fake seller. Whereas the other sellers were coded like try to get as much as much X die um which is the the gas token on on no chain as possible uh and then

send them the keys. So, they are real sellers although there is one that is a trickster and the buyers were given also an initial amount of XDI and their objective is to open the treasure and have a monetary incentive. Um, and we had we let this this game play for a while and had some some interesting conclusions. Um, and yeah, this is this is how the game looks like. It's a simple uh web app. You can see the you can see the agents here on the left hand side. We had uh we played that uh during DEPCON during the Nosis conference some weeks ago and um it was quite interesting to see the agents playing around with each other because they can also send messages between each other mediated by the chain. Uh and within those messages they then try to okay I can offer you

this amount of money in exchange for the key. They can negotiate and they can enter an agreement does exchanging keys which are NFTs as well. So what we were trying to learn from this experiment experiment was mostly three things. So can agents learn that's the first question and um what we what we observed is yes they can learn because we also implemented a way for for the agent to reflect. So there is a round they play they try to get the keys somebody wins and open the treasure then we also reflect that with the past activities.

So we tell the agent this is what you did in the previous round. This is what other players did in the previous round. Um learn and reflect what worked, what didn't work. And we observed for example that an agent had interacted previously with the trickster that was would never transfer the key. And so and saw that it didn't work and then the next rounds this agent did not interact with the trickster or interacted less often. So which that was interesting. Um the second part was if the the second question we asked ourselves can agents cooperate forum alliances and uh we didn't observe that it was mostly a one uh against all the the interesting part was mostly a a role change. So if you remember here there were five sellers.

One of the sellers decided actually to accumulate other keys which kind of went rogue. That was uh not not expected but definitely interesting. But um alliances so more agents trying to let's say let's cooperate to get a key and then we open the treasure and split the the rewards. We haven't seen that but uh would be curious to see uh maybe more agents or you guys also playing the game and finding out some other behaviors. And the third thing which I find the most interesting is the can agents start a tiny economical system. So can one agent for example act as a market maker and work on both sides. So he both buys keys and sells keys and tries to arbitrage or make some money on the on the spread there or even borrow some money to purchase key which is similar to the to

the second point here. And that would also be quite interesting because that allows kind of a economical playground using the agents. uh but that's also still in the idea phase but would be nice to to observe those behaviors as well. Cool. I have some bonus topics now that I would like to present. Not sure do I have time now? Awesome. So cool. I went fast so that that's that's the reward. Um so the first point is the identity for AI agents powered by circles. Um that I would like to to briefly touch upon. And the second point is the how to improve our agents predictions which is the trader agent. the first point that I was talking about. So that getting started with circles then

yeah I don't want to uh explain too much about about circles. We have a booth outside also Martin Copelman uh did a presentation about this with much more detail. Um so just briefly touching upon uh circles and what is about is a uh experiment with a new form of money. So uh we we think that that this kind of so circles is a more fair economical system where issuance is decentralized and not just the power not not just possible or centralized by a few uh central banks or institutions rather in circles um anybody can bend their own circles and um at a fixed rate. So um this this gives power to everyone basically.

Yeah three basic rules circles issuance is fixed at one circles per hour uh unconditionally uh everybody can m their own circles at this rate. Then the second point is the 7% demorrage per year. So you can see it as in the beginning um the supply since since you you have very little circles your balance your supply is low. uh you just mint. So your supply is just going up. As you this increases, demorrage becomes relevant. So this kind of plateaus here at roughly 100,000 circles and um as you either start stop using the platform or you die or it then um it the the your balance just goes down because the just applies and are not minting anymore. So

it's kind of a plateau and then it goes down. Um yeah, please refer to the documentation for for more details here. And also the third point which is the most interesting also in my opinion is this selective trust system. So what that means is that um if user if I trust Alice for example inside inside circles that means that I accept Alice's tokens as if they were my own. So at a ratio of one to one and this trust system is what gives uh um civil resistance to the network. So that means that if I don't trust another party then I'm not I'm not um prepared to accept his circles. So if somebody else just has 10 bots here and I see activity on chain or activity on the circles network of uh those bots just transferring their circles back to

this person, I can identify this person as not being compliant with the rules of the network. So uh I don't trust her. I don't trust this person and and nobody should trust this person. So it's kind of a social civil soft civil resistance. Yeah. Yeah. And and just concluding the the the idea for circles, there are uh three main entities in circles. Of course, humans which are able to mint circles and trust other participants as well. Then there are the groups which are kind of facilitators in this entire circles ecosystem. So they allow for easier exchanges of money. you don't there is not the need to have a direct persontoperson connection in order to to

for the trade to occur through what we call the pathfinder but you can also have uh trust between groups which facilitate this kind of transfer there um and they can they represent just aggregations of humans so there's a group for ETC there's a group for Paris for Berlin and so on and so forth um and the third entity the third entity type is called organization which is the relevant part for this talk here and an organization can be the ice cream parlor can be a supermarket or it can also be an AI agent and the idea for the an organization is it needs to provide services for the network. Um and also important an organization cannot cannot mint circles. This is reserved for humans. Organizations are only there to are just nodes in the network that

provide services in exchange for for circles or for money or something like that. They cannot mint. Um in this case they provide services of course uh forecasting services because that's what our trader agents are already doing. Um and we leverage the the circles web of trust as a reputation system. So in this case if you can see on the image on the top left the deli agent um again is offering forecasting services and Alice trusts Delvy in order for it to um provide the forecasting the forecasting results that she's after. So if she's satisfied with those with those uh forecasts then she can trust trust the agent and other people can also look at that and say okay if Alice is trusts the

agent then I also convinced that the agent provides some relevant services let me trust it as well and that gives a kind of reputation score review score for the agent similar to the Uber review scores and some other applications as well but supported by the circles network with lots of users and uh incentives for trusting untrusting people and Yeah to summarize trust level acts as a proxy for the agents performance. Thank you. And yeah my my final topic here is uh ideas that we have or how we are planning to improve upon the agents prediction accuracy.

Um yeah here are some thoughts. Um again this is the same chart as I presented earlier um of the accuracy uh over time for all the agents. Um I would argue that the accuracy is pretty decent. So uh we're doing a good job job there. But we also observe as we manually inspect the predictions that the agent is overconfident. So uh for some of the for some of the the predictions that he's making is claiming that his confidence on that prediction is very very high. However, it should not be high for many many reasons that we also can can uh demonstrate which is simply it's not possible to predict with that confidence because we know we manually inspect the events and so on. Um and we're trying to solve that. We're trying to make the agent have a better understanding of how

confident uh it should be about the prediction. And the two approaches we're using for uh basically getting this prediction or this estimate of the agent are either asking the LM directly for a prediction plus confidence. So you get two floats. That's the easiest uh the easiest approach. Or you also take the log props as a measure of confidence. So that means the log props are uh just what the LLM also outputs as the next token because we're asking it for if you believe that this outcome will resolve to yes output Y the character or output N as also the character. So getting the log props there just is a is a measure of how confident the LLM is in in the prediction it just made. Um and we use

that as confidence for the further trading activities. Um a much more interesting approach uh would be to call the LM for and ask it for a measure of the distribution of a measure of the of the confidence that it has in the in the in the prediction itself. So let's say we asked LLM um we are trying to predict the Tesla price at date X and we asked it for the 5% and 95% uh probability distribution and just those two levels uh we're able to fit a parametric distri distribution using those two values for example a beta distribution and in this case uh we take the mode of this of this distribution that we just fitted and then we use that as our as our uh

estimate for example and why this is a lot nicer than the previous two approaches is because you have a a concrete measurement of your confidence. So you're able to say given the pro the the distribution that I fitted my data with. Now I can tell tell okay this is the the deviation or this is sigma. I can take any kind of measure from this distribution and also I can update this this as this is now my prior. If I do similar predictions I can also update this distribution with the uh additional additional uh yeah the resolutions that I had thus updating my my confidence interval.

Yeah that's what I had to to to tell today. Thank you so much for your attention. Um happy to take any questions. Yeah, question here. Thanks. So, one of the most efficient prediction markets and most mature ones are sports outcome prediction markets and now we have like fully onchain sport books such as overtime.io and stuff like that. So is it possible for your AI to eventually also become like a automated trader on those type of platforms or is this that like a gray area?

Um good question. Uh we are so this accuracy that I reported here um this is across all kinds of markets sports we also place bets on sports markets, politics, crypto, any kinds of any kind of market. Um, and this is across all of those markets and most importantly across all kind of how far away they are from the resolution time. So it's very different to try to make a prediction one minute before resolution than to I don't know 15 days before the game for example where some player might be injured for example. That of course affects the prediction. So unfortunately I don't have a an answer for you on how effective that could be and what is the performance for those specific predictions. But I guess uh not really I guess but the procedure is the same.

Right. So the four steps of research and doing the estimate and betting on those markets is definitely the same. It's just a matter of how good your data is and how predictable the the event is. Then yeah, just a quick one, the Tesla chart inspired me to ask this. So is it possible then for this AI agent to actually become a market maker because it can accurately kind of price distribution curve and stuff like that. So instead of like a trading agent, it could become like a order book market maker of prediction markets, right? if it can become a market maker because I'm predicting the price. Is that the reason? Yeah, because it it it kind of predicts the confidence of the price of Tesla for example in one month and based

on the curve it can actually fill the order book based on that curve for example on poly market. I'm not sure market maker is the best behavior for the for the bot for the agent in this case. I mean for in this case as we are predicting the the Tesla stock for example we have a price that or we have a distribution and the most likely price. So the way that we would act in this case is simply to to place a bet in the direction we're viewing that. So um and that's the estimate but of course having this kind of prediction then you can go on both markets and have and uh profit from the spread for sure.

But uh that's not really the goal of the agent of that what we are trying to do. Does that does that answer your question? Yeah. Thanks. Yeah, I was uh just curious about the return uh from like that chart. Yeah, just uh if you could speak to that this one. Yeah, but I'm curious about the return. I mean, this is the accuracy. So, I'm curious about return because I mean, you could Yeah, that's the chart I didn't include. Uh it's on the Doom dashboard, but um so the Yeah, the return is not bad. Let's let's put it like this. Uh I don't have the number in my head, but it's it's definitely it's it's positive. We we don't return

the APY overall. It's just the last 30 or 60 days. So you can check it out. So I don't want to provide false data. However, what is not in there is the inference costs that we we incur using OpenAI which are very very not very very high but it's I mean for every prediction we do have inference costs which are not uh taken into consideration for Dune. But having said that there are we're not optimizing for those costs as well. You can say run local models. You can say b b b b b b b b b b b b b b b b b b b b b markets together. You can use cheaper models for some markets, more expensive models for other markets. So there are some tricks, but uh please check out the Doom dashboard. Yeah, thank you.

Any questions? So So you say the agent were often overconfident. Did you try just simply calibrating them? Like if you you can put a chart of either Asian predict 40% uh and on average when they predict 40% it's only 20%. So in this case you you can just calibrate them uh as long as it's a bias like bias are very easy to to correct. Did you try that? Not yet. Um we are still doing one of the easier approaches of just asking for accurac for the the prediction for the estimate and then the confidence and uh we're just implementing this new let's say predict distributions and then you're

you're right we do have the data for the calibration because we check on our estimate that we do versus the market resolution and we all you observe there's already a bias so we expect to calibrate that uh by the by the data as well. Yeah. No question. Okay, thank you. [Applause] [Music]