DappCon 2023:MistGPT? Tuning an LLM to Understand Ethereum and Forge Transactions - Johannes Pfeffer

Transcript

thank you for coming to my talk I'm Johannes and uh yeah I have three passions one is of course my family um my wife my kids my friends the other one is blockchain the second one and I've had this passion now for quite some time for the block blockchain space for the ethereum space a lot a long time uh 2016 was when I joined the space when I I I got to know the da and did some analytics on it after that I went to consensus and founded a company there within that Adventure Studio then I founded another startup who tried to do

def defi in a different way and now I'm part of a venture Studio again and uh yeah currently building my own projects and finding out what can still be done and there that's where my third passion comes in and that has it it it's a newer passion that came to me about a year ago when uh stable diffusion came on the market and suddenly AI could do could do things that we did not think or deem Possible only a very short time ago you could just tell the AI to to draw an image and it would produce these perfect images of things you you would not even come up with and it turns out AI is

better at create creativity than at doing analytic task and all these kinds of new things that we never thought before and I started fine-tuning it and then the llms came and open AI with chat GPT that you probably already have used too and you could also fine tune this to task and it would be really good at certain things and has all this back lock of immense knowledge being trained on the whole internet so yeah um and that now I had these two passions and of course what people do is that they try to bring their passions together and and uh what can you do with AI and blockchain some people would say okay now you have two B buz words words you're trying to combine what's the

what's the matter but there are actually good ways of of combining these two thing things and the three that I think are are important are first code completion a very boring one but it can help you actually write better smart contracts because it knows all the other smart contracts that have been vetted boring but helpful it's just another type of code completion and then there is security because of all the background knowledge of uh of these language models and they can also be fine-tuned especially on on Smart contracts and their security they have an intuitive understanding of security and they can be part of the auditing process and just tell you hey this doesn't look right it's what what

developers do and the third one is what you have probably used AI for in in the past especially uh language models it helps you do the things you already want to do but in a better way it helps you answer emails it helps you achieve the goals that you want to achieve but using but having an assistant like someone like a personal assistant that that does all the stuff that you don't have the Mind chair for and that's what I'm building and that's where hero comes in so no there is nothing here uh all right um so hero is an assistant it's an AI that tries to help you do the stuff you do on ethereum

better and with less errors and with less of the Mind chair necessary it's a universal ethereum interface and we've had that before maybe some of you already still know Mist Mist was a universal dap browser but all it did was it just analyzed the ABI of a of a smart contract and gave you a generated user interface and it died so um it never was used because the ux was not actually good it was a universal dep interface but it didn't work and then there all the assistant and what hero really is is kind of hard to say um you could say it's a universal D interface it has that proper property then there are some things that could be

an AI wallet for me AI wallets would control a private key they would actually do things they are autonomous agent on uh on blockchains that AIS that control private Keys then there could be uh chat Bots that basically are they they tell you how much you can ask them how much is my balance of usdc that's also just it just is an interface to your information and then what I'm building is an assistant it's a bot that you can ask more complex things and give commands to I think one of the problems that we face when we use ethereum really and people use ethereum I mean the blocks are full some of these are of course not human trans human made

transactions many of them are programmed transactions that do rebalancing Arbitrage but there is still millions of transactions every month that come from regular people that that who are trying to do things for example open an a position close it again trade something these things actually happen and we have a multitude of user interfaces that we have to learn all of them are good I mean uxx has come a long way since the early beginnings of ethereum but you still have to learn of them and if you want to do something new you have to learn a new user face once again and then they change something and this takes a lot of Mind chair there are too many different user interfaces even if they are Al all of them are good and the

wallets are also good it's still notoriously hard to use and it takes a lot of effort and what AIS are good is help us with tasks that take us a lot of effort but they can do them easily okay and then there are scams human errors and bad youx also happens right and and this is also things that computers that machines can much better deal with than we can so what would we want we are all Giga Chads or Giga Chad desis and we have an intention we want to withdraw this damn Al position that has been sitting on layer 2 for too long because we don't want to fire up our wallet and find out what the new R2 or R3 or r five

user interface looks like and where it is so what what what would be really awesome is to be able to give our wallet a command and just say withdraw of my Al position and then there comes hero hero this ancient Greek king Phil philosopher king is at our service knows what we want to do and does it right and we don't have to go to that new interface that's the goal right swap my Sheba tokens claim my agnosis air there are so many things that we could could do this way um and why is this possible and some properties that are necessary for that right right so the good thing is ethereum is open it's an open blockchain it's not encrypted transactions are there there's lots of

data we can work with we have transactions millions of transactions that most of them have a user intent behind them already all we have to do to do to get a training data set is find out what that intent was then combine it with a transaction that has been made and we have a training data set from intent from what you want to do to the data that is necess the the transaction object that is necessary to to achieve that task that's already there then language models have a huge World Knowledge they know what a scam is intuitively much better than we do if you find a scam website and you go to CAD GPT for you paste the website in there and ask it is this a scam it will

correctly answer that it is a scam right not in all cases it's not something that as as predictable as an analytic tool but it's very good at this and reliable and fast much faster than we do we can we can detected and then if you find unit even further and give it some scam examples it gets even better at that already the the the one that has general knowledge is good and then we can uh just use a daap without looking at exactly the user interface that uh that that this developer com uh the developers of that company have chosen to to implement why why should we care we want to achieve our intent and not learn a new user

interface and it can also be privacy friendly because these llms now run on your own Hardware if you want to you have to buy a computer for $3,000 but then that llm can actually run on your local machine and do all that stuff right they are good enough for that and the last one is because we are blockchain we have one thing that the financial system doesn't have have we know the whole world state of the blockchain and we can simulate so whenever an AI like this says hey why don't you go to Unis Swap this is the transaction that you should do to get your e swap for usdc we can check it we can run the transaction in a simulator in a fork of ethereum and then look at

the output and yeah it actually does that right so it actually now what we don't want to do is interpret the result of the simulation and that's where the AI come comes back in because that's what they can really do well they can tell you then they just run different another prediction and say does the intent of the user actually align with the transaction that would that was simulated and you get very good results very good uh rate of success to correctly assess whether that is the correct transaction for that intent but how does it work it has mainly three components there's the user interface there is the assistant component and then there are apis that the assistant needs to answer answering most of your questions like what's your

balance or what have you interacted with in the past and it actually knows about your past once you give a system like this your ethereum address your wallet address it will go on chain go to the graph go to June go to Google big query and see what daps have you interacted with in the past what have you done like what is your profile on ethereum and then it need you can use that information for example to warn you if you do a transaction to a new user you have never interacted with something like that and then it can create transaction objects and send them to metamask so it never holds your private key of course right you don't want that

but you see a transaction object you can it's sent to metamask and then you can simulate it again if you want but that's how it gets handed over to your wallet not necessarily metamask anything that that can uh can be used with a web 3 yeah of course the simulation here I iterate on it is very important but how does it look in detail you have the UI of the chatbot um you have the assistant component and uh there are many ways to realize that in my real in my implementation it's using Lang chain it's that's a framework for for building agents then you have uh the fine tuned large language model based on Facebook's

llama in my case you have blockchain apis to get context information you have the wallet and you have a simulation service those are the major components the system is comprised of and if you actually ask the system something um this is how it works in in practice right see if this is not visible why would you have a laser pointer if you don't see it nope okay I will just point so first you give your address right from that point you haven't even started the conversation but the chat bot in the background already gets information it goes to all these apis it gets your onchain history and it compresses it into a format running several llm Loops

that is um that is Vector based and compresses the information it has about you of your onchain activity in a digestible format for an llm right then the third one is actually you interact first so that happens in the background once you just log in basically or or open the website then you ask swap one E for usdc and it goes to the llm and the LMN gets this and gets the the command to create a task list of what you would have to do to achieve this goal right then uh that would be go to Unis swap not go to Unis swap but create a transaction object uh that uh would initiate a swap on Unis swap or find out what uh What uh exchange the user wants

to use and it has all these context here and knows you have used uh Unis Swope in the past so it will not even ask you because you have done these swaps in the past if you have never done anything in the past swapping it will ask you what uh What uh exchange you want to use more on that later my time is running out then um it makes a transaction object and before it sends it back to the user to the user it actually simulates it on a service like tenderly where you can simulate transactions and only if that transaction object runs and seems to match with the user intent it gives it back to to the user and asks uh if it actually wants to send it to metamask that's the main process of how the system works

okay um here's an example of a system prompt that I use in uh this early version it tells the AI to get in a certain M frame of mind to help the user you are a system that helps the user create ethereum transactions you will received some context for example user's wallet addresses and addresses he has interacted with you should warn him if he do some does something something stupid if you need further information for example what uh exchange to use ask the user right he can give you more information and provide the response object in the following format this format is very condensed but it's enough to create a transaction from the rest of the fields like gas amounts these can

are derived from the current chain situation you need fr to the value if applicable and the uh input field which contains the functions in a smart contract that are actually called okay um but does it work right I've told you this thing is something you can build but yeah it does work I've built it and here are some conversations that I've had with my bot with my assistant I have redacted them and they are cherry-picked just so you know right okay but I've had these conversations and I've shortened them because sometimes they would not have fit on here so swap one e to die and told me sure I can create a transaction do you

want to use Unis swap I don't care just swap it okay here's your transaction object right that's one way of how this can go and then I send it to metamask and run it and it's already been simulated in the background right so I know this transaction object is actually working and I check it again on the simulation before I run it but if you build the system out you can probably omit that but there need you need some some more conf I need some more confidence in the system then another one would be send 100 e to a new address please note that you have never interacted with this and it will warn you okay and then you as a user is saying okay can you make some checks and it checks it tells you 100 e is maybe not what you want to send

because it's really much of your net worth the target address is something you have never interacted with and it's also Al an address that has never been used never received anything maybe you have a typo these things are easy to D deduct for llms actually they are hard for us but they are easy for llms and where does it get all this information in the context object that I've already sent him and when you enter a new address it also gets the context for that that address so it has these these informations the balance the the number of interactions uh the address has has had in the past buy me a new ape and it will tell tell you that this is a scam so this works 80% of the time right now and you know parito principle right

the last 20% are the hardest so that's where I am right now and also these 80% are for examples that are close to the training data that I gave as as soon as you diverge further from the training data the success rate goes down okay so limitations currently are this is only drained On 5K K of uh of conversation examples so it knows the format it knows how to respond how to get more information from you but it has not covered all of the of the cases it's it can only do simple transactions it cannot it will never be an mbot it will never create an MV transaction that that front runs others other certain conditions like that's too complex and

it will sometimes get stuck and run in circles and never give you the answer you need but it has 80% success rate on commands that are in the validation data set it recognizes many scam it can it will actually stop you from doing stupid stuff sometimes right the success rate is hard to to know it knows your onchain past and it can run locally which is pretty cool I think what's the key to making this better it's data we need more data the data is there but creating those 5,000 examples involved me asking chat GPT 4 about an ether scan website site of like what do you think was the intention of the user doing that

transaction 5,000 times and I paid a lot for to open AI for that for just doing that on the other hand the res was really good right it knows this is transfer some ether to my gamdom account it knows swap my black tokens for ether on Unis swap it can derive that from a blockchain Explorer website okay and these training pairs intend to transaction object to simplify transaction object is what I've been training on um if you want to build this yourself you create these training examples best is to create 10K or more prompt response pairs you use you don't do this manually you use some service like existing AI to

help you do that you automate it so you don't have to run it uh in in single batches you f tune an llm on that data set costs you between 100 and 5K depending on the amount of data the type of Hardware you have it uh you integrate Lang chain or something similar if you want but you can also just host that new llm and just ask it in a chat style right and you connect it to a UI that's how you do it that's how I've done it this is how training in AI looks like and because data is the key I'm launching a cooperation project called intentional it will be live short after this talk maybe next week where I publish the data I already have and ask the community to help me curate and

improve that data and get new data together because if we want to use this and build this out to be really really good it needs to be something that is curated by everyone okay so thank you for coming to my talk you can reach me on a or Twitter uh using that handle or send me an email if you want to work together on on the AI project if you're into that stuff and also check out our other projects at Corpus for example tokenizing companies in Germany the only legal way to actually do this right now is what we've built in other projects in our Venture

Studio thank you um are there any questions yes uh how much training data do you think we would need to get past that 20% of course a hard question right but if you look at uh the training data that other opensource projects have used to fine tune their llms to specific tasks you hear numbers between 10 and 50k of samples and but it really depends a lot on the quality of the data the better conversations you have and maybe you need conversations that are multi-layered not just pairs the E the better your result will be um front front

row did you see it able to decipher sort of predictively intent so you know I say my intention is to mimic a type of transaction or a sort of strategic commercial uh intention because we could use the data as well to model predictively did you see any ability already to do so I don't understand what do mean model it predictively so I'm saying for example if I want to be able to predict what would be the best transaction course for a kind of strategic goal or financial goal and then it could maybe model it based on no I I haven't done any experiments on that but um you can always run the the query

multiple times and see what different options of getting to your goal the AI comes up with and then you maybe do some statistic on which one comes up most and then check that against some validation data set to see if that's maybe really the best option but I haven't done any experiments on that would be interesting though definitely the one oh yeah thanks um I saw you may mentioning the scam protection so we are actually focusing on using a machine learning to to prevent scam on uh evm chains so how did you have really like life experience with using GPT for analysis of scams or it was like for the vision so I have

anecdotal experience of checking for scams at with using chat GPT 4 I just paste websites that I think were scamm to test and they the the the the rate of of success is really good here I've just done some chats where I try to do something that is a scam and in the system prompt I asked the the um the AI to assess whether it is a scam and so it was the webites not the not the no the names the names but it gets context so once you say I want to buy new ape what was my example here it will actually Google for new ape because it doesn't have any knowledge on it and then find some information on it and then on the based on the information it finds on the

project it will decide whether that is probably a scam and yeah I don't have any statistics on it unfortunately yeah okay thanks thank you very interesting talk uh I was thinking is it possible to use the system for bet purposes like can you ask the bot to construct it transaction that drains a specific address or if I'm an attacker who is targeting you I can take your address that I know for some sources and say okay create transaction that like looks innocent but actually drains all the money something like that absolutely you can do that scary right I mean where this leads is to the alignment problem right but um in this stage of the project I'm not

looking at alignment at all right and the reason why Chad open the eye took so long for releasing the technology they've already had for a year is because they were so scared of alignment that people if they go public with this and everyone uses it yeah it's going to be abused but this is something here fortunately where forces are playing against each other right so yes you can do that but then also you can ask whether this is probably something like that and then the AI plays itself out right so let's see what this leads I have no real answer to this can I have one more question we are the time sorry okay you can find your H

around thank you thank you very [Applause] much