TE Commons

Transcript

okay we are ready to take off for AI power token engineering with Rohan I had the pleasure to work with Rohan on a proof of concept how to apply llms to token engineering super exciting um and I'm super excited to now show you the results but before we get to that just make sure to Mint your nft for attending this first ever token engineering full day track at ECC it's not only a nice souvenir I mean you'll be the first who attended this event in the history of token engineering it's also yay it

it also grants you an additional weight in voting for grants for token engineering public goods T comments will run another grand round for token engineering projects in August and with this nft we want to leverage the knowledge in the community so from the people who are studying tea fundamentals from the people who are interested in topics like this and who can contribute with either talks or with being here with working with us at the bar Camp don't forget 20 seconds on Saturday in Paris pick a flyer there's a full day of interactive workshops and just

networking for the token engineering community it's free you can still register and I hope to see you all there and meet your nft all right state is yours Rowan okay if this doesn't follow me please let me know um so yeah hello everyone um welcome to this talk on AI power token engineering I'm really excited to share my work for the last few months but before I start my talk I want to specially thank token engineering Academy for championing this field and organizing this amazing track I've learned so much from Trent darkem Roderick mark gauntlety on and especially Trentonian recently just

mentioned that llms can be used in the in the process of token engineering and it's so cool that I can actually showcase the work that we've done and the Agents that we've built to actually Aid in the process of token engineering um okay we can skip this uh so basically I'm Rohan Mata I'm a data scientist I'm a token engineer and this project is a proof of concept that has been jointly funded by both the token engineering Commons and token engineering Academy um I have the pleasure of collaborating with Angela the founder of tea Academy and uh Roderick our turkonomics advisor I will be covering the overall goals of this project discuss the token engineering and modeling process um I'll go through a tiny section of

explaining what large language models are and how we can build agents on top of them uh I will also demonstrate the basic structure of how llm agents can be built but more importantly I'll actually demo what we've built and how agents can be really used in a real token engineering model um but before we get ahead of ourselves I'll just give you a brief context of what token engineering is I'm sure all of you guys know this by now but it's uh token engineering is an emerging discipline focused on the design validation and um okay I'll just look here design validation and optimization of token based economic systems uh modeling and simulations are a huge essential part of the token Engineering

Process uh playing a pivotal role in understanding testing and optimizing the various aspects of the token ecosystems so they enable us to run what-if scenarios explore relationships between parameters and metrics that we choose they provide somewhat like a Sandbox environment on which we can run experiments for the ideas that we are stress testing by leveraging these techniques we can usually make more informed and data driven decisions in a nutshell I would like to just go through the process of modeling this is not a linear flow but uh and we've heard this again and again like everyone's using a different model and different structure of analyzing and building models themselves but the general structure is that you first represent the real token

ecosystem using some kind of assumptions some um some kind of metrics and some kind of parameters through which then you're then you go and justify your assumptions by actually getting real-world data and making sure that the numbers that you are guessing for the system to be is somewhat accurate then we change parameters to a b testing We compare the outcomes with each other we do sensitivity sensitivity analysis Monte Carlo runs and other such runs to then do statistical analysis on top of the results that we get um a lot of these tools it's interesting to see that they are modular and this will play a role in what what we are building in the future

um so very often even after having built a great model you can find yourself standing at hundreds of parameters you have no clue okay what to do next what question are to ask the model what scenarios to test and also how do I get it done because every model comes with a different structure and now I have to understand what the assumptions were how did they actually structure the model how do I change the parameter all of that is up in the air depending on how the modeler has built it so I hope okay I'll get to it but what we discovered using llms in this process is that it's really useful in managing simulations and drawing conclusions from the simulations that that are already built they're really effective at searching

through even and pointing to contextually relevant information from databases so if you have a huge set of information that you want to go through it can it's really effective at searching semantically through what the closest meaning is in in all of those documents and fetch that information for you they're really effective at analyzing data frames and through which we can draw conclusions and eventually make decisions our hope is that once we create a structure for agents it will also standardize the process of Team modeling and simulations because it'll make it so much easier to interact with our token engineering models giving a language interface to them um I want to highlight that we don't expect llms to be able to build models

from scratch nowhere in the near future we can see that happening because there are too many creative decisions to be made when you're building a model you have so much context about the market that needs to be put in and I don't think um even with higher token limits within our national language models we'll be able to ever get them but the general structure that we are following the wider uh wider picture we have in our mind is that there are four sections to this one of the models that took of token systems that token engineers build on the other side the AI assistants that provide human language interface to support the simulation process we have um a data benchmarked by humans by

actual subject matter experts which is offered to the AI as a back-end memory and I'll show you how that works ultimately our goal is to use all of these three nodes to finally get to something that's verifiable and auditable for our token systems um in order to actually showcase our results we were we pick point we pinpointed one use case that would be relevant to most users which is the a token sale scenario or like fundraising wire tokens and this is definitely something every crypto project is dealing with at some point so I think it was very useful to find a model that Roderick McKinley has built we used his tokenomic sales calculator which is a spreadsheet model converted

it to a python model and we we built agents on top of it but the general structure is pretty similar it's like there's allocation charts there's a vesting schedules you just uh the couple of things that you have to track like token price fundraised uh the unlock schedules and the token allocations are very stakeholders so very standard like we've all seen this everywhere in white papers um I'll quickly go through the basic structure of the background before I demo the agent themselves that let's say you're a startup and you have to decide how much uh how much fundraise amounts you you need so a very standard set of questions you would have is like okay

what's my monthly cash one let's say 115 000. um you have to plan for contingency you have to cover a Runway of let's say 36 months and let's say there's some listing costs that you've uh that you're looking forward to so when you add that up very simple calculation you come to a total sum that you need around five million dollars very simple calculation from your end this is the money you want but the investors take a different completely different perspective for them for investors they majorly will look at FTV that's not the only thing but I mean we are simplifying for now uh FTV is a very important thing that they look at which depends on the token Supply that you decide also depends on

the token price that you're selling them at the percentage allocated to them and all of these are kind of interconnected and what you realize that there's a multi-dimensional problem there's no easy like oh I can just change this so I can just change that it's all connected together and this is these are the kind of problems where AI agents can support us with parameter optimization after giving them a goal that we want um yeah I have to highlight this since Roderick is here and he's been talking about this a good deal in his talk too that all this these are all like human decisions at the end of the day there's no objective right numerical answer to any of this and it comes down to how new Newmans negotiate who has the uh by gaining power Etc but um given certain constraints we can

actually at least ask the model certain questions and it can guide us through the process of it um I can jump into oh I'll have to switch this yeah so I can jump into my demo now um also how many of y'all are like coders do you guys understand like um okay one two three all right cool that helps yeah that's okay okay good perfect

um okay so basically we went through filling all the parameters like fundraising goals token Supply etc etc we got to a vesting schedule something like this um so we are using this as the base model to actually explore agents and uh this is basically a coding environment where I can write my question imagine this as like your chat GPT chat box where you can just pretty much ask any question that you want the agent itself has information about the model that we're using so you can ask something like okay what is my token Supply not very impressive but um wait let me just see this yeah so what is my total token Supply if you ask this

um we'll see that it runs through a set of codes it's entering a new chain some weird stuff is happening in the middle and it gave a final answer at the end the total token supplies 180 million which is what we had fed in so it's cool okay it can access the stuff that we've put in we can also ask let's say whatever whatever rest is like okay what is our thank you co-pilot um yeah and again same thing uh went through a series of steps did some mysterious stuff in the middle here and got to an answer and I will showcase exactly what happened in the middle and this is 5 million by the way the same

one that we had accounted for um uh this is very interesting this is when stuff starts getting interesting let's say you're just you're at a stage where you have to now decide what your allocation schedule should be uh so your allocation amount should be for different investor groups and you can if you don't know you can just kind of ask like what is the average allocation my sector and um it's actually has access to a memory database so the way it works is it semantically searches through a huge set of vector storage and finds you the correct article or the research that we have curated uh picks it out uh sends it

back to you currently it's hard coded so don't overthink it but basically if you read this based on this similarity search on the benchmarking documents the closest match I found was on page 30 of PDF whatever and it was able to pinpoint the answer so it's very interesting to know that in a normal large language model the issue is that there's no traceability back to the data it's very hard to figure out if you ask a question where did it learn this information from but we are not building our own large language model what we're doing is using open AI or any kind of large language model to um to do a set of tasks a chain of calls to itself while giving it tools and memory and I'll showcase exactly the structure we are using so what's happening step by step is that we are

first asking a question out here what is the average allocation in my sector it understood what it needs to do it needs to figure out figure this out in the Benchmark data it acts it took an action in the real world accessed our database a random similarity search gave an answer and then finally gave an answer back to us in a human readable format which is something like 26.6 for private sale 1.1 percent for public scale this is the ranges that I found and could took an average of it um so we we start seeing that it has access to some Benchmark data and memory and it is able to fetch for the relevant information based on the question you ask there's another thing like it can also give you suggestions about the

things to do uh so you can ask like let's say um let's say your private Town investors uh are very they're like okay your fundraising valuation is too high you need to make it 15 lower so we can um we can just type it on my it remembers this is a percentage percentage higher than my target what should I do boom it's able to understand what the question is and you can see that first it understood that based on the question what it needs to do is first find out what is fun what is FTV connected to and it figured out that there are total

token Supply and token price are the two things that change your FTV uh then it figured out that it needs to use an Optimizer to reduce the empty by 15 while keeping these two like finding the values for these two and it gave you an answer like oh we can either or this is the final answer reduce token price down to 0.085 or reduce token Supply to 153 million to reduce FDB by 15 um again forget the the actual what's really happening and that's cool out here is that it understood the question figured out which tools to use within the skill sets that's available to it uh chain them together in a way that made sense and actually gave us an answer in natural language so it's very

fascinating here um another thing we can possibly go into how much time do you have okay I have time um what's my token Supply at okay so yeah you can query the data set that your cat CAD model has already built so you can ask questions like uh what is my token Supply at month 12 yeah so something at this like what is my token Supply at month 12 as a percentage of the final token Supply so um this for this like I what I would do as a data scientist I would first locate the 12th uh row like the month and then find the sum all the tokens on that then go to the final row sum all the tokens divide this let's see how

um probably the same yeah and it also understood that it needs to remove one of the rows uh out of the column which is interesting that's probably the month row and we can check third oops not this one but yeah so it's actually doing like straight up data science stuff and uh answering questions based on what you have asked yeah so the final row was month and it figured out that it needs to not include that in the circulating Supply calculations figured out what the value was for month 12 observed it had a thought okay what should it do next uh used an action on the python shell that it has access to run the pandas command uh got the answer 180 million which was

our total token Supply and finally gave us an answer in natural language so again what's fascinating is it has access to so many tools which it can use based on what the question is being asked it can change these tools together it can fetch memory and then chain tools on top of the memory so that's where like the real power of AI you can start seeing that it's you're just giving it a goal like this is what I want you to do it's because of all the intermediate steps based on the skills we've given it to get to the final answer uh I feel like I should show you all how we've done this because it's really fascinating let me try so we followed uh two major uh prompt engineering breakthroughs that happened recently one was

um one was a Chain of Thought reasoning that people figured out that we can force llms to Think Through the task way more than it normally does you can force it to have a thought then take an action then observe the results then have a thought and take an action keep doing this loop again and again this forces an llm to have a much deeper understanding of what it should be doing and the second thing that we did is I don't know if you all heard of baby AGI but they came after this plan and executor Loop which is what we are following we built a planner agent all it does is it knows what information it has access to the memories which is here it knows what skills it has access to which is like a python shell or a SQL shell it has maybe access to a Wikipedia

search so you can fetch information in our case it's a database um or it knows what it has it all it needs to do is make a plan if this is the goal step one step two step three step four and then the executor agent has this thought observation Loop which is what we saw here um out here so it has a thought I need to find the token Supply at month 12 and then compare it to the total token Supply and then it executes every set of tasks one by one so what we are doing here is not building a language model what we're doing is leveraging the already built language models giving them memory skills putting them out in the real world giving them a goal to achieve and that's why we call them agents there's an ingestion engine understands the input figures out which agent to use

plans the set of tasks goes through Loops of thought observation action loops and gives you an output at the end what's really cool about this is that to me like within data science or like even token engineering we can isolate the kind of tasks that we generally do something like sensitivity search parameters sweeps a b testing we can abstract them away as skill sets and teach an agent how to go about it like if I was a and that's probably what I'll be working on next now like is like build a data frame of this is the question these are the things I would have done I would have first done an A B test then based on the result I would have done this then done that if we can make an entire data frame of how I would have gone about answering the question

we can dynamically prompt engineer really powerful agents we can figure out with every model not even like um in our case we are using a very simple model but you can have really complicated models where it's understood the series of steps to take to answer questions the way our data scientists spread out uh it's still uh there are a lot of limitations it sometimes makes a lot of mistakes in picking tools sometimes make makes mistakes and actually sending uh the relevant information to the tools so it can't run the functions that well it happens but to me it's like the as the language models keep getting better we will be able to have a higher token limit through which we can solve this issue and in conclusion oh can I share the

slide here is that okay it's working good um in conclusion what I just wanted to say is that this Chain of Thought reasoning method is super powerful and Trend and everybody else like uh that spoke about it like Trent was going from like the bottom up like using AI to go all the way down to the ground and solving the um questions from down below we are going we are attacking from the top what we're doing is making these tools easy to use because the issue currently is that we have all these amazing tools that have been built like card card token space but no one's using them so maybe if we can build these models and give language interface access to everybody this possibly helps a lot of people use the tools way more make it more accessible to people at least the

easy stuff can be automated away and um yeah if you want to see how we build these agents there's a drag and drop method to actually build these chatbots uh meet us at the back cam we are going to Showcase how to go through the process of building agents how to give them memory skill sets and um yeah this will be super exciting you can uh you can scan the barcode here and I'll be giving a workshop though yeah thank you [Applause] we have time for one or two questions thank you very much for a great demo a really great job guys one question with this specific prompt about lowering fdv

so the agent figure out that he needs to either change the total Supply or the price and it was based on the skilletry that you needed in upfront to code okay okay this is how it works so what will be the the effort or I know next steps in order to bypass this step of providing some upfront skills because we can imagine right if for example agent who if agent would be able to write cut models or do something with these models then if we will need to in up front to provide the

the whole skills tree to learn his I mean that would be pretty challenging so what do you think how we could bypass this step in order to basically not provide any skills tree and still the agent will be able to deliver the task so there are a couple of approaches we can use uh first of all good question I do think that one way to bypass it is first not so in this case we had very specific uh skills that were used to do spec vectors not very impressive honestly but I think what we can do is abstract away the steps that data scientists generally use to get to an answer and build a data frame of question so um that's the current

current idea I have which is that you can dynamically build few short prompts through which so it's called few short learning which language models are really good at so we have a data set of all the tasks that I would have done as a data scientist on this model it fetches that uses the a b task sensitivity search based on the question um so that's generally the approach everyone's doing I I Aviva at this hackathon yesterday crypto AI hackathon where at least two people were using similar approaches for their problems so it looks like that's the state of the art practice for now um but honestly every month you see a new paper coming out in this field and something completely fundamentally shifts we figure out a new way to use these language models in a much more

effective way so that's my current understanding of it but I do feel like if we abstract away the basic data science tooling it should be applicable to most models and we won't have to redo them for every model yeah uh second question maybe yes thank you hi uh Malisha here from streamflow we help teams with the token investing So This is highly relevant for us first of all very nice talk and seems very well thought through and looks like these are The Cutting Edge tools that you're using my question is how can we access these tools and how can we use them as well happy to talk

seriously let's catch up because I think there's a lot of value in combining data the aggregation of data benchmarking models for running simulations the AI on top so there's a lot of potential here distribution the whole flow exactly okay but these are not publicly available tools not yet but my idea would be that if we can figure out a standard practice through which token engine modelers can build API access to their models maybe we can put a language model shell on top of it where these agents are available that could be the best way to

standardize and like make these open source even if it's possible but there needs to be standardization for the modeling process first and but yeah we have to discuss this yeah okay thank you all right we have a short break and then we continue with two discussions actually how to establish token engineering at your startup we have now heard about verification tools verification approaches design approaches the big question is how you can make this work at your project integrate this in a roadmap for a project and get things going so stick around and we'll see you in a moment foreign

[Music]