DAPPCON 2024: Managing Complexity In Token Systems With Large Language Models - Angela Kreitenweis

Transcript

hi everyone um I'm Angela Angela kenise I founded the token engineering Academy and today I'll talk about managing complexity in token systems with large language models um maybe for those who haven't heard about token engineering some words about this New Crypton native discipline it's um the design verification and optimization of token based economic systems and as such it is a new discipline in its own right combining computer science cryptography Game Theory economics we draw from industrial and systems engineering control theory AI operations research and we have the human layer with

psychology decision science political science governance if you think of D and philosophy law and ethics and this combination of disciplines is new and te Academy is the first organization to offer education in this sector complex systems token systems are complex systems they show many properties of complex systems uh like in software complexity of course this is a diagram showing AR's contract interaction so we have dependencies on the software level and we have this dependencies on the economic level this is a typical d diagram token Engineers draw when mapping out the system design so you see

the arrows the value flows uh the cylinders the pools initial Supply reserves uh you see The Romp symbols uh these show the policies and the mechanisms involved and we also see some clouds which are external systems uh we can't control and we don't model at this point uh we have several layers that we want and have to observe and keep in mind in the design of token systems we have Dynamics over time so these systems emerge and evolve over time and we have interactions uh between mechanisms uh and on top of that we have activities of um

actors um whose Behavior we can't predict and that's why token Engineers run system verification and optimization because we can't go with spreadsheet calculations we have to simulate our system to observe the properties and to see if we get um if we have flaws in the design or if we get to edge cases or dangerous system States and to observe these systems and to test them to stress test them we build models of our systems and you can imagine no software no simulation model can replicate the real world so we have to take decisions on how to build a

model to represent the world that we want to test and that we want to verify and we have to understand and gain a good understanding if this model is capable to produce reliable and valuable insights so this is decision making when it comes to how should I model our model look like and how can we simulate the system here's another example um of a tool this is Senate Finance it allows project teams to simulate the token distribution and if you look at the left you see some inputs like a token price or a total Supply to then gain an understanding of supply and demand of

buying and selling Dynamics and if the token price and the supply are inputs then we determine if in such a simulation with what values with what parameter settings we'd like to run meaning we don't predict price here we go with assumptions about price and the right setting of Supply to understand the Dynamics of supply and demand under certain circumstances so my message is as a token engineer and generally if we want to understand and design token systems well we have to take decisions and we have to go with assumptions ions and as in any other example of complex systems

we have to be very aware of our assumptions to then draw the right conclusions in order to then make the most of amazing tools we have at hand Ai and machine learning to Cluster and classify detect patterns in transactions and behavior to go with symbolic regressions to find mathematical relationships and data to go with stress testing based on synthetic data llms or vae or to use agent-based models and adaptive agents even to simulate our economy so we have this interplay between human and machine reasoning we human designers project teams

communities take decisions on the purpose on the goals of a certain design and on the purpose of a token Network then we use machine intelligence to understand to run vast amounts of simulation runs uh process vast amount of data and apply smart methods to understand what could go wrong if we are on the right track how to secure an economy and we draw conclusions from these insights in modeling and simulations adapt the design and then again machines help us to understand the consequences of these design changes it's like a conversation with which brings us to

llms some months ago um we started asking can we use llms to interact with models and simulations in a way that we can prompt llms to run simulations um and we not only expect an answer on a certain question we can interact in order to change parameters and to um let a I agents uh run analysis or even visualizations of the results this is a project um where I was collaborating with my colleague Rohan MAA and we called it cat cat GPT so the general idea is that we have a system

model what I was mentioning earlier a moment ago we build a model of our token system and we have optimization techniques at hand machine learning reinforcement learning and on top of that we use a layer of AI agents so that we can interact with the system model run optimization techniques using human language CatCat GPT is the result and it's a framework it's available open source and it's basically um you can use it on top of any cat cat Redcat model um

cat cat and Redcat is probably are the most powerful python libraries for running token system simulations today and we use the the objects of cat cat models um to access the model and then have three main components memory the core cat cat GPT chatbot and toolkits memory on top is information that LM off the shelf don't have access to information about our token system information about uh a protocol it can be documentation about the model at hand and data um in tabled forms and uh also

in Vector databases and the toolkit at the bottom are python functions we can access and ex execute python functions through our chatbot and this is the core piece of innovation in catat GPT so we have a chatbot that for the user has a user input in natural language and a user output in natural language and at its core there's a planner agent and an executive agent and to be of any value for a token Engineers um these AI agents have to a make sense on of the human input B reason about the steps needed to

get to a certain output also reason about the memory and toolkits needed plus decide on the sequence of steps and here's where this deviates hugely and provides additional benefits compared to software as we know it today because we don't predefine and determine the workflows it's on the agents to decide how to combine memory and toolkits and how to run the steps to get to a certain result and they do contextual they run and execute based on contextual decisions in every single step and it's amazing to see the capabilities of AI agent when it comes

to um taking a simple user input in a natural language and then go through all the steps needed to run a simp ulation and produce an output um some more words uh so at the top we have long-term memory and short-term memory short-term memory is the uh history the message history what we have at hand based on an individual conversation with the chat we have new opportunities coming up with GPT 40 and we have long-term memory which is for example uh documentation of the model or other think of the the document the docs for any protocol to um inform about a

certain token system we can store it or or create embeddings and make it accessible and searchable for the AI agents in Vector databases and at the lower end some examples for toolkits um we can run typical python functions statistical methods we can access apis to get data onchain data offchain data and we create can create dedicated agents as part of the toolkit and the planner agent takes in the user input reasons about the steps needed about the tools and the context creates a task list and then hands over to the executive agent who is delivering results of substeps Step by Step again

applies reasoning about the steps needed and then produces the output including a documentation um of the steps and this is inspired by baby AGI um AI power task completion uh or react now again looking at this combination we have the system model in the back we have optimization techniques and we have the layer of llms to make this accessible via natural language it would not only be suitable for token Engineers basically it makes reasoning and positions about a system design accessible to many more

stakeholders so we have again the a perfect Loop in human and machine reasoning which could be applied to many more cases not only cat cat simulations and this brought us to retr pgf retr pgf is a funding Paradigm and concept in invented in the optimism Community to fund public goods that are valuable uh to further develop the optimism EOS system and uh these public goods uh have been funded in three rounds so far and if you look on the right uh round three distributed 30 million optimism tokens to the ecosystem

to projects providing value to the optimism Network and by the time of decision making in round three this was worth 120 million USD so this is a significant amount and at the same time on the right we have the number of projects also growing rapidly to 500 or 5001 projects in retr pgf round three and the decision making on what projects are actually valuable and since retro pgf is retro active funding evaluating the value they created for the optimism network of course this is a

challenge so we started exploring in February how AI co-pilots could help the decision makers in optimism retr pgf to evaluate projects um to interact with data the system model with um applications in human language to to run data analysis and even have access to much more complex simulations without writing code and this is actually highly needed in dowo ecosystem since on one hand we want to have this diversity of projects so it's hard to put them in boxes and evaluate with only

one or five metrics and on the other hand equivalently we want to have decision makers with a multitude of backgrounds so how can we facilitate the evaluation in this case and how can we build tools to have better insights and better data driven objective decision-making and this brought us to building retr pgf GPT so again what would be the equivalent of memory for retr pgf GPT of course data that is relevant in the context of optimism and retro pgf and this can be for ex example application data data that project submit and again more than 500 projects

their data uh their achievements this is something that is hard to evaluate manually right um then here we have an example of the funding results um in a public GitHub so this is accessible publicly or the impact equals profit this is the key Paradigm of how should distribute funding to grow public goods in the optimism Network um we Define it as the grants receive from projects should be equal to the positive impact they created for optimism round by round by round this sounds simple it's actually hard to

translate now uh let's take a look at potential queries so with llms we can query applications like on the left this is the user question and on the right you see the agent output so please list impact categories with the number of applicants and percentage of applicants funded so bch holders and decision makers have access to Prior rounds to uh the number and the sizes of these rounds we can query funding results so for example provide the funding range per impact category so we see the total applications the average median received in funding the percentage funded Etc we can as a decision maker I can compare my

voting to overall results to see my alignment with the overall Community project by project here we have a list of projects how much optimism tokens they received and my vote and also the deviation so there are many lenses we can explore the data available and the funding rounds in retr pgf some more words on the toolkit the equivalent of the tool kids in K GPT applied to retr pgf some months ago um a discussion started a very valid discussion started on how can we improve the evaluation of projects and there has been some experiments now also in round four that

is starting in early June to apply metrix based evaluation so in the past uh it was up to the decision makers to Define their own criteria how to evaluate projects and to come up with their own decision how to allocate funding and this is great when it comes to diversity and diverse perspectives on the other hand then it's hard to see if the community is aligned in in the goals and in the ultimate um purpose of retr pgf okay um metric based evaluation could be one solution let's set up some

metrics that seem to be useful for measuring the impact impact equals profit and this is a screenshot of um early tool and you can see on the right that there are some metrics like total transactions a projects um how frequently uh the smart contracts of a projects have been utilized or total transaction fees so how many fees were created based on the Project's contracts and these numbers of course matter uh when it comes to how the value or the impact they created for the optimism Network now going

further we can look at okay um if we have this toolkit and the community decided on metrics to apply how to evaluate projects uh I can take a look and play play with these matx to see for example um show me the alignment score this is now um a metric to show the difference between my vote versus the overall outcome and I can even better understand metrics asking in can you explain me the the alignment score and this is often the challenge when people are looking at dashboards how is this metric actually calculated what does it say um what metrics or what values what data does it

use to uh what data is processed what's taken in what's the inputs and what's the output to understand how this metric matters for my own decision and llms can help us in understanding these metrics now you might ask okay a voting single round is basically a calculation and this is something that is not so super complex however if we look at this voting design the logic how retr pgf distributes funds this is actually relevant for the total optimism economy let's take a closer look if we

have retr pgf we distribute fund to Val valuable projects and that way we grow developer activities and projects now remember 30 Millions distributed in round fre this has an impact on secondary markets right because increases it increases the optimism Supply significantly and not all projects might be able to just hold some might be forced to sell their optimism tokens to cover their costs and so it has an effect on the price and it has an effect on the fee revenues of the protocol because it affects the price of transactions now this could be a positive effect because in you us value

this decreases the costs of every single transactions this would be good it definitely has an effect long-term on retr pgf because the fee revenues fuel the funds available for retr pgf and there's another effect of the developer activities and the projects on fee revenues because remember the metrics shown beforehand um depending on the projects and what they build they can of course affect the fee revenues via increasing plug space demand via the applications built on top of the

protocol and there's a final um impact if we look at secondary markets to retrop pgf because of course the dollar value of optimism tokens drives the funds available uh if you look at the volatility of the token in the networks of course it has an effect if the value decreases by 50% in how attractive is the funding retrop pgf for projects so if we take a look at the system perspective we see that only this tiny tiny this individual retr pgf is not an isolated thing it has an impact on the total optimism system and Via simulations we can ask

how does retr pgf impact the network growth of optimism or any other token system how does retr pgf impacts the network value and what are the best metrics to evaluate these evaluation criteria that should be uh also hard to gain and how can we align incentives so that not only the projects also the decision makers have skin in the game as well as the optimism token holders overall retr pgf GPT is one way to make the systemic perspective more accessible to any stakeholder to any decision makers

in Dow it can help us to understand individual rounds voting designs the system perspective how this affects the total optimism economy and it helps us in taking objective and datadriven decisions these are some of the requirements we've seen set up for retrop pgf GPT like we want to make uh this equipped with an Ever growing set of tools and information this makes total sense uh if you think of the memory and the information that is ever growing and and is in evolution every single day about the um optimism Network and Collective the tools to measure

impact the tools to measure the effects of changes in Repro pgf on the total ecosystem this has to be modular of course it's um also relevant um how much we open source to grow the knowledge of the overall ecosystem um no open AI lockin whenever possible let's see how alternative llms uh can cater the needs um a modular framework on the tool side on the metric side with the Dow to take decisions on what are the most valuable um metrics to observe value add and it also um Can inform how we collect data about

projects how this data is structured to make the most of the llms and have optimal quality in the results and last but not least this is also new areas for all decision makers because it's different to interact with llms in human language it has many benef benefits uh but it we also have to understand the flaws overall can llm support decision-making endow systems yes are they capable to become the interface between data metrics simulations and human decision makers think of this Loop absolutely yes can we um improve datadriven objective decision- making

and optimization yes and work is needed for sure when it comes to the memories the information available the function calling the metrics and the the improvements and further development of this planning execution for this is the most important aspect we want to have auditable and reproducible results and uh reliable results based on llm interactions that's it and finally we are running a study season at the token engineering Academy you can find it at token engineering.net if you're interested in complex systems and token systems check it out thanks

for today [Applause]