ETH Evals: testing agents on Ethereum

Transcript

[music] So uh code with oppus uh codex with GPT or open code with Gim uh which agent would you uh trust with Ethereum work today? Uh I'm Reinat. I'm from Build Guild and we build tools for developers and now those developers use agents. Uh at the start of 2025, we used to ask questions to chat GPT and uh read the uh read the answer, copy code uh maybe uh and uh copy to our ids uh and debug

everything and fix ourselves. But now we delegate everything to our agents. They uh create contracts, they send transactions, uh they uh deploy the contra those contracts. So agents are here and uh most of the work already theirs. Uh what actually agents are? Agents are three agents is three things. Model inside the harness uh plus instructions. A model decides uh harness acts and instructions steers. Uh every piece can change and every change called help or heart.

So this is me with my agent every single week. Uh there's uh new uh MCPS, new skills. Uh I'm changing my agents. But uh is this actually help? We didn't know for a long time and um we decided to measure it. But how to measure it for test? Uh for code it's easy. Uh it's deterministic. When uh we add two and two we uh all the time get four. Uh same input sum output uh one run is enough. uh every test test green and you know that everything works.

But for agents it doesn't work that way. Uh when you give the same task for single agent three times it could uh go to uh with uh three ways three different ways. Uh for example, you ask your agent uh send 12 and a half tokens. Uh agent should uh verify if uh transaction sends if uh tokens arrived and many other things. So in this example you see that uh two uh runs passed and one rejected.

But and at least we have uh numbers two out of three and we compare uh we can compare those numbers for different agents and uh actually one run is not enough. Uh we need uh more tasks always better uh for when testing with agents. Uh so uh which tests uh which evolves we could uh actually do uh skill evol do the skill helps harness evolves uh do the tools and the loop help and uh the agent eval uh evaluating uh the allent agent

uh for which one is better. Uh we uh evaluated two out of three uh in our uh three projects. Uh skill evolves in uh scaffold if and if skills and uh agent evaluation uh with uh if evolves. Uh scope grew every time. Uh but we use the same method tasks checks and uh repeated runs. First uh is uh scaffold is uh as chief already said we had uh extension and we uh changed them uh for uh skills and now

we decided to measure them. Uh first we sorted the skills for uh two groups. First uh capability uplift uh it's uh about uh the things the model cannot know. For example, uh model trained for previous version of library and uh and now we have new one uh or something from inside of scaffold if and uh second group is encoded preference. Uh it's about uh the mo the things uh model already knew but uh we wanted uh them to make uh another ways

and then AB testing uh five runs each with uh with skills and without skills uh with separate grader. Uh here's the results uh for capability uplift groups there was a huge difference from 42% of success to 97 but uh for encoded preference uh much smaller from 90% success to 90 6%. And uh even for some skills there's uh was almost no difference and uh we removed four skills as I remember uh

eighth skills it's uh Austin Griffith pro projects and uh it uh it's a library of uh 19 skills uh which uh already uh helps many developers ers to build on Ethereum. Uh do the skills help? In from July to September, we created uh uh 97 tasks for those skills because uh when we read them at the July, we uh wor a little bit uh here. there was very long and uh

some parts could be already outdated or some parts uh models already could know or at least fetch in uh one web search. So we decided to check it. Uh again we uh we had two tasks two kind of tasks. quiz task where does the agent know the fact or not simple task and goal task. Uh it's uh it was mostly uh task to build some something and to check if uh agent do uh things properly uh when we don't ask about it directly

and then we checked uh every skill uh in three setups with no skill skill with uh old version of skill and with new version of steel when we uh reor them uh uh with two uh models and also we had a blind judge that uh knew only about uh result of the uh run and uh it no knew about checks. he need to uh about the checks and uh we decided uh before the first

run uh about the skills or about the parts of the skills uh how they perform. If uh run passed without the skill then we need we don't need that skill. If uh passes only with skill, so we keep that instruction. If uh run fails even with with skill, we need to fix that skill. Uh again, we uh we rewrote some parts of uh those skills and uh now it was uh 63% less. Here's the uh results of our runs.

With no skill, it was uh 72% uh of success. Uh with old skill, 85% and with new skill, 94%. Uh all the results you can see uh onskills.com/evolves all the runs and all the statistics uh for with uh for regarding models. Here's uh results of our models we used for the tests uh OPUS 5 and uh DPT 5.5. Uh there's uh better oppus was better and uh you can see between uh states difference uh almost

the same. But to trust uh results, you need to trust uh evils because uh they could be wrong, they could be broken. Uh we had some issues and uh here's what we run into. Three simple examples. Uh when we started to evaluate with uh skills in scaffolded uh the same agent uh graded its own work and uh that agent uh give yourself [snorts] gave yourself uh credit just for trying not for results and uh he graded itself for uh 100%. We

fixed that and results changed to 40%. uh is skills. Sometimes agent quoted the skill and uh judge saw that result. Judge judge saw that and it cost uh could cause different results. We also changed that. And third example with skills uh the task was find uh cheapest uh chain and uh measure the gas. uh we always had results but uh actually uh agents never uh checked it never measured it. So we uh fixed the check

reged and now results uh change it from three out of three to zero out of three. So uh before before evaluating you need to test evil itself. Uh so uh after those two projects we learned uh what uh good evil needs. First the task or what the agent is asked to do. It should be uh same words for every run for every agent. Uh the same environment every trial uh agents run in a container uh one image fresh copy per run. If image or a container

will be different or environment some kind will be different it won't it won't be fair because uh result could change just because of that uh a verifier uh court or judge uh court if possible because uh court is deterministic again it's uh cheaper it's faster so uh code whenever possible and uh AI judge If code is not possible most of the time yes or no questions or uh yeah yes or no question and uh four is uh every run should be saved uh transcripts uh changes of code

of the chain state so later we could uh try to debug find out what's wrong and or just uh compare results I don't Part three uh is evolves. Uh so that learning led us to uh third project uh if evol. The question was is uh how real agents handle Ethereum work uh for real developers.

Uh there's three mods on if evils uh model, agent and agent plus skills. Now model is uh what uh what a model model without uh internet without uh tools just uh by one API call. Agent uh is again uh model plus harness whatever uh they need and agent agent plus skills is the same as agent but uh plus if skills uh three types of tasks uh con concepts mostly for model uh just uh again

knowledge if model know it or not build uh task for building uh contracts or pairs and act build something on chain change the state and uh work with chain it built on harbor uh framework for it's a framework when you can run uh containers with uh with uh any uh harnesses models and uh get all the logs. It's uh very good for uh getting separate environ environment.

So here's results uh OPUS 5.5 uh and uh GPT A leading here and uh here's some other interesting lines like you can see that for example Grog has uh 90 on concepts but only 22 on build because you know is not the same as building and for example DPT uh not DPT GLM 5.3 52% on concepts and zero on act uh on at the start of the uh not at the

start this year Antropic uh released road map of excel intervals and uh here's the stage uh first uh evaluation suite mostly about tasks uh second harness development uh and uh evaluation m maintenance. Uh looking back, we did very very similar uh way uh uh task with scaffold mostly uh uh harness with uh these skills and maintenance uh open uh public port with uh ethals.

Uh but we didn't have that road map. We had uh bugs. So answering the question uh which agent should we trust on uh with Ethereum work today? Uh it depends on your task. You need to measure uh not only knowledge but uh building you need to get numbers and uh only after that you you will know which one is better for you.

Thank you. That's mostly it. Right. Thanks. Let's check the questions. Uh yeah, so from the previous talk, somebody asked that if they want like how do they contribute with creating better like uh learning materials for AI >> for uh >> I mean training materials for the AI itself. Is there some way to like contribute for that or should people just write the blog and explain?

Yeah. Uh it's uh you can create a tasks. It's it's uh it should be uh not so difficult to add it to uh if because for now it's uh just one line in config to add the models. When we add models it uh run every uh single task and when you add the task it runs after uh merging for every single agent. So create pull requests. We will review it.

Yeah. >> Uh could you uh >> maybe I give my view here. So the easiest thing probably is to contribute to the skills because if you contribute by creating training material, it's going to take forever for that to land in the model and there's no way for you to like track that or or anything. But if you contribute to skills, if you recognize that there's a weakness somewhere when you're trying to do something, if you write it into a skill, then you will immediately benefit from it yourself because your model can have access to that skill and then you

publish it for others. And what is the best place to contribute the ETS skills? >> Uh there's uh you can go to eskills.com. You you see the uh GitHub repository here. So uh and the same for ethals uh here's uh basically uh ethals repo link >> sorry can you repeat uh okay so to to contribute you can go to esklls.com on and ethas.com

uh in both websites you will the GitHub links and go to the repos and create issues. >> Okay. And related one of the questions is that what were the types of evils when the that uh it missed when when testing with model plus agent plus skill and why? So maybe this is more generic question like when you saw the >> the failures then what kind of things were they failing in?

>> Uh there's uh a lot of uh different situations uh where skills are failing. I don't remember can't say exact examples but uh I don't know they're they're not ideal at all for now. Okay. >> So yeah there could be uh a lot of mistakes.

Uh no exact no exact examples for now. I I can tell a personal anecdote what when I do like Linux development stuff, I noticed that the models like obviously recommend the average thing. So they kind of recommend the old thing. So as soon as you're doing something that's very new, they're going to give you wrong advice because you're going to tell you the statistically average thing. So, so the more you have like niche and the more new it is, so that's where it most likely it's going to fail. So, like if um in in my experience, the agents are or AI is very good to get you like up to speed that

you're in the same level as all the average developers quickly, but then when you start to do like niche stuff, it it fails. >> Yes. Yes. Exactly. It could make mistakes in when building uh contracts obviously when building uh front ends when uh just again adding knowledge not adding using that knowledge there's a lot of different areas.

Uh then there's a question of uh do you have percentage with actual models such as Opus 5.5 and GPT 6.1 which effort do you recommend? So >> uh yeah here's the result >> and anyone can go to evil.com right? >> Yes. So you will see that board you can click for every cell and uh you see uh exact uh tests exact evaluations uh statistics and all that stuff. People in the back won't see this, but you can go to ee evolves.com

and and see this it there. >> Or what was the URL? >> Could you repeat that? >> What's the address? It's e >> evol.com. >> Yeah. Right. Um then question that could evil tooling be used to create like uh Linux evil or go evil or something like that. Are is your tooling in a way that it's easy to general generalize to domain specific evils? Uh I'm not sure. uh it's good for creating uh evils. So adding uh models on Ethereum so

but for uh other stuff I'm not sure I well maybe going forward if you think about this then maybe you can make your evil more generic so people can then fork it and do >> do it repeat it for other domains. Uh and then we saw in your presentation that having access to the skills makes the models way much better. >> Yes. >> Because it's it they get the domain expertise and also like the the model have their cutoff date. So there's no way for them to know about current without the skills. But there's one question that are skills becoming less important or even degrading agent

performance with the new frontier model releases? Uh yes and no. Uh sometimes it could be right and uh because uh uh yeah models became better and better but uh after release of every new model we need to measure again because uh we need to know uh exact numbers exact percent and only after that we uh can say if uh success rates uh changed or not. So answer is always measure.

>> Yeah, it's it's moving too fast to make any con too generalized conclusions. All right, then there's a question of what are the gaps you think are in the current models or harnesses and skills? uh gaps regarding Ethereum or >> uh it's just generally what do you think are the gaps in the models, harnesses and skills at the moment? M uh not sure I understand the question like uh what models skills what you couldn't do with uh skills and models right now or uh if if it meant that

I don't know maybe in relation to Ethereum development like what is the biggest gap not just in the models but also in the harnesses and skills Uh what are they not able to do? >> Uh you know uh agents uh can do a lots of stuff and in every area they they're good but on the top of every area there's a lot of the things uh models and harnesses uh uh couldn't help. So uh again for general stuff

uh they are good for some niche stuff they're worse and uh I don't know always uh check uh what your uh agents and harnesses what your models harnesses agents uh did before uh using it using the results in production. >> Yeah, that's definitely good advice. Yeah, we have one question in the room as well.

I'm just curious about the design of the judge of the eals and I was curious if it's deterministic or nondeterministic checks. um if they are deterministic how do you specifically like design the task for a skill like how do you choose one two three or different scenarios if it's nondeterministic and you use an LLM how can you trust another LLM which models do you use okay uh if it's deterministic uh try to use code try to uh write regular tests uh which can again check that code for nondeterministic

uh it's uh most of the time it's easier just to uh have yes or no checks for your exact task. So another agent uh can uh make that checks and uh based on on the result uh you understand if uh if the uh work done well or not. Uh am I answered or no?

>> How can you know that? uh you don't know. Yeah. So you can uh evaluate you can uh use different uh models as a judge for example. So you could see if results differs or not and based of that those results you will understand which will fail or or not. Maybe you go to ethals.com and then follow up on the source code and maybe that will help you understand like exactly how this evolves have been designed. All right. Thank you, Reinat.

Thank you. [music]