S.10 Ep.2 - Deep Funding: The Future of AI-Powered Public Goods Allocation — with Devansh Mehta

Transcript

Welcome to the Green Pill podcast. This season, we are exploring how technology, community, and new forms of coordination can build, fund, and accelerate what matters in the mid 21st century. This season, I'm talking with builders, thinkers, and innovators who are pushing to the edges of what's possible. From onchain capital allocation on Ethereum to AI alignment to reimagining markets and governance. Together, we'll discover how these ideas can bring us more close to a world that's improving every week, every quarter, every month, what we call Protopia, a realistic, preferable future that is constantly improving through small incremental changes over time. If you want to learn more, visit our website at greenpill.network where you can download the Greenpill books for free, play with some of our projects, join the Telegram, or participate in the network. All right. Today I have Devanch

Meta who is the AI and public goods and governance lead at the Ethereum Foundation where he unblocks pilots that apply AI prediction markets and evidence-based methods to fund what matters in the Ethereum ecosystem. He's focused on helping DAP developers deepening ties with the AI and machine learning communities and exploring AI agents and governance and funding flows. On this episode, we are talking about deep funding. What is the latest and greatest for the mechanism design that maps Ethereum's open source dependency graph and uses markets of bots, models, and traders to predict repository values. This is an episode with Devon, I think, that is worth listening to if you've been following the deep funding pilot. There's been an evolution in how it works and we actually have some final

results for the first pilot. So deep funding is one of the most interesting experiments in how we're funding what matters in the Ethereum ecosystem and uh and I think uh for that reason alone it's an interesting listen enjoy. >> Hey Devans, how you doing? >> Yep, I'm doing great. Uh we just like wrapped up the first uh pilot of deep funding both the onchain and the offchain version. So really excited to share the results with everyone uh on this podcast.

>> Yeah, same. Uh maybe we could do a minute revisit of what is deep funding and why does it matter and then let's get into the results. >> Yep, definitely. So at a high level um all of us agree that impact evaluation is important. Uh we also agree that impact evaluation is really hard uh especially if you want to do it well. So deep funding kind of combines both of these intuions that we need some level of manual human evaluation and the fact that it's hard to do a lot of them um to actually create a scalable mechanism uh of humans give a few evaluations. U models give a lot of evaluations and the

human evaluations choose the models which then give a lot of the evaluations. So you should really just think about it as a way of scaling up human evaluation because it's so expensive but so needed. So that's kind of in one word how I would describe deep funding. Scaling human evaluation. >> Great. Um so there was a pilot that I believe it was funding 250k that you ran of deep funding. Um is that number right? And and why don't you take us into the results of the pilot? >> Yeah, so the pilot was more like a trial run. The main version is going to happen in GG24.

This was a pilot run to just iron out the kinks, see that the mechanism is working because there's nothing scarier than allocating a lot of money through something that's completely untested. Uh you end up doing more harm than good actually with that approach. So we've at least got some preliminary weights between repos. We've sent it to the judges who gave the data and said, "Hey, do you think these weights make sense?" So we've been doing a lot of that trial run right now but we have not allocated any money through it and we don't plan to allocate through it either. That's all going to be happening as part of GG24. Uh we basically got two versions of the results. Um so there are like I'll back up to the two ways that we can scale human evaluation. The first way is

through a data science competition. So here you should think of something like Kaggle uh where you have ground truth data and then you have many AI or model builders competing to try and predict what that ground truth data is and then you get an error score based on how close your predictions are which is basically the foundation of AI. The ais and LLMs are trying to predict the next word. So we are extending that by asking AI to predict the rating that a judo would give. So we asked that if a had to um compare go ethereum and loadar what rating would a would a walky give like

which one would he choose and what rating and that's what the models are trying to predict essentially like what rating the jura would give uh to these open source repositories um so that's one version of it which is structuring it as a data science competition something cool which happened in the round was that we got white hat civil attacked in the round. So one of the model builders submitted 30 different models all under different accounts and ids and one of his cibils actually ended up like and he was good enough to like inform the organizers saying hey I've attacked around these are all my accounts which have cibil attacked them and um as a result of that like one of his sibils actually won like so there were 30 there were totally 50

models submitted which were valid and 30 were syllables so 80 models totally 30 syibles and 50 good ones and uh one of those 30 cibils turned out to be the winner but he was good enough to inform it. So that's the main issue with all of these data science competitions that you basically run the risk of civil attacks by model builders who submit many versions of their model. So we simultaneously did this other really cool experiment of uh allowing model builders to upload their predictions and actually put some money behind it. So this actually is civil resistant. It's like a shift to proof of stake where you have to put money for it to affect the

weights or be counted. So this is where it logically starts becoming more like a prediction market where you have initial weights between repos and then a model builder uploads all their weights and as a result of that it changes the price of each repo and the model builder wins or earns money based on how the data resolves. So I think for me it's just been super interesting to see how we ran this data science competition which is purely on skill uh and it's like small enough that it wasn't majorly attacked by civils but it had that vulnerable point and then we ran it as a market based onchain mechanism and the amazing part was that the price on the prediction market was submitted as a

model in the in the data science competition as well. So we could actually see how accurate the model is uh compared to like the other model builders. So, uh, are are you satisfied with the results of of the round? Do you feel like it it set out to I you set out to fund open source software and to, um, do it with dependencies. Do you feel like we've achieved a structure that can be built upon? >> Yeah, I do think so actually that we have achieved that. So, I'm going to share my screen here. So these are the results of the onchain version of uh deep funding where we have like about 45 open source repos and all of these weights add up to one. So something like EIP is 9.2%. So if you give money to

like through deep funding for open source repos helping Ethereum EIP would get 9.2%. Consensus specs would get another 9.2 Go Ethereum would get 8.2 uh Lighthouse gets 3.84. Prism gets 3.8. So we actually managed to get um kind of a credibly neutral funding mechanism which can give unequal weights. So if you look at a lot of other mechanisms like protocol guild, they basically just give equal weights uh like based on time. So very objective measure like time. The more time you have, the more weight you have. But like as any of us who started companies or worked in one know like that uh seniority is like asking for government levels of efficiency by

rewarding on seniority. So rewarding on how important something is can actually be much more important and much more valuable. And I think that's the one thing that we've got like I've shared these weights with with like a lot of the repo owners and maintainers and they haven't come up with major objections. They said, "Yep, I wouldn't agree with all of it, but it seems to be in the right ballpark." So, I think everyone's just been more impressed that we've managed a process for getting unequal division of funds and not just done the equality stuff that we usually revert to. Um, so I think that's been the biggest success that we've actually got unequal weights which seems to be broadly agreed upon.

>> Yeah. Well, that feels like a an achievement. And uh I think that Vitalik and you were talking about deep funding on the Green Pill podcast last December. So, it's been a 10-month journey. It's probably been longer than that uh if you if you've been working on it, but um it's it's neat to see you stick the landing on the on the experiment and uh have a wrap on the pilot and the the first funding distribution. >> Yeah. And I think we played it very safe in the pilot which is that we know that repos which are part of protocol guild uh chev tooling guild and argot collective like there are 45 repos which we all agree are valuable like these repos no one is going to say that they are scam repos or something. Um they're

all valuable. So that's how we played it very safe in this round. We're like just between these 45 repos can we now get a division of weights between them. Um so that's kind of turned out to be fairly successful and we've also built a nucleus of very good AI and and LLM like model developers and that's been the real strength of the deep funding community that like you know actually getting these people to comes like take effort in using the training data build their own model submit it that's been some of the labor intensive work and the other love of and the other work of love has been from the jury uh we recruited juries to give comparisons and that's also been a labor of love which has been entirely uncompensated for them. So

that's been maybe the two biggest things that require work. Uh one is getting a community of AI and model developers and the second is getting expert jurors to actually evaluate the different repos that are uh part of trying to get weights for. >> So what what's next from here? I mean are we just you know we're proud of the experiment that we've come so far but is there is there any inkling of what uh round two will look like? >> Um yeah sure sir I I'll again like share my screen here. Yep. So this is like the data science competition that also got concluded. Uh so if you go here you can see that this ended uh two days ago. So now it's 2 days since this competition

ended where we give weights to 45. And here you can see kind of the leaderboard that like who is the winner and what was their error score minimized. So in this like very meritocratic process the winner was someone called David Gasquz who basically won with an error with the least error score over here. So then like I think the next thing which we're really going to do with this is take this winning model and apply it to repos more than just these 45. So we've got 45 repos and as part of GG24 we are going to expand it to another 45 repos. So there'll be totally 90 repos. So then the question becomes how do we select which 45 repos that we should accept in

in addition to these 45 and that's where the winning model comes in that as the winning model his like he can decide like his model can decide which additional repos to take into the round and actually get money for it. So right now we have about $350,000 to allocate via this mechanism. Uh 175k is coming from the Ethereum Foundation and 175K from Gitcoin. So I think the real next step is just going to be making sure we can distribute the money hard like like distributed well. Uh I'm reminded of something which Julian Zavistoi said which is that um getting money is easy but like spending it is super hard spending it well. And I've worked in the

nonprofit sector and I've seen how how harmful it is to just like throw money around without thinking about how you're doing it. Um, so I think that's really the next step like we've figured out >> the rough outline of the process and now figuring out how to spend it uh and how to actually distribute it and see if it actually is a good way of distributing as a mechanism. So I think now is when the like rubber hits the road. Yeah, it's interesting how a capital accumulation and capital distribution are two sides of the same coin and I feel like a lot of the work that we do in this space around retrofunding and quadratic funding and deep funding is about finding better allocation mechanisms which will make it easier to fund things in the future. Uh another frame I really like is you always talk

about connecting revenue centers and cost centers for these blockchains and uh the open source software is a cost center. So excited to see uh now that we figured out a good way of allocating to it, how we could uh channel more revenue into open source software, >> right? And I guess like the hard part I've been wondering is how repos can claim the funds. So like I can see two ends of the spectrum and this is where we move to GitHub repo level governance which is something that I've got interested in recently that if we give like a a GitHub repo it gets money from deep funding so we allocate it to the repo. Now who gets rights over spending that money? Should it be the

maintainers? Should it be a consensus amongst the maintainers? Uh should it so I can see two ends of the spectrum here. one end of the spectrum is that it's a it's it's an auto redirect. So if a repo that's part of protocol guild or dev tooling guild receives the money, it gets redirected to their splits contract. So some of the repos which are part of our god collective have that in their constitution that any money which is earned by our repos has to be given to the splits contract and shared between all their members. So that's one extreme or one end of the spectrum and the other end of the spectrum is u something which merit systems has piloted where essentially what they do is they create a counterfactual wallet where and by

proving ownership of that repo by proving you're a maintainer you can spend funds from that wallet. So if there's uh a repo for like loadar so what merit systems will do is create a wallet and the only one who can access that wallet is someone who can prove that they're a maintainer of loadar and then by proving it you get access to those funds and then like each maintainer can choose how they want to spend that money. They might even like spend all of it on themselves but we assume like they assume a high level of trust between the maintainers themselves. I was going to say one time uh when we were pro promoting Gitcoin Grant Stack like three years ago, I walked into this event and ran a QF round just kind of like yoloed it for

for this event and um you know it was a big success. We raised uh I think it was like5 or $10,000 for this community and we did it all emergently with the QF tool. And I I walked away thinking, "Oh wow, like we got to demo the tool and these communities got some money and that's isn't that great?" And then I found out at Devcon six months later that that one of the the we were having drinks and one of the one of the guys said to me, you know, Kevin, appreciate you giving us money, but you also gave us a huge thing to fight about now that we had we actually had money for the first time and we it it it started to like tear us apart a little bit because we all had different things that we wanted to spend it on. And so, um, I had

never had that perspective before of allocating capital to these projects that have never received money before kind of creates an internal political stressor that that we hadn't anticipated. And I'm curious to see if any of that happens with these open source repos, which some of them probably don't have a constitution or a way that they're supposed to spend them spend the money, >> right? Yep. uh but I do feel like funding mechanisms at large should start focusing on the on the unit of work which is the repo. So which is why we explicitly discourage any project from applying like we don't allow Gitcoin to apply for deep funding. We'll apply a specific repo of Gitcoin like

Gitcoin/quadratic funding or some specific repo that you have. So I think actually focusing on the unit of work rather than the brand of the organization is something that is long overdue. Otherwise we see weird distortions in the market like the fact that because protocol guild represents so many different repos they apply as one and they get much less money compared to if each repo had applied on its own. So I think those are some of the tendencies that we want to fight against uh kind of that by coming together under one banner you actually end up getting less money in these funding mechanisms. That doesn't intuitively feel fair. You know we should be rewarding you based on your

contribution and your value and not on whether you apply as one project or as 10 projects. >> Yeah that's definitely a problem um in other mechanisms. Well uh as we get to the bottom half of the episode, is there anything I didn't ask that you want to say? Yeah, maybe just talking a bit more about the tweet two weeks ago and uh kind of what an onchain version of deep funding is and why that's useful or beneficial. Yeah. >> Yeah, let's do it. I've got the tweet up. Uh why don't you dive it in for people who are audio only and don't have it in front of them.

>> So I guess like [clears throat] like one of the issues which we talked about is that we don't have an accurate price feed of repos. So in something like the stock market, we have an accurate price feed that Nvidia is worth so much. Uh OpenAI is worth open AAI is a nonprofit. So I guess they don't count but like um all public limited companies we have a price feed which shows how valuable each one is to the world um and how much value they can capture. But for public goods, we don't have the equivalent of that. there is no live price feed which shows that go ethereum is uh is worth x amount uh loadar is worth y amount and that's basically the problem that we

need to solve if we want to make public goods more meritocratic and less about social gains so kind of how do we do that I guess is the like biggest question so kind of like the pilot we ran was setting up a multiscalar prediction market so that's just a fancy way of saying that all of the outcomes s have to add up to one. So we see that in elections for example where we say that this party is going to win 30% of the seats this other party 40% and this last party 30% so they all add up to 100%. So just think of that as the mental model and now apply it to uh different opensource repos which are valuable to a project. So we say that okay there are these 45 opensource repos they're

valuable to this project. Now we have to figure out uh the price of each one and they all have to add up to one. So that's what's different with the stock market. In the stock market there's no rule that all Nvidia plus Apple plus Microsoft have to add up to one. But here there is that rule which happens. So that's kind of one part of it. So then the question becomes how do we uh decide which repo is worth how much. Um and that's where the human evaluation comes in that the human evaluators don't directly decide that yeah you are worth 0.02 you are worth 0.05 because as you referenced earlier a just leads to a lot of conflict that like hey why did you say I'm only worth 2%. That's not fair.

I think I'm worth at least 2.23%. And like these weird discussions can be had. So the way we do it is that evaluators keep the market honest and the market keeps the uh decides the prices. So even if a repo is upset that hey I think I was scored very less uh they don't have any single person to blame. It's it's it's based on market dynamics of who is buying and selling your repo and those people who are buying and selling it win or lose money based on the human evaluators. So this is like another way of scaling human evaluation where the market is deciding the price of different public goods and evaluators come in to make a judgment which makes the market participants win

or lose money upon their bets. So normally what we've seen in these markets is a lot of manipulation where a repo will artificially buy its own shares so that its weight goes up and the weights of other projects comes down. In this market, we actually surprisingly had we like you know I actually tried to like my like like my friend tried to hack the market by artificially buying up a repo called ACT. So ACT was bought up to 10% of the market with like $50. $50 brought it up to like from 1% to 10%. Uh but then like because so many model builders were uploading their models, it automatically got traded down to a much more uh decent level. Like let me see what it ended up

at. it ended up at uh 2%. So we saw it automatically traded down and whoever tried to attack the repo basically ended up losing money because they did it. So I guess like unlike a quadratic voting system where if I voted for ACT, I still keep my money because I've donated to that project and then I get to keep that money even if it's an attack. But in this if I vote for ACT, my money is basically going to smarter traders than me. So it's actually creating a penalizing like a a penalty for people who are trying to pump their own project at the expense of others.

>> Yeah. Seems like one really interesting attribute of prediction markets is that they're sort of self-correcting in that way. Every every action is game theoretically optimal only if you believe like truly believe in the action that you're casting. >> Right? But like I'll tell you the thing that only works when the market is active and we had to do a lot of work to do that. like we like because we had this data science competition going on which was using the same data. Whoever was on top of the leaderboard, we reached out to them and gave them trading subsidies. We literally went and told them, "Hey, take uh $2,000, take $1,000 and use it to trade on the market. 20% is a grant. 80% has to be traded. If you if you get a profit, you

can keep all of the profit and just give back a principle. And if you make a loss, just give back whatever's left." So that's what is actually involved in bootstrapping these markets and making them efficient actually reaching out to the people who you want to get them to participate and giving them money to trade on your market. Uh so that's kind of like even poly market has 5 million in subsidies to liquid like for liquidity and that's maybe the last point I'll say before concluding on the prediction market stuff which is the hard question we've not figured out yet is the right level of liquidity to put into each repo because the trade-off is if you put very low liquidity then the price goes up with a very small amount

and the example being act where I bought $50 of ACT and it went from like 1% to 10% like just with $50 which is a very low cost of attack. That's what happens when you put too little liquidity and if you put too much liquidity then it has no effect on like even if good model builders are uploading their models uh it doesn't impact the price at all because there's so much liquidity that whatever price the market started at it basically remains at that price itself. So, it's really an art of figuring out when to put in liquidity injections and at what point and at what is the right level of it. And that's something we need to figure out in the future iterations.

>> Yeah, feels like something that's in that's going to be improved upon in future iterations. >> Well, I think we're about out of time now. So, anything to say before we wrap? >> Uh, no. Just congrats to all the model builders. We announced our winners today and they're all getting prize money from these competitions. So, just congrats on all of the people who submitted and the results and the winners. >> It's been great to see the experimentation. Where can people go if they want to learn more about deep funding? I think it's deepfunding.org. >> Yes, that's right. That's the website name deepfunding.org.

And uh the other version to check out is deep.se.pm. Uh so that's kind of where the prediction market version of deep funding is happening now. So check out deep.se.pm PM to see kind of a live implementation of it. >> Well, it's been cool to follow the experimentation and um you know, I'm really excited to see where this experiment goes in the future. So, um thanks for coming on the podcast, Danch. >> Yes. Yep. Anytime. >> Thanks again for listening to the Green Pill podcast. If you've enjoyed today's conversation, be sure to subscribe and share this episode with a friend who cares about building a brighter future.

You can find our episodes, articles, and resources at greenpill.network on YouTube or on all podcast platforms. Until next time, keep building, keep experimenting, keep coordinating, and I'll see you next time. Peace.