Deep Funding Scaling Human Judgment for Open Source Funding - David Dao

Transcript

cool so slides please yeah so today I'm going to give you a lightning talk on deep funding which I think is one of the most exciting mechanism experimentation that currently happening in our ethereum ecosystem so quickly let's talk about open source and open source dependencies because we all rely on that very famous graph um rebuilt these large modern digital infrastructure projects and there's this guy this random person from Nebraska that has been thanklessly maintaining this crucial piece on the open uh Source dependency stack right and why is that important because your backs of money when it comes to ethereum dependency depend on this random guy from Nebraska doing his job really

really well you can look that this is for example on the top these are all the liquidity exchanges built on top of the defi apps and that is built on top of digital infrastructure like Gaff metam Mass parity theum one blockchain and all of this teag we need and is a public good actually to create our ecosystem to keep it performance to keep it secure right so that our bags of money don't collapse but if you look at how this random guy from Nebraska is right now raising money he doesn't have a lot of time to develop he doesn't have a lot of time to maintain he needs some kind of because it's open source some kind of funding which is either a s hassle which his dependency probably is or he needs to apply for Grand applications that is not a way for suain

stainable and healthy ecosystem that is the ecosystem and so vitalic he came up with like this really simple and fancy idea which I think is actually a paradigm shift in our community because we talk a lot about blockchain and AI but this might be one of the first use cases where we believe it's actually very scalable and useful so he talks about deep funding and that's something I want to introduce you to today so deep funding is a coalition of several Partners in the ecosystem um voic gitcoin Alo protocol open source Observer drips payis um evil science which I represent and vitalic himself that has been proposing a new idea to fund these critical dependencies and how

it works is basically we trickle down funding from the seed graphs noes into its dependencies and then this random guy from Nebraska once more is maintaining this since 2003 but making a ton of money and that's the hope so how does it work well it requires two components it requires a deep dependency graph and requires deep learning that's why it's called Deep funding so let's go step by step so deep dependency graph we basically propos to map ethereum's ecosystem through dependencies starting with ethereum itself uh in its core protocols and repositories 32 34 seat noes and then from that we analyze all its dependencies through software dependency analysis and derve more than

4,000 child nodes on the first layer and that initially can go deeper and deeper and deeper but let's start with this layer first for visualization then next this is how it would look like after we mapped it out these are the 34 seat notes in red and the rest Blue parts are the 4,000 child notes um that data set is already publicly available in our GitHub repository we are going to label that data set now and how we label it as we ask F's core maintainers and some of the most important people in our ecosystem so how it works is vitalic propos proposes two judges these judges proposes two more judges these judges proposes two more judges so we have an exclusive list of judges that all they do for the next couple days is

to label thousands of these dependencies in a pvse comparison saying okay given no a node B which one would you give more money in right and so we collect all these rates we create a machine learning training data set with pair wise comparisons where the rate is the probability of you getting funding from that judge and then we we perform deep learning trying to see if we can train a machine learning model that can distill human judgment and scale it to the whole dependency graph and how we do this is of course we give you that training data set and we expect you to bring your own data llms GitHub API scores reviews whatsoever what you need we run then

this model into an the whole rest of the dependency graph we keep a certain kind of test data secret from the judges to evaluate how well your models align with the secretly labeled data set and that should run hopefully according to millions of comparisons and that is the competition so the jury has a secret test data set and in the end we're going to be like okay AI model number three is pretty well aligned to whatever the film committee the judges themselves think that is a really cool and distillable judgment but we envision that future actually to be more interesting than you think so as we are scaling this competition of multiple machine models trying to predict the weights of which

dependency should be funded or not you can imagine a future where you as a funder have a certain kind of preference that you would like to fund for example code quality that is really high or someone who wants to fund um code that is produced from certain kind of developer groups that are marginalized and you can start labeling your own small test data set and see which of the models that have been submitted aligns the most to your funding judgment and so we think that deep funding is a really interesting approach that's right now happening and open source is just the start because dependency graphs exist all over our community it exists in retr public goods funding and exists in our science and research through citation graphs so to site Kevin um that is open source is

just to start we believe that deep funding could be as an algorithm scaled to much many more public goods mechanisms um than just softare and DEP maintaining our dependency stack so the competition started a couple days ago a couple weeks ago actually and a v has pledged 250k 700k 170k to be dispersed by these winning AI algorithms 40K for the best model and 40K a special price for really interesting like Solutions and ideas and this is the platform where we are submitting right now it's pond.

xyc where you can submit your model U we measure the predi error depending on mean error and you can see currently this is a snapshot from just today um how we have more than 5,000 submissions and several of the members are already striking a really really nice error margin here yeah if you want to learn more visit deep funding dooro and um I'm here for questions da thank you very much yeah I'll just pause the mic um thanks for the talk really really

good um I'm curious to um see if it makes sense um when you evaluate projects um and you sort of have this uh par wise testing um am I right to assume that this requires those people to have a lot of knowledge and then the other question would be um are there maybe instances of dependencies that are so Niche and so rarely known that they would be neglected in par wise comparisons just because maybe just one or two people really know of them or aware aware aware of them very good question so it definitely requires expert knowledge but we believe that the jury design how we right now doing it is pretty flexible so

we actually don't disallow the jury members themselves to use AI models to distill information and help them support the pise labeling U model that we implemented also drags some GitHub API scores and metrics for you to compare directly oneon-one so you're not starting from scratch you're starting already with an interface that provides you minimal information I agree with the thing that potentially Niche dependencies are undervalued but we hope that everywhere little bit of signal comes to the niche um dependency when we do it at scale such that the rep um the distribution is still reflected in a fair manner after the initial testing phase where do

you see the funding actually coming from in the long term so we believe that deep funding itself um is certainly like some kind of a bedge protocol Guild could propose some parts of the money to deep funding to fund the rest of the dependencies and we believe in the future in the long term 1% of all the Deep funding money itself should go to deep funding itself because the measurement and the Improvement of the algorith should of course also be funded um what are other uh use cases that you are excited about um Beyond open source software i i as a as a recovering academic and scientist I think this is um one of the most powerful ways to fund

independent scientists so imagine right now the citation graft itself right now when you are working in science and you're not part of an institution you're maybe part of if you research or your own small nonprofit lab it's really difficult for you to apply for Swiss National funding and other resources because you're lacking that institutional reputation but deep funding on the citation graph can provide finances depending on how well you cited how important you are in the graph so I think that is an really interesting area that if these algorithms work really well on open source software this could be the next step we're looking into yeah please um so Suppose there in the dependency graph there are some

projects that have already received like millions of dollars through retrop pgf from optimism um but now in the Parise comparison they are still like valued because they are super impactful yeah um is there some kind of a proposed mechanism to take into account how much funding they have already received or is that part of the Judgment In the paway comparison how are like the judges supposed to know or is there some kind of thinking around that or is that kind of a later problem if this really works we do an we do right now analysis on the human judgment that we have received so far from the judges and passed funding and see how much that is correlated it does have a correlation coefficient but it's not very strong as we expected so but definitely there is a correlation

one of the things we can imagine this to happen is in in the Deep funding round that is personalized you can yourself account for this because imagine you're um imagine you're metamask and you want to donate some of the money back into your dependencies and then you know that certain kind of dependencies have received a lot of money and you can instinctively through the labeling of your initial private test data could assign this consideration implicitly and then the algorithm itself could distribute it out there are also ways where we are working right now with agent allocation so large language models that um can get a prompt and allocate so some of our users have actually submitted large language models and in these large language models you can even put it potentially as a prompt

as a value that hey um I would like to use you model number 32 Waton one but like with this specific prompt of not using too much funding thank you very much um thank you David for all the great insights I think any startup needs funding so this is great initiative uh and it's been amazing to have you here um our next session is starting in in four minutes and those big Applause [Music]