DAPPCON 2024: Abundant Compute And Intelligence Through Scaling Decentralized AI - Vincent Weisser

Transcript

hello everyone uh thanks for having me um yeah so I'm Vincent the founder of uh Prime intellect and um yeah for the context our focus is really to decentralize AI also by really uh figuring out how to do decentralized compute but also another ways to advance AI models themselves um yeah and for big of backstory like spent the last few years um kind of helping build up some of the first communities in decentralized science so learned a lot about how to coordinate uh concretely scientific funding on ethereum over the last three years uh started a project called vow that some of you might know and spent also the years before that already in kind of like tinkering in AI but also in crypto so it has been always

been uh two of the spaces that I been close involved with and so naturally evolved and asked myself also like really how can we um decentralize AI in a meaningful ways Beyond kind of like hype um and that's what this talk is going to be about so really about how to get to relevant amounts of compute and relevant models for solving and and funding and advancing AI models through decentralized AI yeah so the the broader Vision kind of like with and motivation for starting this was really to kind of um emerge from talking to a lot of AI developers founders academics at AI conferences over the last few years and

just friends starting AI startups is really empowering them with a more instant and scalable uh compute but also really with novel incentives to make open source AI models work and maybe to focus on the first one really like if you talk to AI researchers and startups like the only thing they all have in common is that they don't have compute basically like and I've been chatting with them like over the last even like few days with a bunch of researchers doing some of the most meaningful models and the only thing they really uh want and don't have is compute because otherwise you can't do anything um but then also like they they kind of lack the incentive models and Frameworks to get funding for open source AI models like basically they rely on donations to

an extent um and the broader goal with like solving this is really to kind of like Advance scientific progress and really make uh access to compute universally accessible as well as to intelligence and I think like it we becoming increasingly clear us throughout the presentation but like that uh it's really should be almost like a basic right to have access to compute and intelligence especially in a world where like all of your intelligence sorry to break it but will be basically worthless to an extent probably compared to artificial intelligence so people will rely probably on having access um so talk is kind of like in three parts on like going into compute and intelligence and then also how Pro a protocol fits into that and like how it could look like in

a decentralized world and maybe like to start off I'm not sure how many of you are aware of like the scaling hypothesis or have heard the bitter lesson of the scaling hypothesis okay not many so that's useful because like it's basically like uh the biggest lesson in 70 years of AI research is that generally methods that leverage computation uh are ultimately the most effective in by large margin and this uh quote from Rich Sutton like one of the key uh former scientists at deepmind reinvent uh invented reinforcement learning um yeah and it's obviously clearly shown also on this chart to the right is kind of like since we enter this like deep learning era the main thing that changed is actually not algorithms like gbd3 or four is quite similar to gbt2 the

difference is just like orders of magnitude more compute and literally like gbt2 I think was like 20 30k of compute gbd4 was more like 200 mil so it's like a huge um jump in compute and and obviously also some algorithmic improvements and so the first realization we had was like to an extent it's obviously highly centralizing Force because you have these gigantic data centers like op has like one or two data centers with literally 10 20 billion worth of compute in them and the only way to extent to compete is like if you can bundle and aggregate all the other Compu in the world um in every other data center in every country literally at some point maybe even in every laptop and smartphone into like one kind of like aggregated uh network but it's very challenging to Train AC cross

distributed compute even if it's in two three different locations and especially if it's like hundreds or thousands of different locations so that's one of the core problems uh we started to tackle is really how to train the centralized uh how to orchestrate this compute in a yeah like global fashion um and we we started basically with this first goal and like launched um our compute platform in private beta like which basically Aggregates compute from uh all the different GPU clouds but then also enabling people to contribute their their own gpus basically onto the platform um and it already like enables people to basically find the cheapest chip uh that we can find in the world like to train AI models and I think the key thing to focus on I I think also

from lesson from observing crypto is uh to like not only focus on the decentralized clouds which some of you might know like aash or like Jensen or a bunch of others uh that are doing decentralized clouds render Network and a bunch of others uh but also the centralized ones which still are 99% of the cloud so it's kind of like just focusing on a decentralized clouds would be like a drop in the ocean of like the computer that's out there um and a lot of people also uh like I don't really care if it's like centralized or decentralized so we kind of like go with both um yeah and the focus really is on the one side to really aggregate all the compute we can like in the world and make it usable like for people to use

for training fin tuning for inference and other things um and yeah this is kind of like I'm not sure if the view works but like it's a bit like how the platform works like it's just people basically looking for the chip they are looking for uh and finding like the compute that basically is like the cheapest but also most performant for their need um and I think where this gets interesting and which is one of our key technical uh research focuses is uh on decentralized training and for context basically um most of the big models have been trained in in a centralized fashion like gbd4 or or others but an interesting shift was actually recently by by Gemini by Google like they started to train distributed like across multiple clusters because

not all gpus could fit in one cluster so they basically were forced by like like the natural limitations of space and energy to uh train distributed on two or three clusters um and this will only increase like we'll like it's basically now kind of like known already from the comments also from the I people that like we won't stop at 200 million like the next training ones will be a billion and then there will be 10 billion and 20 billion probably like stops out at like 10 20 billion but like that's like huge order of magnitude jump from where we are currently so it's only possible in a decentralized fashion and and in fact deep minder is already doing it in a decentralized fashion um and we are basically right now focused on actually building on top of all the research and decentralized training actually

replicating what Deep Mind is doing for the open source community and making it freely available so they haven't uh publish a GitHub they haven't uh but they they've done a paper on this so we are basically right now um yeah like open sourcing this and and building this out but then also adding a few properties like the key is really to make this fall tolerant so if one of the gpus doesn't work that it it can just replace it with another different uh GPU but also to an extent um enable hydrogenous like different basically gpus so you don't are limited to just using for example Nvidia h100s but also could use old the generations or different kinds of chips uh and then I think the key requirement which we focused on is really this low Network bandwidth because you can imagine if you

take a GPU in different places say in the US and Europe like you'll have to send data between them and that's one of the key limitations and uh there's ways around that basically by synchronizing less regularly less frequently uh and and figuring out ways to approximate this in a way where it doesn't become a huge issue and that's basically what Deep Mind published on and demonstrated and what we are now replicating and and Publishing is figuring out ways to um yeah like solve the inherent difficulties with these training um and the key benefit is of course really in going back to the first point is like if dollar is kind of the input you want to get obviously the most amount of compute and the cheapest compute is usually the underutilized compute somewhere in the

world um so the whole idea is basically to always enable people to basically plug together the cheapest computer they can find in the world and and train across the uh the cheapest yeah basically gpus people are willing to provide um yeah so that's basically what like like we've uh built on this um open source framework also called hi mind additionally to what we're replicating from Deep Mind which is uh like an older already decentralized deep learning framework written in pytorch and we basically enable people like straight from our platform already to basically contribute their compute uh to this uh decentralized training run to train the first like AI models in a decentralized fashion basically where like hundreds of

people can join with their compute to train it it's more like a proof of concept to show that it's POS possible at similar actually um yeah like efficiency and um yeah that's basically like one of the first things we're building right now is really um like building in in MVPs and in in quick iterations like different paths to to this vision of like hey let's train on like every GPU that is underutilized in the world um yeah and then maybe like going into like how for what to use all this uh gpus and this compute we basically of believe that um obviously we'll have these gigantic models trained for like 10 billion 20 billion by open AI Open Source by folks like uh Mark Zuckerberg with llama uh but we'll also

have like a range of like specialized models obviously some of these fine tune models and also kind of like for the related to the space I spent a bunch of time in with um decentralized science there's also a bunch of like domain specific scientific models that are really interesting obviously uh also like high impact and that actually don't need 10 billion of Compu maybe like $100,000 of computers enough for most of the researchers we talking to and like for example just yesterday I talked to a researcher working on like pandemic prevention and like they need like $25,000 to build their model and this model could literally prevent the next uh pandemic like and so there's a bunch of this almost like lwh hanging fruit to um yeah figure out like basically also coordination and funding models for impactful AI model Builders to get their hands on compute and obviously the key

challenge there is really that we've seen this close and fully open model where like obviously the big guys uh train in a close fashion and then try to sell the AI model as a product like open AI or anthropic or Google even and then obviously the fully open source models like llama just try to like cross subsidize with Facebook for example and I think what we'll increasingly also able to see is some models in between like the best example is um stable diffusion for example which obviously is freely available like the code everyone can use but if you want to use it as an Enterprise for example you have to pay them and I think that's like one model that will be more interesting and more common place is that people can still like everyone can access the code

everyone can use it for free but if you make a lot of money with it you have to basically uh give some Revenue back to the people that trained this model and that's what we're basically exploring as a funding model um to give like millions of compute and then hopefully much more to like a bunch of people building imp for AI models in exchange uh for some sort of royalties and there's this like good quote by actually a former employee from open my I now with one of the biggest like humanoid robot companies and he he himself said like the only um way he could see for potential disruptor to the openis and Googles is if we could figure out ways to on the one side incrementally Fork open source models and train all of these models uh further and and merge them together with all the

distilled compute basically that people are investing in open source AI that people have obviously globally kind of like available um so that's one of the key motivations to also um explore novel ways for people to advance AI models in a decentralized fashion and what you could imagine for example what we're looking into is like if uh say uh Facebook has announced that they'll announ that they'll release one of the biggest AI models like 400 billion parameter model with llama um soon which is probably on par with like the best closed models but there's basically ways to continue training those models much more cheaply than than uh they spend on to really enable a model that always

stays ahead of um of the closed and open models by just like more and more people contributing compute but also data and novel insights to this model and how does a comput like a protocol fit into all this like the the core idea for us is like to uh actually like not start like crypto first but like what's actually does the AI World need and like what do the AI developers need and then figure out like does a protocol even make sense here um and basically how we view this really is like um this like ultimately comput is a commodity and a commodity needs like a market needs like Market mechanisms to be traded like efficiently so like like I'm of the firm opinion that like

majority of GDP will be spent on compute like won't be spent on humans won't be spent on other things like it will be humanoid robots will be like factories but it will also be like a lot of lot of compute because it ultimately scales infinitely right like if you imagine you have like a few like tens of millions of this worldwide you could have like 10 times or 100 or like a billion times more compute and intelligence spent on science just by ramping up basically uh scientific AI for example so like the the whole idea is that like Compu might become the biggest Market in the world and uh and it's still not very efficient like it lacks kind of like a NASDAQ it lacks a way to just uh use all of that compute so that's at the heart of it and um really it enables a more efficient

price discovery on basically every sort of compute so basically um obviously what we've seen with a bunch of the decentralized AI compute protocols it's kind of like peer-to-peer compute even in an abstract way for me Bitcoin or ethereum are an example of this like it's like actually 10 100 orders of magnitude like more compute uh was contributed to bitcoin or ethereum than for example to gb4 so it's basically existence proof that like there are ways for people to contribute enormous quantities of gpus and and compute power to decentralized protocols um so that's one of the ways that I think one can bootstrap these networks is just basically enabling people to contribute compute directly um but then also really making more efficient use of

this compute and thus like driving up the the uh value for the people providing the gpus but also lowering the cost yeah so that's basically at the heart of it and then really this decentralized training also enables like effective utilization because you in essence could always use the cheapest compute in the world to train uh which obviously is not possible if you need 10,000 gpus in one location but like if you could use a few gpus here a few gpus there and and plug them all together then you could always Train Find You inference on the cheapest gpus in the world so that should in essence make the market more efficient like it it should basically create demand for the cheapest providers of compute power in the world

um and then really connected to that like the the interesting aspect is also figuring out this model for um almost like Community owned Community funded uh AI models and I think there's different paths to that and ultimately a bunch of protocols have obviously tried this out like one famous example is benser which obviously is just incentivizing basically AI models to become better and better and I think there's some really interesting things there but like also like some signs that it's not working perfectly or very efficiently so they spend like a billion or two actually in incentives per year uh and like they haven't produced like set off the models thus far and so they have been produced for much less resources in other places

so which basically leads us to the conclusion they needs to be like a more efficient or better model to incentivize basically AI models uh with compute uh that actually make relevant progress yeah so that's um ties also into this high level idea also like again from Rich Su which I mentioned earlier on the um scaling hypothesis and actually from David silver who's like probably the most relevant scientist at at Deep Mind like it was behind alphago behind most of the key inventions at at Deep Mind and they did this paper uh reward is all you need and I think it really uh also shows nicely why crypto will matter in the AI world and like how also decentralized AI might shape up which is really this idea of like cryptoeconomics

and this concept of almost like viewing crypto as like kind of like an environment where you design on the one side reward functions but because it's also inherently Financial there's obviously a lot of incentives for Bots to get involved there's a lot of basically uh efficiency driven already by kind of like AI in in in crypto like I would argue like from looking at transactions like the majority is already done by Bots and I think that will only increase like wouldn't be surprised if it's the majority at some point and I think that's also how decentralized AI will shape up as to an extent a lot of these like agents and Bots like will look for compute and they'll look for compute in a trustless permissionless way and they'll want the cheapest compute and they'll also like

want to make money right like on prediction markets on defi on other areas so I think there's basically a future where one could see like a almost parallel and growing like ecosystem of AI like happening on crypto rails and um not only kind of like making money on crypto rails and creating value and services but also using compute uh inherent and through crypto rails um and really this this hypothesis of this Antica is that like intelligence and its abilities can be seen and understood as um subserving the maximization of reward so that almost like intelligence emerges out of like a reward function and obviously uh like evolution is kind of like a reward function for humans and

obviously for AI it's it's usually actually uh compute or financial rewards that it's trying to maximize because it's kind of like it's oxygen like if it doesn't have compute it can't run so the one thing it needs or wants is compute intern extend um yeah like resources to run which obviously is kind of like fungible like yes so that's about it like um happy to like if there's time for like questions take some questions and happy to collaborate you can find more out about us on prime intellect. or on on X and yeah feel free to ping me and Havey to collaborate and explore ways yeah to work together [Applause]

thanks thank you Vincent for your informative talk a great example of why uh Boutique and smaller projects can have such a big impact right now uh in AI uh any any questions for Vincent do you have a question okay okay uh if not then we'll we'll wrap up there thank you so much Vincent thank you