DappCon 2023: Decentralized AI And The Future Of Web3 - Lum Ramabaja

Transcript

good afternoon everybody I'm Luma and I'm the co-founder of open cybernetics and today I'm going to talk about decentralized Ai and what I believe is the future of web 3 now to elaborate the structure of this talk I will first Define what decentralized AI is afterwards I will introduce some of the technical challenges that this emerging field faces and at the end I will do some of the building blocks that we at open cybernetics are developing to overcome those challenges now to be fair this talk is going to be slightly technical but I'm going to try to keep everything as simple as possible so everyone can follow what I'm talking but yes so before going into the technical

details though let's first start with what decentralized AI isn't so when I use this term throughout this presentation I do not simply mean a large language model that is somehow connected to a Dao or a blockchain I also don't mean a large language model with weights that are somehow stored in a distributed hash table like ipfs for example even though I think that those technologies will play a vital role in future decentralized AI applications when I use this term I have a very concrete definition in mind decentralized AI is a new branch of machine learning that tries to combine techniques from cryptography database

systems and peer-to-peer protocols to essentially distribute the training and inference of very large models it's ideally in a trustless and permissionless way now of of models that one would have difficulties to run on a local normal machine for example essentially the goal of this emerging field is to truly democratize AI now there's a problem though despite the clear utility of large models such such as llms we haven't really seen an emergence of peer-to-peer AI systems in the wild and I argue that there are two fundamental technical reasons for that that first need to be solved before

dentalized AI can even become a real thing the first challenge is that of trust right now even if I were to perform a forward propagation and a backward propagation in a deep learning model in a PE PE fashion there's no way how to cryptographically verify that the embedding that I'm receiving or the weight update truly originated from the deep learning model that I'm interested in furthermore there's also no way for us to know that the input that we provided actually trigger that output which is quite problematic to take an example that would be like having a torrent like a torrent system like bit torrent and let's say you want to

download a file and you simply collect file chunks from peers without ever actually checking if those file chunks are legit or not now such a system would not only not work it would never actually take off it would never emerge as a thing because any malicious actor could immediately corrupt the whole thing now again for decentralized AI to even take off to begin with we need cryptographically verifiable a cryptographically verifiable way to prove that an output from a layer or that the gradient updates for a layer are legit the second problem is that of cost training large language models and in

general very large associative memory networks is very expensive performing queries on them is also quite expensive and the reason for that is quite simple it's computational complexity we know for example if we perform an operation on a linear layer for example it's the computation complexity of that is o n however if we perform operations on Transformer based models that have the self attention step well there the computational complexity Rises to n^ squ now for the non-technical people here in the audience essentially what that means is that the higher the computational complexity of of an algorithm is the more resources it will cost us to use that algorithm essentially in the case

of machine learning models what that means is that only because of the high computational complexity and the huge amount of parameters that we have only a handful of Corporations actually have the resources to train these enormous machine learning models which is also a problem because if our goal is to decentralize and to truly democratize AI we either have to figure out how to reduce costs of these very large neural networks significantly and slor we have to figure out how to collaboratively train them in a f tolerant manner meaning and that's that is important this this gets back to the previous challenge any node in this in

that system could be malicious and technically corrupt our weights so there there we have to Define to essentially build the tools that overcome those challenges now luckily though the complexity problem is not futile there are ways how to fight off complexity a lot of clever researchers and scientists came up with a a lot of different ways how to reduce that kind of complexity in the self attention step by creating very clever archit architectures like the Lin forer the the perform the reformer and a lot of other interesting formers um however there's also another way we can fight of complexity and this other way is the way we at open cybernetics chose which is by leveraging the inherent

sparsity within deep learning models to elaborate what that means let let's first look at the famous self attention function down there in the slides at first glance it doesn't become apparent what that function has to do this with well sparcity in this case but once we look underneath the computations and essentially look what happens at at the Transformer heads some something very interesting becomes apparent so let's walk over it usually we have a usually we have an input and we have three weight matrices in the self attention step we have the WK the w q and V so usually what we do is we perform a lot

of coine similarities meaning that we take the input we do a DOT product with all the patterns in the weight K and then all the again cosine sties with similarities with uh WQ and then again with WV after that we perform an inner product between queries and keys stap a soft MOX function on on that and perform again a do product with the the value Matrix now what is interesting here is that one we are performing a lot of computations and two if we actually look at the submax distribution after that step we can notice as long as the Transformer was properly initialized we can notice that most of the values there are near zero or like almost zero

meaning that actually just a handful of patterns in our memory do contribute to the output of a layer so most of the computations we perform are actually for nothing however we don't know which patterns are relevant before so right now what we do in AI is essentially brute forcing our way through the layer and Computing everything and just finding the right patterns now this brings us to the first building block that we're working on at oven cybernetics I will not get too much into the technical details because the second building block is more relevant to the conference but essentially what we notice is you know you can change the math of the Transformer model and make

some cool stuff like the L for and performer but what you can also do is just perform the operations in a different way in all in the self attention mechanism as it is now so what this means is if we assume that there's a lot of sparcity going on in the model then what we can do is that we can use clever tricks like heris stics like approximate nearest neighbor search and stuff like that to figure out which patterns in within our memory are actually relevant and then we perform only the coine similarities to those patterns which should in theory save a lot of computation so if you would make an operation that is o n in complexity if you figure out how to make the data structure properly like the

approximate nearest neighbor resarch you could theoretically reduce that to login and this brings us to the second building block that we're working on which is of more interest to us here the second building block is the cryptographic ghost proof and the cryptographic ghost proof essentially Embraces this sparsity approach and what it does is it's a very simple way for us how to generate cryptographically verifiable proofs for the output embeddings like for forward propagation and backward propagation in deep learning models now to be fair technically this also works with densely with densely trained models as well but I don't really have the time to go into that we will focus on the spars version

right now so the way how it works is using Mir Miracle trees as everyone knows we in the crypto space love Miracle trees we can find Miracle trees in blockchains to verify transactions within blocks we can find it in git we can find it in bitor systems so we can find it pretty much everywhere and what it does is it enables us to generate very compact presence proofs for elements in a list okay so if the way this usually works is if you have a list you'd simply build a binary tree of hashes your Merkle tree so you hash the hashes together till you reach the root of the tree and when someone asks you to if if something is in a list for example

if a transaction really happened in a block instead of providing the whole block to that user you simply provide the hash of the transaction together with the Merle proof which is just some the green hashes here in the screen and if you sequentially hash that together you regenerate the root and the user knows yes this element really was in that list or yes this transaction really occurred in that block the cool thing though is once we Embrace sparcity we can use the simple same like the same trick for neural networks as well we can build Merkle trees on top of tensors either either rowwise or column wise and then when we per perform forward propagation in a sparse sense in this case of course then we can simply

generate Merle multi proofs and provide together with that with the output of that mle multi prooof to verify that it's true now to elaborate a more concrete example how this would look like this is a an example of a layer more exactly this is a modern hopfield lookup layer in this case what we have is we have a key Matrix a value Matrix of Weights in that case and we want to propagate our input through it to to to essentially get an output so what we would usually do in this case is we provide the input mean we perform a DOT product between the input and the key Matrix generating a matrix of scores and then we apply the soft Mark there and we

perform a weighted summation over the value Matrix simple enough now we just attach some merel trees on top of that so on top of the key Matrix we build one Merkel tree and on top of the value Matrix we build also a Merkel tree and assuming you use whatever kind of turistic to SP sparsify your forward propagation what we do in this case is we perform a DOT product between our inputs and in this case the key uh weights we pick the top scores in this case simple example the top scores are H1 and H5 and for those two scores so to say we also Pro generate like we compute the miracle multi proofs for

those scores then in the value Matrix we do the same thing for the corresponding indices and we can generate the output the cool thing is then that the output gets sent to the user together with the sequence of Miracle multi proofs which we call the cryptographic ghost proof and the user can know for sure that okay this forward propagation really happened from the deep learning model that I'm interested in now this simple trick is super cool because it opens up a whole new realm of possibilities for us usually how the web works is that we have a client server model and we make we send some requests we get some response however with this kind of approach we can essentially create now now new kinds

of cognitive models where I can send maybe an embedding to a server and that server then propagates that to n layer whatever because the server might have several layers and return to me the desired embedding together with with its cryptographic ghost proof however once you realize this you see that you don't technically need the client server model at all you can dis that all together and have a peer-to-peer Network and as long long as the nodes within that system are aware of the model that you're like interested in or have parts of the weights of that model this is where the ipf where protocols like ipfs might come in handy then you can directly interact with those peers and they can compute forward propagation and you technically

could also collaboratively train very large models because once you get those embeddings the proof to check if something is legit or not is quite Compact and doesn't Co cost that much now because of all these things I described I'm quite optimistic for the future of AI for example I'm quite sure that in the near future like less than 10 years probably majority of internet traffic will actually consist of cryptographically verifiable neural activity by neural activity I mean forward propagation backward prop backward propagation stuff like that and I'm pretty sure that someone here in the audience or someone who's viewing this will come up with even smarter ideas that have something to do with knowledge

and make even more elegant Solutions than that however this very cool hypothetical future will depend a lot on us we can either remain idle and indirectly enable a handful of Corporations uh to become even more powerful than they are with this type of technology or we can start working and developing the tools and protocols required to truly decentralize and democratize Ai and in the process Empower communities worldwide I say let's build that future together thank you a lot for your time I'm super interesting talk thank

you very much do we have any questions there's one question where there um yeah this is super cool um I I probably won't have have pretty good questions but I can ask some basic questions like uh is there a code to explore presumably yes and then or hopefully yes so we are what's the stage of that yeah we are very early stage so whatever I I talk today we have also written it in a quite a Long blog post so you can read everything yourself um right now we don't have anything public um we will probably have public codee in

best best case scenario I would say 5 to six months U sorry quick followup question and like this is super interesting idea I was really fascinated um so congrats on that but like you have you thought on then adding privacy to it I don't know whether you could do this uh yes like you could dis distribute it and possibly also use either ZK or maybe F or something for this nearest neighbor approximation I don't know um to to like even to the network which inputs or which vectors you were searching for yes um that is something we want to explore um definitely first though for the first phase of this project we want just to

show that look guys this works you know and after that we definitely want to explore like zero knowledge approaches and all sorts of stuff yeah any other questions on the right hand side here wait wait for the microphone please thank you for the insightful presentation uh how do you see the open data market for training models in this schema so you talked about like the end result but to distribute like data for learn it's like another another site of this medal yes um that's a good question and I would have to talk quite long to

give a a satisfying answer to that um essentially so the thing is this um I personally am a bit SK so the thing when people say we're going to run out of data and stuff like that um I'm always a bit skeptical about that because as long as we come up with novel Sol new Solutions how to train the models having simply the Corpus of all the books in existence should be quite good enough for the majority of services and tasks we wish to have AIS for so I like when people say like for example oh we scraped all of Twitter sure but what kind of quality data do you get from tweets like we all like to post so

um I regarding the data scarcity I don't worry really about that you can just grab all the books in existence and that should do the trick in the near future um and regarding the market part that you mentioned I'm also a bit skeptical about that um because unless so technically data once it's the data is out there it's very difficult to keep it private and not reuse it for other stuff as well um there are approaches how you could do could do that but generally I think that markets for data will not really take off that

significantly because what I believe is that for the majority of tasks um the the Surplus that these AI systems will generate will inevitably converge to zero the reason why I say that is that for as long as there are many models that can offer the same kind of value and there's no huge discrepancy between utility when the performance slightly increases then you can expect a lot of competition and as a result like the profit you get from that will converge like really rap ly to zero so what we will see is that for the majority of services because I believe that the majority of services are not that that difficult we will see a an

immediate price drop so in that case markets will kind of not benefit that much anyways but for some tasks and services we will most likely see winner take all scenarios and in those cases the market idea won't work again because the real value is in that data no one would want to sell that data in the first place like you you can you can make way more profit by keep siloing it unfortunately yes hopefully that we can talk privately as well yeah any other questions I see one we all on the right yeah um yeah related to what you just talked about if the data for llm and llm

and embeddings is kind of publicly available that centralized players can basically just use all of that data um like where do you see the value of decentralization for AI in terms of the data and the different models and results that will produce is it like the labeling of data or is it different subsets of the data or or how do you see decentralization driving value there in general yes very good question here I would differentiate between use value and exchange value um so what I believe is that big corporations are not really incentivized to adopt the decentralized

AI approach um because a decentralized AI approach until someone comes up with a zerar solution will inevitably um enable the propagation of of Weights in a peer-to-peer system and essentially the weights won't be secret to individuals um so what I see is that these systems will have immense use value in that you can just use them just like that without requiring access to an API uh but they will not generate exchange value so most services will you won't really make a lot of money um you need some sort of information asymmetry

to generate revenue and unfortunately in the case of AI the only way how to do that kind of information asymmetry is either by having access to data that others don't or by somehow performing regulatory capture and others cannot really do what you can do so those techniques corporations will Embrace very freely uh the decentral the Dynamics that come with decentralized AI would not really be in their favor yeah okay good I think we wrap it up here um if there are any other questions for the

speaker I think L we find you around here yeah I'm around perfect thank you very much for your insights give it up for again