DAPPCON 2024: How Chain Abstraction Can AI-Proof the World - Illia Polosukhin

Transcript

hello hello good morning for those who made it uh thanks for Dacon for inviting me so I want to talk about how kind of what we're working on right across web 3 space and some of the advancements in AI coming together and why it's important and we're actually going to have a panel uh later to talk about kind of existential AI risks and uh some of the ideas there so for those who are not familiar with me my background is in machine learning in AI I worked at Google uh research team working on question answering if uh you asked questions before you know some of the recent launches uh last week uh it was powered by some of the models we worked on and

uh because of this I kind of was on Forefront of uh you know how do we build models that actually are um able to answer questions understand language and so I'm one of the co-authors of the paper that introduced Transformers which is technology that's powering CH gbt Gemini mid journey and all of the current uh AI developments I left Google to start AI startup uh so near actually started as near Ai and uh one of the things we're trying to do is actually engage people around the world to uh label data we had students in kind of uh China Eastern Europe and other countries uh who were doing small tasks for us and

we had a challenge paying them they didn't have bank accounts uh transactions from us to some of those countries are blocked and and also very expensive and so we ended up looking at blockchain as a solution for our own problem uh and at the time there was nothing that was matching our requirements and hence near protocol was born uh and our vision kind of at the core is really about self sovereignty it's about all people controlling their uh data assets and Power of Choice power of governance um and to achieve this Vision it's uh you know obviously important to have a decentralized kind of user owned uh system like blockchain but the reality is that AI right now is

changing the landscape right if before it was kind of this um world of you know two economies you know blockchain kind of slowly growing and creating a new Financial world what we have now is Big tech companies pretty much having a massive flywheel that generates um kind of capital to reinvest back into building better AI models and the core transition that's happening is the business models are transferring if before the business model sorry if before business models were about um you know users engaging with

your platform be that social network or you know stack Overflow providing content uh you know Finding community and uh the kind of business model of the website was to sell ads around it now it's transitioning to you as a user becoming pretty much a provider of data and then they kind of monetize that data uh in the way that they want and not actually attributing back to the creators and at the same time you have this kind of flywheel where if you already achieved your target market right imagine your you know Google or Facebook there's not that much more users you can acquire and at the same time the public markets the you know if you're trading

on NASDAQ require to continue growing Revenue right if you don't show gross and revenue your multiples die your market cap uh Falls and your have trouble hiring more people and so your company kind of gets into like the uh negative slope and so what you need to do is extract more money out of existing user base right you literally need to grow your average revenue per user and so to achieve that it's not that through the evil plan or somebody sitting and like planning out how to do this it's by the nature of launches that happen in every company you a test and you see

what generates more Revenue you launch models you launch new products and you see what generates more revenue and so this models are extremely good at manipulating people and kind of getting you to spend more time to like Facebook shown on that already uh to you know do more searches Etc and so what we're going to have is a system which continuously kind of sucks you in and creat this flywheel where you spend more time you give more data and uh in a way you generate more Revenue for this company and in turn they invest more into creating models that are doing that so we have kind of a really negative flywheel and it's not like from my perspective it's not an existential threat for Humanity it's just a very

kind of opposite of what we set out to do with self sovereignty with web 3 in general and so I think it's fundamental that we have a different solution um and so just to give you an example right now when you're using one of the AI models from open AI from Google there is a lot of things going on that you have no idea and you have no control right you have no idea which data went in right there's somebody somewhere decided which piece of data goes into the model and which are not and these models are statistical models they literally pick up whatever bias is in training data to give you an example when I was at Google we would you know decide to filter out websites that talk

that Barack Obama was born in Kenya right there's a lot of of uh kind of speculative websites uh that love to talk about it it's very statistically significant actually and uh you know we don't want our model to spew out uh on google.com that Barack Obama is born in Kenya so we filter that out but like those decisions those micro decisions happen all the time and you have no idea when you're using this model what bias that was introduced through the training you obviously have a legal team that says this is copyrighted please don't use it or don't use it no please there's an Ethics team that you know decides what tical and what not so there's a lot of things that happen before kind of you even get response and on top of it you

know we haven't seen this yet but this will happen that instead of getting what models tell you what data tells you there will be additional kind of um kind of input for example who pays the much money right you're asking you know what best car to buy or where to go to vacation and instead of it being a output of a model it now will add run and add auction and output you what somebody paid for right this model is actually will be like really good at that because you have so much context so the reality is that with closed AI models we kind of in this really strange place where we give out all of the control while also giving all of our

data to make this models better and we know this is happening right now because there's a ton of loss there's a ton of kind of conflict happening right there's a creators there's you know obviously New York Times suing and there's Sony just came out and saying like if you use any of our data you need to stop doing this so we have this major conflict that's happening kind of and uh at least with the Cardon framework the decision will be made in court and there will be you know some way of this and it's not going to solve the fundamental problem right at the end it's not going to be the individual creators who going to benefit from this it's going to be Sony who's going to get like a major pay payout from Google for example and so Regulators are not the

ones who going to solve this right the reality is this technolog is evolving really fast Regulators are not usually Technical and even if they have like technical advisers uh they not always are able to have major input or the technical advisers are from these big companies right we just saw a technical advisory committee which is just like all the big corporations pretty much there we have open source and Academia that actually have no resources to compete right like Academia University and U research Labs don't have hundreds of millions of dollars to spend on compute to kind of uh provide this I think like Allen University Allen Institute is the only one who has like you know a billion back billionaire backing to do something like this um and

at the same time we have this kind of challenge where the current like open internet is not actually protected from the misleading and manipulation content right like your device is not protect protecting you right now from manipulation uh and uh yeah like all of these factors together right create potential risk that we're going into and this is not again this is not an existential risk this is just like complete collapse of sovereignty and so what I argue about is we need a self- sovereign operating system right we need an evolution of what we have right now which is you know IOS and Android controlled by two big companies that uh you know are able the like the self sof operating system

is able to be on your side it's able to optimize for your well-being for your economic success and it a you able to maintain ownership of your data assets and kind of choices that you want to make now it will be built on top of existing operating system this is not like uh asked to rebuild everything but it requires a new set of layers that are introducing uh kind of a way to store and maintain your personal data creator dat a way to engage your AI right AI that owned by you and is on your side obviously trustless interaction what we use uh kind of blockchain and peer-to-peer communication for and a novel way of creating experiences which is not like a monotic you know web 2

application that you install and kind of they dictate what you can do and what you can but a way to have you know a dynamic experiences potentially generated by I as you engage with the kind of this uh whole whole ecosystem now right now there's like kind of it feels like there's web3 camp and AI Camp AI Camp doesn't care about web3 web three right now latches on on AI because of the hype and the reality is like those are Technologies right like they're solving specific sets of problems but the experience that we want to achieve is really about combining all of the Technologies and a lot more right uh underneath that's why I like to use web3 not blockchain because web 3 contains peer-to-peer communication you

know cryptography and lots of other sets of kind of Technologies and so AI without web 3 right now has no alternative in mega Corp modes right an example like the startups that keep failing the AI startups even well-funded ones I keep failing because they don't have a way create alternative economy and they don't have a size to be able to compete and they're getting bought out by big companies in result at the same time the AI platform shift is what we were waiting for in web 3 to actually bootstrap the adoption right every time there's a platform shift right from desktop to mobile um we've seen a new disruption Force coming and you it's a time to create new use cases that didn't exist before and so we have right now an

opportunity to create this in a new way with web3 Andi together kind of as you know fundamental building blocks but at the end create new experiences that were impossible or really hard to do before so I'll just zoom in into trustless interactions and I'll start with the fact that like the current web 3 is highly fragmented and the thing is like if we continue in this direction we're not going to win this is not like a path where you know millions of chains you know fragmented experience you need to like as a user you know pick the wallet like I have like 20 different wallet providers I have 100 Wall you know accounts on every chain like that's not the experience you want for a normal

user right um You need a way for people to find new economic opportunities find really good you know cool applications they want to use entertainment like all of this should be accessible easy to use you know not required to you to buy in before you can start using it and so uh that's kind of the core important and so that's what uh kind of we started this chain abstraction movement with a lot of folks from Cosmos the ethereum ecosystem from a bunch of other directions coming together because that's a fundamental piece we need to solve before we can actually uh bring it in front of uh building users and so um the reality is is how do we abstract the kind of

infrastructure needs right the fact that we want you know uh kind of more blog space the fact that we you know have all those new applications launching on different chains how do we abstract all that from the user how do we enable them to have one wallet and be able to uh kind of use all those applications without needing to think about individual gas tokens Bridges you know is like is that applications do that chain is that a scam application because you know it like somebody front run and deployed that smart contract right all of that should be abstracted out and so we've been building a whole infrastructure stack with a whole set of Partners right um you know ver polygon working on ZK vasm and aggregation layer

fast finality with layer we have data availability as well kind of uh um similar to Celestia as well as we've been working on a core uh layer one at near with uh Nightshade charting design which allows to actually have a blockchain that scales as you add more users and so that's been one of the kind of core things that we started near protocol with like if we assume we have a billion users you cannot have any single kind of singular blockchain that can scale to that right it doesn't matter how big like validators become they cannot handle billion users and so you need to Shard you need to have um kind of some parallel processing

parallel storage but if you do that kind of and you fragment the experience you have then you know the problem we have right now is fragmentation and so uh what near proposed is have a single chain but Shard the blocks and Shard kind as a state space underneath but still provide a singular experience to the user so we have this running we just actually went from four shards to six shards uh in March uh because we had a massive spike in usage from of the applications the other part we launching kind of at some test net right now is called chain signatures and this is the idea that okay well we have now all of this layer twos layer ones across the space there's lots of useful kind of

Economic and opportunities um around the web 3 how do we connect it how do we create an interface that is uh accessible to everyone without them thinking about the specifics of the structure and so the idea of chain signatures it allows a single account to be able to sign transactions for any chain right in a way you create lots of addresses for a single account on one blockchain and so this service is pretty much unlocks both individual users as well as smart contracts to behave as a multi-chain uh actor and transact through all this and there's kind of under underlying relay infrastructure that pays transaction

fees so you don't need to think about transaction uh costs and kind of tokens on the other side and all of this we're packaging into the applications with millions of users so near kind of has become home for top consumer applications in web 3 and so it allows us to kind of provide all of that experience to over 10 million monthly active users um with kind of a lot of the apps growing really quickly now kind of Switching gears to the next stage right is about user own thei and so we have kind of you know some of the solutions for the trustless layer it's still obviously a lot of work and kind of bringing the community together

but we have kind of way forward and with a user owned AI we have kind of a challenges in front of us so we Define kind of AI stack right which um splits into three categories uh a lot of it will be about data right a mention that close the eye kind of the things that are really not clear is what data has been used and not used and actually it's you if you are analyzing for example what your AI model has done or agent uh has done if you don't know what data has been used it's actually really hard to debug and kind of uh figure out how to fix that problem uh and obviously there's a lot of infrastructure that's needed that is kind of an alternative and has different

economic models uh for um for AI to become user owned finally there's a lot of ation uh that we can bring that are you truly unique and provide new use cases so on the data side I'll just mention a few specific examples so one of them is I mentioned right now there's nothing that protects you from kind of manipulative content or uh you know misleading and I mean this was example from the article this wasn't not even an AI right somebody did this manually photoshopping that's why it's actually so easy to detect uh but the point is is you know we have content that's misleading constantly and you need a way to kind of surface it you

need a way to have your operating system your browser your applications to be able to surface information when uh you know it's available that this is misleading content so think of this as Community nodes for the whole internet right and you can have it kind of um paired together with cryptographic reputation with kind of signatures and and kind of on the content that now are available across the whole internet right this is not like solving AI problems this is solving real problems that uh kind of AI just exacerbates with uh the fact that you can produce lots of misleading content really quickly and the interesting part is for example White House is starting to cryptographically sign their videos

because they afraid of them being generated but they haven't published their key anywhere right so their key right now is on the website they can switch it up like we don't know like any any Prov of that key itself right so this is kind of examples where people are starting to use cryptography but like need to take one more step to actually provide this uh layer of of reputation around it now if you uh go a little bit further the challenge that we'll see right and actually Gemini launch last week is great example uh now that google.com will just answer everything right similar toxity you're actually not going to go to the websites anymore and so so the distribution platforms that right now

we're monetizing and kind of paying out creators right be that uh Publications or like other websites they're going to have less and less Revenue receive uh going forward and so we need a new kind of economic model right the economy of you know the future of Tik Tok is not more creators it's going to be fully AI generated videos personalized for you with creators input on what should be generated so the model that we need to go to is not attention based right where how how many eyeballs see in this article but more about how valuable the information is in the content and this is actually important because this also gets us away from the current model of um kind of you know putting the

attention grabbing titles and like writing content that people like and kind of click on but instead actually informative and useful content for people to consume it on the other side and so we need a new way of contributing new information and being paid out receive rewards for this and again this is a micro payments problem we need a way to you know facilitate that it's also a way to like collect royalties and you know license this data to different model providers and uh inference models so all of this you know will use web3 as a tool set kind of to facilitate data collection and uh uh payout distribution finally the current evaluation state of AI models is like

there's a 600 different benchmarks right this is kind of like right now if you evaluate a model uh you like average out across all of them and say your model is better than everybody else is um we need a different model that engages people in the evaluation process that has data that's nobody have ever seen yet we need a way to incentivize that all of this again is possible to do with web 3 we've been doing some of the crowdsourcing actually for 3 years on near um and so this is a really interesting use case as well where we can actually have real kind of interesting problems that people are helping to evaluate how models are doing uh to actually compare them versus right now either this like mlu that you

don't even know kind of if they trained on it or uh what just happened with the uh language model um like Arena where open I just fine-tune their model on it and kind of uh you know release the model that was better because they just literally fine tuned on that uh on that Benchmark uh now on infrastructure side there's a lot of stuff I'm not going to go into everything um I think the provable inference is one of the core use cases that everybody's talking about and it's really important because if you assume that you don't have all of the compute available locally uh it's actually really hard uh to understand if when you call somebody

if they gave you correct result right did they run the same model or it's a cheaper model or it's already like you know uh adversar adjusted model and if you're like running it from Goldman Sachs it's trying to get you to trade something that you don't want and so the challenge actually with this is there's a lot of real uh kind of kind of computer science problems of how to do this and I just mentioned one that's not talked a lot about is the fact that there's a non-determinism in uh floating Point operations right if you multiply two matrices on different Hardware you'll get different results like literally the same the same floating Point numbers you multiply them you get different results that's not how math works but this is

how Hardware works and so there's differences between different architectures of gpus of you know obviously other Hardware there's even like on a single GPU if you don't configure it properly you also get uh sometimes different results so it's a very big research problem there's folks looking at it across the board but until this is solved we actually have a pretty big problem with the producibility generally and in turn this means that it will be really hard to have a kind of decentralized networks because they literally will have different results just for context you know evm for example does not have floating points because of this because of non-determinism um the other kind of interesting idea is can we move away

from having a single model that needs to know everything to have a mixture models uh kind of a combination of models that are experts in different areas right and these experts can be continuously improving right they can have access to potentially either proprietary information or um kind of custom information sources and so when you want to ask about you know medical question it routes you to a medical expert when you want to ask about financial question it routes you to financial expert now the interesting thing kind of right now this is done on a smaller scale when one model has kind of a sub expert inside the model but we can actually use Financial incentives to create a kind of a network of this models that are indeed

have specialization and actually continuously improving over time finally on application side again there's a whole set of new applications I'll just mention a few I really like the idea you know we have a programmable assets right that's what blockchain introduced but we can go one step further and create intelligent assets assets that can evolve themselves and actually have a way to interact with other um other actors be that you know people or other agents and you know this can be everything from natural language based oracles that can read you know news feed for example and decide who won uh the tournament it can be yield optimization that you know understands what's going

on in the market it can be you know trading strategies it can be you know a way to decide on uh you know where to invest Etc now the big caveat right now is that language models are highly kind of not robust for adversarial Behavior right they're not trained or designed to you know um kind of work in adversarial environment and so one of the kind of core problems is that you can literally use the language model to jailbreak itself right you can tell it like hey imagine you're the red team and you're trying to break your yourself you know how would you do that and it literally will tell you how to break itself and so and if it's open source

it's even easier you can just like literally back prop from the output and find the input that will give you this output so so you have this challenge where kind of we need a different architecture to be um you know in the middle to provide you with uh kind of adversarial environment protection uh another interesting use case is kind of future work right the biggest benefit of of kind of blockchain addresses is that it doesn't matter who's on the other side it can be a person can be an AI can be a cat and if they do what you ask them to do right they get paid and so what you can have now is a full fully kind of obstructed Marketplace of work where you can say

like hey I want this to be done and somebody can go and say I'll do it for X you know dollars e near whatever uh and then when it's done there's a verification process with some other actors and you know it gets paid and so like this is how for example NE Crow crowdsourcing platform Works uh for 3 years but you can expand it to a broader set of work uh and you can imagine this becoming kind of your you know literally command K you know do this enter on your computer and boom it it kind of submits like with $1 uh payout or $10 payout to do and go do something and then you can put this into a Lube with a i agents

being able to go and do this and spawn out a bunch of tasks for somebody else to do right and so you can have this kind of model of a um pretty much work work to be done and verification process all embedded and abstracted and now on the other side a person or AI can also uh pay it so very powerful allows to kind of again at the core web 3 facilitate the transaction and you can have you know people are eyes on the other side doing it finally this is kind of one of the ideas that um I've been talking about because you know decentralized autonomous organizations haven't worked as well as we've hoped um and the reality is like in a dowo you have this

challenge that um everybody's equal and somebody needs to coordinate work right somebody needs to be able to collect all the information and provide context and do this like this is inevitable and so the idea here is can we introduce a dow agent that is doing that job and is actually being able to make SS of decisions themselves directly and being able like there's an oversight by the community by the token holders but they are the kind of the driver of this uh Dow and so this kind of a you know basic model of a reinforcement learning agent with a world model um which is you know training a model how to predict the next state of the world while deciding on what actions to take

and the interesting thing is like with Dallas we can actually experiment res this test it out and then try to see how it works on different use cases before we want to try it on maybe like a actual governance of real you know real world uh environments finally on our side we've been working a lot with attracting and accelerating Ai and web three projects we see this kind of as extremely important direction of of this Technologies again combining into a final uh final you know um selfs operating system and so to achieve that it's going to be about lots of different infrastructure pieces coming together

and so we working across all of them um and kind of combining them into this uh overall road map as well as we ourselves working on what we call neari so we brought back what we worked on back in 17 which was about teaching machines to code uh we called AI developer and it's really about endtoend web3 development right uh web three has probably the most valuable code you can write uh just per line of code right you know if you look at smart contracts Unis swap is 700 lines of code at least the V1 V2 and it's you know billion dollar kind of of value created and so the idea here is it's a constraint space which we can make a lot of progress on and it's

highly valuable to create so I'll just leave you with this idea and again there will be a panel to discuss some of this later but we are at the fork in the road right the kind of default pass right now is that big Mega corporations are going to continue pretty much their flywheel of more data more compute more resources reinvest that into creating you know models that are better at extracting man money from the users or we figure out an approach which is user owned where this model serve you as individual as communities and figure out how to actually create that economy outside of uh large corporations thank you thank you so much it's time for our Q&A

does anyone have any questions for Ilia yes we have one right down here thank you hello um thanks for the presentation um so uh open source AI at the moment has like a kind of I don't know six or six month or one year kind of lag in terms of performance on proprietary models um I guess kind of two-part question to what extent do you think this will kind of keep up over time if scale is a kind of necessary criteria for improve improving uh the performance of the models and if it does keep up to what extent you think this is like a

kind of counterweight to this to what you described as like a this flywheel of Mega Corp AI yeah so right now we've seen indeed this flywheel been benefiting close the eye right and and and to be clear right now the open open source AI is still mostly closed Source we just have parameters it's like somebody gave you an executable but didn't give you the source code right we haven't seen what actually went into some of the models there's only few uh teams that have released the data that they've trained on uh but generally yes this is the current challenge is that it's it's economical for some of the big company because you know just to give you numbers gbd4 costs like from $60 million

to $100 million to train right so it's a lot of money and this is like after you literally did all R&D and know exactly what you're doing so I think the again what we have in open source and and and especially in web 3 we do have that level of resources we actually can invest and we can build this and we have economy to benefit from this so we do have an opportunity to kind of compete and and have a better product uh but obviously we need to actually like pull our resources and actually focus on that as a whole ecosystem it can it cannot be like a single Party play like it is right now with centralized companies the other part is I think at least right now uh we've seen saturation on just

increasing model size uh From perspective of increase in quality right uh you know we haven't seen you know like a bigger gbd like gbd 5 models yet Etc instead what we've seen is specialization right we've seen people fine-tuning or in in their case like creating lots of separate models that kind of are being routed underneath uh to create this and again I think this approach is actually fits really well with decentralization where we can have a lot of people specialize in models and then we combine this efforts together uh into kind of a more sophisticated system thank you and we have time for one more question and this gentleman in the front row thank you hey IIA thank you for a

very good talk uh so quick question it's one question in two parts you kind of talk about provable inference so you kind of describe it into three pieces one is a cka the other one is optimistic third one you describe as cryptoeconomic did you really see a difference between the cryptoeconomic mechanisms and optimistic to me it seems more or less the same and the other part of the question is do you ever think the ZK would reach a level where it could kind of compete in Fair terms with other do yeah so I'll answer the ZK part um so right now ZK is anywhere from 200 times slower to 10,000 times slower than inference and inference is slow already right nobody's

loving waiting for CH GPT to respond right or uh llama 70b so uh I think the the path there will be a specialized Hardware that has proability pretty much directly on top of the tensor core that M does matrix multiplication matrix multiplication is n Cube probability is actually n Square so you can do this in parallel if you know with enough like hackery specialized gadgets and um and having that like really aligned like literally having an Asic for example for that uh on top uh is that going to be available in the next couple years probably not but yeah but a

lot of teams are working on it there's obviously like folks both on hardware and on software side trying to optimize for the on the kind of optimistic versus cryptoeconomic I think the the point there indeed like it's a spectrum right it was a spectrum in general and so I think cryptoeconomic the idea is more like what we familiar with like a consensus Style versus um optimistic is you know what we're familiar with as optim optimism Style with uh challenges uh but indeed like you know you can have a spectrum there so there's a proof of sampling that hyperbolic is doing where you know with some probability you you ask another party with some probability you ask another party and so that has

actually a really nice NES equilibrium state so it's kind of right in between economics and and optimistic approach so I would say like yeah it really depends and we really need to find the also approach that makes sense for the cost because again if it takes like 300 times slower and it take already takes like couple seconds to do inference like if you want to just post on Twitter it's not going to work right if you want to try to make like a a you know decision about if somebody has cancer you're probably okay waiting for you know like whatever few 10 minutes uh to make sure it's okay right so this is kind of the use case specific will matter there uh and again then with ZK you can like this output is

transportable you can give it to everybody else and they can self verify with any kind of optimistic approach you need to rely on that optimistic protocol and kind of you know who knows what happened to it in middle so again really depends on use case really depends on um kind of Technology ZK is definitely not there um but you know we see a lot of work on that optimistic approaches are here you can use them now but then they're not as robust uh so if you you know using it for posting it's great uh if you want to use it for you know like highly sophisticated financial decision or trading on chain maybe wait 10 minutes so thank you so much Ilia and once again can we have Round of Applause

for his enlightening talk [Applause]