DAPPCON 2024: AI Risk Panel

Transcript

this was a heavy talk um and uh I yeah I think I have to give I guess even warning so I had um I'd heard this talk in a in a very similar form already um what is it now almost a year ago in Inu um and to me it actually well definitely took me or for many months to follow that uh well affected me quite a bit and or and uh and I was um very much um well yeah concerned about it and and

and and and thought about uh the consequences um I think my perspective now has um slightly shifted or slightly become uh or see see at least um different perspectives um on on on the issue and I think that's also why we um yeah thought it would be important to basically immediately discuss in a way this talk uh on this panel um yeah and again uh share different um perspectives so I think El does now doesn't need an introduction anymore so now please

convince me I'm wrong though yeah yeah yeah no no I mean we will we will we will go I mean we will well discuss uh um that so um yeah I think also IIA just gave his talk so probably also doesn't need an introduction but but but but still um yeah has is is is is working on near and that's his most or this his largest the project right now but of course has also very very famous um background in in the AI research you've been well on this famous uh now very famous paper uh one of the authors that yeah led to I think always many things lead to to

breakthroughs but this is kind of still kind of cited as one of the most influential papers for for current um llms yeah and we have Andreas who's also now working on something quite different file verse but also has an academic um background in in Ai and also kind of specifically looking into um into risks so I want to start with uh roughly summarizing your argument at least G giving it a shot and then um yeah and then seeing kind of at what points we well either agree or disagree so I would say your argument goes

um yeah we are quite close to have um agents use you use many times use the word parasitic species um I I I myself I'm actually well currently we're still trying to write this blog post a new form of life or agents as a new form of life so I do think I absolutely uh follow that logic that uh a agents um can become its own independent um form of life so just roughly what would this mean so that that means they they are actors that that do stuff for themselves so I think

um that that is something they they are able to reproduce they are able um to evolve I would say those are probably the three things uh necessary to um to to be be a form of life um and and yeah again also to share uh to share that this is really not that far out um we are also with with others and in nosis um tonas and many other projects we are right now setting up agents we are right now setting up agents and right now what they do is trade on prediction markets um but but

but to do that they already need to uh get knowledge about the real world they need to understand well I mean what does it understand mean but they need to well effectively uh yeah be able to make good predictions about what's what's happening um and now the important part comes by doing so they can earn money um and that money they can hold on a substrate uh that is the blockchain where um where no one can yeah potentially take it away from them there are like technicalities questions of private keys and can they kind of secure this private key but so before we before

we go before we go into into into the but but but just just to kind try to make the argument okay there are agents and they will quite likely be or can be considered a form of life and and then again the the argument further goes um blockchains are an necess um necessary substrate for them so so or kind of my my way of phrasing it would be to say if you just run on a regular computer you cannot be considered a form of life because you too much depend on this computer if someone shuts this computer up down you you you you're dead but but blockchain itself has so little agency

it doesn't make decisions it's it's kind of so neutral that it allows agency to develop on that substrate and of course um and of course well projects like near and and internet computer are also kind of pushing pushing the boundary of of uh how much computation uh and yeah you can actually do on blockchains I mean obviously on on on ethereum the amount of computation you can literally do on chain is is completely neglectable um but but yeah you're also making the point kind of that this computation will or I think you're using the term blockchain slightly um broader I

guess okay so we constructed the argument that there will be a new form of life um and um uh and another the question is will I mean or again I mean f first maybe we should start to uh ask the question are you following this argument that there will be new forms of life kind of uh enabled by uh by blockchain maybe IIA you start with it with that with you yeah I mean I think there's like kind of a spectrum that I I look at um and I think like there will be kind of full agency systems right which like yeah maybe

instead of saying I mean that's why I like web 3 instead of blockchain because yeah like it's kind of peer-to-peer economic system right and so you can imagine a peer-to-peer economic system that is that has like um an agency itself and actually uh last nearon introduced this idea of AI president which is indeed like a reinforcement learning agent that is modeling the world and making decisions and has you know pocket of money to pay people to actually do work uh to continue uh optimizing its value function and uh the idea there though you have some set of token holders which are kind of you know have veto power and a way to provide

input to this model pretty much both in in the form of changing value function as well as literally just hey do not do this or here's some train more training data to you fine-tune your core uh core decision- making model so I think but but maybe to immediately intercept there so just just for the sake of argument you would probably agree you can take away that veto power or that that token holdershome uh it doesn't have kind of the quote unquote token holders right then um now you have like somebody's operating the hardware somebody is

operating like the networking like all of the pieces and uh I mean we can keep going down as like okay well it it it went and bought all of the hardware right like like there there's a conjecture somewhere but like at least at the carent stage we have a legal entities or individual people who are part of the system and they are either way like stakeholders in the system even if like who's who's who's running the near blockchain the validators so exactly so so so my I guess the argument goes um that that connection between uh Hardware physical Hardware and and the um and the um computational task is pretty much

this or kind of or there there's an abstraction layer that uh where it doesn't matter where the physical Hardware is but you kind of get no it doesn't matter but they have technical governance over the system right to to extend they are they can't they are the overseen body of the of the system and and I mean they have stake as well so they have they like both economical and Technical they are the governance so my point is like at at this current stage the way this uh kind of let's just call them computation decentralized computational network not blockchain structure there is a uh human input right that is kind of governing the system one way or another now to be clear like I'm I'm I'm on board with you

know selective censorship and like a governance system that actually maintains such systems that has human input I think you can continue going down this route and assume okay well let's imagine have an economic model that's driven by AI that then goes and starts buying out the validators right and so like I mean there's an interesting progression there that we see uh but this goes into then what is the economic function of this and I think in general this whole con I mean this whole very interesting thought thought kind of process is like we do need to look at it from an economic perspective right like what is uh what is a value function economic function

that aligns everyone to work on something because we do have actually you know millions of people working together toward singular function but yeah I I would say before we go deeper into that direction I would bring bring you in and kind of again Ask the the general question more do you agree with the premise of that yeah a new form of life in a way uh is born and and that it uses web three in a broad sense as a substrate no um so I'll make some enemies here I'm quite conscious of that but um it seems that a big premise in the discussion that we have today is the is a sort of analogy to human cognition

that we're projecting onto artificial agents we're lucky enough to have 80 years of research on what's the difference between human agency artificial agency now we have a new type of artificial agency with non-deterministic outputs Etc but the scenarios that are described here um assume a lot about those types of Agents what they can do and their capabilities mainly the main one that I want to point out is planning capabilities everything that you guys described assumes a capacity to plan a capacity of having a world model that is approximate correct or representative of

what is happening materially and acting on it and following through and we know from research today uh that the planning capabilities even of our best models of the latest CH GPT model uh are Quasi inexistent uh I'll just name one uh one uh research that came out a few months ago um in artificial intelligence we've had maybe 60 years of planning tasks to see how much AI agent can actually follow through on an idea on a plan executing things leveraging other agents human

agents Etc and it turns out that even with basic uh models and uh and games like block Works where put the right blocks in the in the good order um the performance of the B GPT model is at 30% compared to 100% for humans doing the same tasks if in those and I'll stop with that if in that same planning task you obfuscate the references to what kind of blocks you need to move around or the names of the blocks the performance in planning capabilities of those models drops to

3% what does that tell us it tells us that those models are doing at best approximate retrieval with no planning capability and extrapolating that there'll be a de generation of material impact on the world is something that I have a hard time accepting although I accept just to finish on a good note I accept that uh those agents can have material impact on the world and blockchain technology and peer-to-peer networks like I PFS and this is something we we're doing a lot of work on enables those agents to be much more powerful and materially impact uh the world can I uh absolutely you can hear

me right yeah okay so I 100% agree with what you just said about the current world um they're terrible at I think Martin might disagree actually so we had this debate yesterday I meet a lot of people that think this is a common debate I have with people I'm like the thing can't plan it can't think strategically and people are like no it's amazing it's I think it's it's make belief um The Wizard of O I think it can plan somewhat very short memory and so on because it's kind of the average CHP at the at the when you ask it for strategic to plan it strategically it will give you the average answer of a Reddit user because that's the data it was trained on and AI is really good or llms are really good at finding hay in hastech but not needle in hastech so it

won't find the perfect stry or the good stry it will find the average strategy the person on Reddit which is its World model language model would give uh however um that's for right now and we are working everyone is working so hard on reinforcement learning and control theory and other approaches so I think we are going to that's the most lagered part of progress but we will uh get there and there's one more thing that you you kind of started mentioning uh which is we don't know if the thing is conscious if it wants we don't know how artificial intelligences will behave and maybe they won't behave like it we want to kill each other and we want to survive maybe it won't want to survive

so I'll just pin a couple ideas uh there's Darwin pressures that I think will operate to this system uh there's a great Ted lecture about te gen memes and thems that I I link on the Twitter and I think that a lot of these AI creatures are going to be subject to Darwin selection pressures that will apply to them and therefore I can I I model them in my head roughly like humans make decisions but maybe more rational but but so I think these will behave like humans and I think they will be as good at strategy as humans but are not yet thankfully otherwise we probably wouldn't be healed so I don't agree with the that they will behave like humans because I mean that is a huge conjecture G given they're completely different

structure right like the fact like human cannot just copy itself AI model can copy itself right so there's like a completely different kind of manifestations either way but I do agree with the like I mean Alpha go is a great example of indeed being able to plan and kind of approximate you know all the future combinations of states and like estimate the uh estimate what the options you can imagine you know having like lots of rollups of the world State and estimating like what are the better uh better better evaluation right like the llm is better estimator than predictor indeed and it cannot kind of roll out a lot of the like unlikely

States very well uh but if you have a like policy roll out right you can do that and then you can use the estimator llm to like okay which which of the states is better when you're actually picking and choosing right I'm not even saying like that we haven't trained them on this kind of tasks we have a few more topics so I I I want to move on but still I want to give one one comment on the on the planning capabilities because that is actually something we are uh exploring in our agent work right now so we used to have those um those agents those yeah prediction Market agents that were somewhat preprogrammed to say okay you first need to do you first need to gather information online and then you

need to kind of create a research report and then you kind of need to come up with a prediction and then you and and and so on but now we also did an alternative version where we didn't do any of that uh planning or any of that hardcoded so we just said here is a list of functions um that you can do without any and basically just describing the functions as uh in in in text what it what what this function is doing um what what the input and the output parameters are and and and that's it and then we said um make money or or or or go and do something interesting or or whatever and it turned out that in my view it did did reasonably well so it it kind of uh it

exactly said yeah I first need to do this and then I need to do that and so on um and if I combine that that essentially this um that this knowledge this planning knowledge that is um can be um um yeah that can be accumulated or or or that that an agent can then see whether that strategy worked and then kind of um uh um synthesize that strategy and kind of Spawn new agents already with that uh strategy um as a as a preconfiguration essentially yeah then then to me I I think there is a chance that this will evolve into into

something that is yeah self-evolving and increas or is essentially increasing its planning uh capabilities while it goes through its generations and of course it can do it um very fast I want to bring up um slide uh about um perspective on on risk so maybe so for the audience um I personally found it helpful to um uh to um to to think about uh to to try to think about it in a in a quite um uh structured way so um yeah I just just leave it a little bit for the uh for the

viewers I post it on my Twitter as well yeah so that that that is the way I filled it out uh personally this um um this chart but but yeah leaving it for the um for the audience so I think one one topic I want to go into is in your talk you mainly described this one large um entity or or um yeah chap Chapa yeah whatever uh yeah kind of this one large dominant um uh agent um but but I think there is yeah a good chance that there will be

many many and but but potentially in a different way and and potentially in such a way that they keep each other in Balance um and I think one one perspective is to look at companies so if you want you can look at companies today so whatever Google and apple and you could say those are already super intelligence so those are in a way those are agents I mean they do stuff company does stuff they are more capable than any uh than any um human Alone um uh and they um and they I mean the first kind of thing is to try to survive and kind of try to accumulate

resources but so far it worked out pretty decent I would say so companies overall produced uh um largely I would say good outcomes uh um for humans so may maybe how do you see that perspective yeah so I kind of address these points in the talk but not but just as throwaways so um it's not singular I think there will be many AI there's attacker Defender asymmetry which means that if one of them decides they want to kill all of us like they can if I if I try if I want to kill you no one's going to be able to stop me before the fact so anyone in the world that has it a singular goal to kill you is able to sorry um we live with this um so we can't stop each other uh

corporations are very inefficient uh they are constructed by many people that don't play along middle management is pervasive uh so corporations are much weaker than aund of people working with each other in concert as this is why startups succeed because they start from scratch as a small team that does stuff rather than a big Corporation does nothing um and you can see that the Manhattan Project didn't came up in the talk but the Manhattan Project is like maybe the most concentrated scientific Endeavor of people who are incentive aligned and manag to build by the way World destroying technology because they thought it's really important and then regretted it after and we got to the world that we live in where we almost killed ourselves thank God for aripov many many times over and so I think there will be many AIS and they will uh

not balance each other out because they each get too much power individually and they can't watch over each other is my prediction here maybe IIA so I think you agree that will be met or yeah yeah I think there's kind of few few aspects right one is I mean there's a like a AI is a self like the organism itself right and I like to to look at it from the individual like Community perspective and the fact that like as individuals we have a huge motivation to create a tool for ourselves right that and like a lot of the you know AI has been more about improving our own intelligence this like

actually back in like I don't know 13 I wrote this augmented intelligence versus versus artificial intelligence right because we actually want to make ourselves more powerful kind of more intelligent um be able to do more and so uh so I think like first kind of my point will be that and this is you know why we call it user owned AI for example or not what we trying to build is that we want individuals to be in control and in turn AI models and kind of AI capabilities to power and protect them and be aligned with their with your kind of internal interest and again remember this models at the end you are designing the value function right whoever kind of

builds it launches Etc and so yes there can be like models that will have a value function explicitly to you know uh that has negative effect but that's why we do need to have uh kind of in like individual Community level country level capabilities as well who are actually focused on their well being and are protecting and pushing back against it right in a way similar how you know your individual Freedom stops where my individual Freedom starts and same on community level and City level and Country level at every level so that's kind of the I would say like first you know fundamental piece the other fundamental piece is assuming let's say we do have an artificial a like agent that is running and is coordinating a

lot of people the its kind of value function because it's coordinating a lot of people has something to do with success of those people as well because otherwise these people will not be participating in this and so imagining that then this model decides to kill all those people seems very like it seems very non-economical right it's not not flowing one from another and so you can imagine having like different models trying to kind of attack each other for resources Etc which is you know our current world sadly uh where uh people do that but at least like from perspective of you know like kind of looking from economics perspective right and in general the systems are all about

economics um like what is the value function that kind of you know oh I'm going to plan you know destruction of Earth uh but you know I am right now coordinating a bunch of people uh to me that seems like a very uh kind of unclear like why why why would that be economical for this model and and Andreas how do you see it so uh do you think well I mean I think you you don't think there will be one large AI that is but is do you think there will be many AIS that keep it each other in power or or do you even say no they won't even be that we don't even reach that stage yeah so um definitely we won't be all watched over by Machines of love and grace I think we at most will have um each

individually our own AI agent very useful run them locally synthesize knowledge summarize knowledge uh uh use systems that are rule based that are too complicated for you to leverage but an AI with the proper API Integrations can do amazing but a lot of the next consequences or capabilities or scenarios that we're talking about assume so much more about the capabilities of the agents that we have today and really uh I would love to hear it from you guys is there anything from the results we have today in AI research that allows you to say that okay we're

going to get to that next stage where there are much more powerful and I just want to make a quick analogy uh which is from this professor at the University of uh Princeton I think Arvin something um who makes the analogy with uh the airline industry uh in the 1960s the the rate of improvement in the speed of airplanes exponential and so Pan-American started selling tickets to the Moon because they were like okay if it continues like that for the next 15 years we can get to the moon in a few minutes and come back and you can do your little trips over there 100,000 people bought tickets to the Moon by the

by the' 70s until it plateaued I see us arriving at this Plateau already with the architecture that my honorable colleague has has produced and with the mental model that Richard Sutton kind of mystified the whole generation on that put enough compute enough data something is going to happen uh everything will be comput computationally reducible to something that you can um kind of interact with and produce useful things about we Compu we realize that okay language is comput computationally reducible and we can do a lot of useful stuff from that derive stuff from from

that but all the other things we were mentioning are scale Beyond yeah can I so I think that's an extremely important perspective to to uh to just yeah say we might be wrong here on on on on on that side that AI will have that dramatic uh progress still want to for a minute entertain the thought that it will have this dramatic progress and kind of come back to the question of to what extent is it possible that different AI systems will keep each other in power in in in in Balance so you mentioned a few times this idea of uh attacker Defender asymmetry so the counterargument would say uh everyone has their perfect uh AI

agent they're personalized on their kind of maybe we kind of even merge in a way with with with with that uh um uh with our personal AI agent and we are essentially One agent as a um so yeah you said it uh well not creating one God creating many gods um yeah instead of I mean in general as as a Humanity right every individual instead of wanting for us to create one single God right which is kind of this idea of you know having one model that controls like coordinates and controls everyone the reality is we all want to become God right like we want all to augment our own intelligence or we want communities to have our like

um uh kind of our own like system that is you know propels our own uh needs right and we see this with blockchain right instead of having one blockchain we have lots of blockchains right and uh in a way they already kind of behave like this and trying to uh you know uh inter like interact with each other sometimes hostile sometimes coordinated and um sure sure can I suggest a thought experiment imagine there is a thousand scientists that work together that are super in agreement and you somehow they're forced to be super in agreement they decided they want to kill all humans um and Suppose there are many many other groups of a thousand scientist each working separately and they know that someone out there is

trying to kill all humans and they're trying to stop but they don't know how and they're trying to stop it and all of these people are working separately in secret and no one knows they exist but why are they working separately like if they know like hey we need to coordinate to stop it yeah yeah but I'm just oh um they like you know a million okay it's they all coordinators a people try to protect humanity and there's thousand very motivated scientist in secret to kill it who WIS okay but I think I think the the analogy would go or the story or kind of the counter story would go every person has a million people working for them to just protect that person and that has every every person has that so every person my point is like they will coordinate together when when the value

align which is like for example preserving Humanity now all of everybody's personal agent will coordinate together to protect them right even if even if a thrill in agents even if all humans together work to protect humanity and if a thousand well motivated people want to destroy Humanity I think they'll succeed Maybe I'm Wrong um that's my mental model of the world science scientists are amazing and attacker Defender symmetry is very strong again I think the the argument is that well that there is at least a pass that every person individually become so much stronger because uh well they now have this thous thousand or million scientists working for them so everyone has this everyone has this super intelligence that is working for them the Gap is like quadratic at least if

not exponential we can do the formulas um we can do the game theory of this um oh can I can I add a couple things okay sorry um just two things first of all I meet whenever you say computer scientists can't do something it's been wrong every single time like uh learned computer science I I grew up when neural networks were supposed to be not working and you shouldn't even try so so when people tell me we're going to build reinforcement learning and smart strategic computers I believe them and second I meet three founders a week building that say they're building smart strategic humans and they're very smart Founders um so okay I want to raise money as well I going to sell anything to anyone that's willing to hear but

scientifically we we've had for 2,500 years uh method induction abduction deduction that helps us make sense of the world reason better not fall into biases and here I hear a really aspirations or dreams that are not grounded in much indication like in 2016 you would say vinegrad you know schemas are not solvable right and we're like few years later and like I would I would say actually in 2017 I I thought we're an exponential ride and everything will be like that's why we started neari and yeah was wrong it took like five

like six years instead of you know one year but I mean I I agree that like at this point yes it's probably not going to be solved tomorrow like we're not going to have magic planning system tomorrow but like you can imagine that you know some number of years and some like given especially how much money is invested in this it's not impossible like it's clearly not impossible to do planning right we can do it we we we we are running out of time so I want to uh I want want to ask one last question and that is what event uh would need to happen or what what thing would you need to observe that you would dramatically change your view and and that would

probably be for you to to well be less concerned and for you to be concerned and for for you but let's start with solving climate change massive coding successful massive coordination we solved the ozone layer Hole uh we haven't decided we managed to save World from nuclear weapons partly by luck if we manage to actually coordinate as a species successfully I would change P Doom to be almost zero um we have not managed that and I think crypto does have a lot to contribute there as do other digital people sadly they're all working on better monetization and AD networks and scamming and gaming and whatever so what what would need to happen that you would start to say this

is huge risk we should we need worrying about AI yeah I worry about AI today I think that we're all fooling ourselves into not taking care of the material impact that AI is having on the environment on social inequalities structural inequalities there is so much to worry already about there are people that are not getting government welfare there are people that can't get access to uh Credit Insurance there are people that all that because of there are some recommended systems or or other AI systems that are affecting their life so I'm already worried I think we should all worry about the Practical things that are in front of us for the scenarios that are you're describing I

think I'll start worrying when there's a robot at home that guesses what I want to eat today and they come and serve it to me and I want to see a lot of coordination and follow through and planning in order to be able to say okay we have something that is artificial agent resembling much more human agency but it's a category it's a completely different category of uh complex problems compared to yeah cool thank you I think we have very little time left but oh I was just opening up for oh wait for questions I mean yeah I think for

for me the like on the risk side it's it's two two things one is I it's not about me changing opinion it's it's like we need to work on solving this like I think like the way I look at it is not like all those risks exist the centralized risk exist which I talked about the decentralized risk exists you know the inequality is completely exist like all of this exists and they all like you know some percentages and we really need to lean in on solving coordination problems empowering each individual making sure that people enable to have you know access to resources on equal part not being uh kind of challenge so I think like it's less about like how to change my opinion

it's more how do we actually coordinate to solve these problems and uh you know use this technology that we're building to do that thank you very much we yeah um we have I think we are out of time but let's do two questions yeah here all right so um my question is about the best argument first I'm pretty much with elot with all his concerns but

uh the best argument against that I've heard and I'd see you like to see you counter is whether um this has to be a step function and it's not something that is going to evolve evolve so gradually that the coordination of humanity against it will come with these things evolving slowly and we seeing them I hope I very much hope I'm I hope I'm not with me I I don't like people who agree with this is very scary um yeah it can evolve the problem is so the crypto problem is we're creating this substrate and it's once you create something it's very hard to put the genie back in the bottle and a sensorship resistant layer tends to stay moving and existing and so what we're creating now it doesn't matter how how

slow the AI evolves what we're creating now will probably persist uh through it and if we do privacy preserving computation then we won't know it's even there um but I hope you likeed all right I see uh yeah then here hello um on a serious note if you do have a very strong reservations against uh kind of the direction that AI is taking um how do you reconcile that with kind of uh working in the industry in a way that you're kind of accelerating that progress h i don't I uh I'm a computer

science I I I can't tell you that the AI wiping everyone out and then colonizing the universe the AI is us like they're people as well they're they're trained on R it I mean I don't know if they're great people but uh they're people as well if you watch battar Gala it's it's like this I I I want the AI to exist I I want us to buth a new species into life a species that's made of metal and silicon and can carry itself through the Stars much more efficiently than us and is built in our image we're playing God um how can you I have an opportunity to play a little bit part of God every day how can I not take it and that's what I dedicated my life to and also I'm kind of hope I'm an optimist I hope it will

be fine I hope we'll survive it and we and AI is going to be amazing when it's here I think there are lots of risks but I think as long as they're not existential we will succeed them and we will make an amazing world with AI together I so I want to be part of that and that's my life's Mission and that's all I know how to do um but also there's this guy that built both biological weapons and fertilizer Indian I forgot his name or German whatever UMO sorry Santo oh I don't remember but um it's the same guy and and it's the same technology I mean all of the strong Tech nuclear weapons

nuclear power um oil fueling gunpowder conrete automation so all of these things have to I I want to be part of progressing technology and I'm also hoping that we manage to coordinate to be on the good side of it all right well thank you again to all of you and U probably the discussion will continue thank you