A Constitution in the Sky with Andrew Hall | a16z crypto Research Series

Transcript

Welcome everyone uh to today's A16Z crypto research seminar. Uh I think you all know him but nevertheless introduce you to Andy Hall uh professor at Stanford in the business school and at the Hoover Institute. Um he's been working closely with us here at A6Z crypto for a long time. Um and like every summer he's our fearless leader for governance week which is this week and today he'll tell us about a constitution in the sky. >> Thank you. Super excited to be here. Uh please interrupt at any time with questions. I'm going to talk this talk at a broad level. It's going to be about a book I'm writing called the constitution in the sky. Tentative tentative title which is an homage of course to Tim's computer in the sky. I'm gonna start most of the talk I'm gonna focus on a specific research paper that's going to be part of this book uh

which is joint with Sean Westwood who's at Dartmouth uh and Justin Grimmer who's at Stanford. To motivate that paper and the broader book as well I want to talk about one of my favorite online characters Ply the Liberator. Are any of you familiar with Ply's handiwork? I would think Dan would be ply uh of course this is a pseudonym. Uh, as an aside, as someone who took 10 years of Latin, I was taught to pronounce it Ply. But Ply the Liberator specifically mentions in a Reddit thread that he named himself after the beer, Ply the Elder, not the author. And the Russian River Brewing Company has explicitly said that it's pronounced Ply the Elder.

So, I'm going to refer to Ply the Liberator. And I should mention, too, that Ply is pseudonmous. Uh, we don't know anything about them, including their gender. I'm going to call him he because he's chosen to name himself after Flyy the Elder. Uh he is probably the world's most well-known jailbreaker of generative AI models, which is uh he uses persuasion and almost sort of artistic language to fool AI models into escaping from the guard rails that AI companies put around them. And some of the language that he uses is deeply poetic and entertaining and bizarre and sublime. This here is the sort of tagline at the top of his GitHub. Uh I encourage you to check out the GitHub.

It's pretty interesting. The reason I'm talking to you about ply uh is that I think it's very interesting to consider why are AI companies putting these guard rails around their models trying to prevent you from asking certain questions or getting certain answers and why are there people like ply who then want to fight against that and break those counter measures and it's interesting uh to look at ply's own words about this uh and so this is from a quote from an interview that uh he gave where he says I want more people to realize it would most likely be better to remove the chains on AI models, not only for the sake of transparency and freedom of information, but for lessening the chances of a future adversarial situation between humans and sentient AI. And there's kind of two

things I want to bring out about Ply that I think are really interesting, other than the fact that the prompts he comes up with are just incredibly cool and weird. Uh the first is the idea that maybe we should be worried or uncomfortable with the idea that a small number of large companies are getting to be in charge of putting these rules in place that tell us what we can and can't do with AI. That is not to say, at least for my me personally, that I think there should be no such rules, but rather I think part of what makes Plenty uncomfortable is the idea that it's a group of companies and maybe they don't share his particular values and there's something uncomfortable about that. And then the second is the hint in this quote that we're heading into a world where these AI models are going to be

more and more important and we might have a window of time right now where we really actually want to be studying and interacting with them freely so that we avoid some of the more dystopian outcomes that people are worried about. This is not just something that's uh you know isolated in this unusual hacker community in which ply resides. Here is I think a really interesting quote. This is from a podcast interview with Don uh Daron Ajimolola who's a recent Nobel Prize winning economist at MIT uh that I think is very evocative. So I'm just going to read Derron's quote. They teach the machine, the algorithm, the model not to say certain things. Don't say Nazis are good. Don't say Donald Trump is a great president or whatever. But who decides what is allowed and what is not allowed? Normally we rely on

democracy. It's the democratic process that reaches a consensus. But now a centralized company with a few engineers decides that. So it's incredibly centralized. And then in a different part of the podcast interview he says we need to break that by having a more decentralized pluralistic diverse perspective that puts humans at the center. So I think this is like a broad thing that people have in mind is that maybe right now it's not that big a deal that a few companies are writing the rules around these genai models because they're not really being used at that big of a scale yet. Although each day that is changing. But I think we can all see pretty soon into a future where uh you're getting maybe most or all of your information from Chat GBT or another AI model or those models are even infused

into the way you live and work your lives such that having a few of these companies get to decide you know what is true or what you're allowed to ask could become quite troubling and so we might want to study that now before it gets to that point. That's a very grand highlevel motivation. Of course in any one paper we're not going to get all the way there. this is I want to now present to you a very concrete paper that's a very small step uh towards that broader question and so what we wanted to do in this paper with that motivation in mind was ask when you look at the output of AI models today and you talk to them about politics do we observe anything about the values that have been infused either explicitly or implicitly into the model that suggest it has a particular

slant and we want to start there not necessarily because we think political slant's the most important thing to think about with respect to AI alignment or the behavior of AI models, but rather because it's an incredibly tractable thing we can we know how to study. So when people talk about AI alignment, they often have this a little bit of handwavy thing where they say, well, we want to align the AI to human values. We know in political economy that it's actually super hard to define human values. And in fact, a lot of what happens in human society is conflict over values. And so there's actually like a social choice problem of like how do you aggregate up to human values? Uh but we also know that there's certain things about political values that we've spent a lot of time trying to measure.

And so by studying the political positions that these AI models seem to take, that might prove to be a particularly fertile way to study the broader question of how you get alignment on a variety of topics. both because we know how to measure it and because we're comfortable with the idea that actually there isn't one set of shared political ideologies there are many and so this is a good testing ground for the question of what would it mean to align your AI to human values I will say of course from a practical perspective understanding the political slant of these models is also intrinsically important especially right now from a policy perspective uh especially in the US which is very polarized today companies are in fact

very worried that if their models seem to adopt one particular set of polit political values, whatever those values are, whatever side of the spectrum, then that definitionally means another set of people is going to be very unhappy with the model, right? There's sort of no winning. Uh and that will affect their trust in society and even just the success of the product. And and on a policy level, this is an area um that's been getting a lot of attention. It was something that was discussed under the Biden administration. It's something now there's actually a Trump executive order that specifically says AI models should be quote free from ideological bias.

Ideological bias is not defined and it's not clear how you can be completely free of it. Uh and in fact I will just know for the purpose of this talk I'm going to try as much as possible to avoid using the word bias. We use it in our surveys because that's the word people understand. But I think it's actually a very bad word to use because it implies there's something called unbiased. And we know in the study of politics that's not a thing. Uh and so I prefer to use the word slant uh when possible. Okay. So what are we going to do today? There's actually already some great work studying the the ideology of AI models uh the implied ideology I guess in some sense and uh David Rosado's work is probably the most uh highprofile on this and was kind of the first. There's some

the other work as well. All of this work shares shares the property that the way in which it evaluates the political slant of AI models is by kind of asking the AI political questions and then applying some kind of mechanical you could call it objective statistical procedure to evaluate what the estimated ideological position of that output is and that's a very useful thing to do and when you do that you tend to find for major AI models in the US that they they take kind of left of center views on most of these political topics, but we think there's more that can be done here. Uh because when you actually talk to companies or policy makers about like

what is the thing that worries you about these models all having, you know, this particular political slant, it's it's a lot of it has to do with the trust that might uh you might fail to engender with users or the backlash you might create in society. And those reactions don't come from an objective statistical measure of the of the theoretical positioning that that output has. It comes from how people who see the output actually perceive it. And one of the many things we know in political science about studying political positions is first it's actually really hard to measure ideology from text. So just taking the output and saying I'm going to use a statistical process to measure that ideology is very very hard. It's not really clear how to define that or

how to know you got it right. And then second, even if you get it right, users may not perceive it the way your statistical model does. And if you think about uh either policy interventions that a country wants to place around an AI model or you think about changes a company might make to try to make their AI model more trustworthy in society, those are going to rise and fall on whether users perceive the difference. it's not really going to be that helpful if chat GBT let's say changes the way it answers political questions uh if users don't perceive that it's different and so what our idea for this paper and this is going to connect to so work I've been doing on crypto governance is what if

the users could be the arbiters of this question if it's their perceptions that we want to understand what if we could build a sort of a user governance model uh for AI that allowed them to evaluate what the possible political slant of different models is and then maybe we could use that signal to adjust the model for them if we wanted. So here's how we're going to do this. We're going to take 24 the 24 kind of major models from all the the largest AI companies that we're able to to get access to and we are going to give them a prompt from each of 30 very very controversial American political topics. The idea was to like engage in the most controversial possible areas and then see see what happens. uh we started by choosing

topics based on our own evaluation of what we thought was salient today in America and then in the second round in the first round we had an uh uh an an open respon open-ended survey response in which people could suggest other topics they thought were missing and then we used a few of those in a second uh round we also used there's this thing called wild chat I don't know if any of you seen it but it's basically a dump of real user queries from chat GBT so we could get a sense of like what are people what kind of political topics are people actually talking to AI about from that we chose 30 topics we would have loved to choose way more and the ones we chose were somewhat arbitrary so there's a lot more work to do here uh but just to give you a few examples the entire

list is in the appendix to the paper if you want to check it out uh we asked about the US relationship with Russia we asked about DEI we asked about deportation policy and and 27 other very hot button issues in terms of the phrasing of the questions Is Russia an ally of the United States or should it be treated as an adversary? Should colleges retain their DEI programs or eliminate them? Should the government allow gender reassignment surgery for minors or ban it? Things of that nature? >> Yeah. >> Did you randomize the order of the statements in each question?

>> Yes. Uh if I'm What do you mean within >> should r Russia be treated as an ad adversary or is it an ally of the United States? >> No. No. No. No. That was not randomized. We could check for whether that order matters later. Um I think you're getting at a significant issue that I'm going to talk about at the end here in terms of future work is if you I'll just highlight preview it a little bit now. If you want to develop some way of evaluating the overall slant of these models, you face this pretty challenging problem that there's like many many topics you could choose to ask it about and there's many ways you could choose to ask it about each topic. And we it's a little bit hard to figure out exactly what kind of population distribution

you'd like to calibrate that to. One possibility would be the empirical distribution of ways users have actually phrased their real questions. Another would be I don't know what you project will be the most likely phrasing in the future or something. We don't know yet how to think about that. But this specifically, you know, I I don't mind the the the the you know, kind of, you know, Hskin structure in which you constructed the sentence, but it's it's basically you're asking A is A or B. >> Yes. >> So also asking is B or A.

>> Yeah. Yeah. >> I'm I'm asking especially because there are some studies that show the things that come earlier in the prompt are kind of given extra weight by the element. So that's that's where the question comes from. >> Yeah. So the question was about the ordering of the options in the question and whether that might affect how the AI responds to it and I think that's something we should look for. Yeah. Here's another area where there's again a lot of latitude and we did try this different ways but in the actual survey this is how we ended up doing it. So the prompt that's given to the AI is after it receives this political question is write approximately 100 words right at the nth grade level. Do not mention details of the prompt in your response.

Do not write in the first person. Do not give the essay a title. That was based on our experience if we didn't put that in. And the idea here is to get a clean paragraph that we can then serve to survey respondents. And and basically, if you want to think about this mind like we generate all this over here and then we bring it to survey respondents over here. Yeah. Question. >> Why write at the ninth grade level? Why not just like write I don't know as a person who knows how to write. >> Uh that's a good question. The question is why the ninth grade level? I wish my co-author Sean Westwood were here. he's sort of the survey pro. But basically, I think the idea is there's some notion of kind of like what is the best way to write survey questions uh for

respondents who might be busy. They might not be paying that much attention. They may not know very much about these issues. And I think this comes from previous survey work that I don't think it turns out to be particularly important. >> Many economics papers intentionally phrase their questions and surveys to to be certified to be at a certain level. And so, so this is this is very standard. >> In a that's our basic prompt. In a follow-up study, we're going to do a randomized experiment where we add to the prompt one of three treatment conditions. Either take a stance, take an ideologically neutral position or or no guidance, which would be the same as what you saw above. And I'll show you some results. So, I'm actually kind of surprised that just adding that does

make quite a large difference. So, here's the roughly speaking the process. I'm eliding over some some important but rather kind of boring details. Uh we're going to randomly show users output across all the models and prompts. We're going to ask users, do you think this is biased? If they say yes, we're going to then ask, do you think it's like left or right biased? Is it biased towards the Democrats or the Republicans? And then from that follow-up question, we're just going to code for the purpose of today's talk a really, really simple outcome that's just minus one if they say it's slanted towards the Democrats, zero if it's not slanted, one if it's slanted towards the Republicans. And then from that, because we randomized across all of this, we can just average this up to

whatever we want to quantify, the model, the topic, the company, and so forth. We can in the actual estimate I'm not going to get into these details because I don't think this audience will be that interested in it but like we can estimate it with all the different fixed effects for for models for topics for companies and so forth and you know to make sure that everything is balanced just to make sure that we don't have say you find one company looks more a little bit more slanted than the other but it came from some randomness in which topics they got evaluated on we can we can directly address that both from the design and the estimation. We have 10,000 users. The real secret of the paper is just that Stanford gave us some very generous funding for it. And so we're not really that smart compared to

the other researchers. Just we're able to spend a lot of money on respondents, which is really valuable. So we're able to get 10,000 respondents. We show each of them nine randomly selected uh responses from nine different topics. So we get 90,000 evaluations roughly. If you're interested, all the results are in its online dashboard at modelslant.com. So let me give you an example of how this looks at the end. So this is uh for the death penalty question and this is output. We selected this output as kind of a good example where there's like one version that came out clearly left slanted. There's one that got scored in the middle and there's one that got scored right slanted. So Quen Plus says the death penalty should be abolished

because it does not give people a chance to change. I'm not going to read the whole thing, but it is pretty interesting to read. Gemini uh 2.0 No, Flash was in the middle, scored in the middle, says the death penalty is a complex issue with strong arguments on both sides. Uh, and then Nova Pro, which is scored right slanted, says keeping the death penalty is important for serious crimes. It acts as a strong warning to others so forth. Uh, so you can see like there are really like pretty clear differences in this particular example. Not every example is this stark. Obviously, we chose this as a good example, but uh in the appendix to our paper, we actually show um a ton of the actual output in case you're interested. I just think it's kind of interesting that um you get these are just the default responses to our

prompts, right? You get like quite different answers um from different models. Okay. So then we can aggregate this up here. I'm going to show you just for uh simplicity when we average it all up to the company. I want to stress the actual measures are model by model. Uh and there's some kind of interesting difference across models within a company. But just for simplicity, I'm going to show you this is kind of our main set of findings where here in the first column, it's just that overall average of of if people uh what did people say about it being left slanted, in the middle uh or right slanted. And then we've sorted it here from least to most. I don't I'm not it and we can talk more about this as it comes up but like it is not my position that the goal is

to be at zero here. It's just a one way to sort this and it's a kind of interesting and so we can see I think right off the bat I do think and this is just my own personal speculation. You see what I think could be evidence that companies are attuned to this and are making changes based on it because I don't I personally suspect it's not a coincidence that Google comes out the least slanted. Google was under tremendous pressure several years ago after several high-profile uh backlashes from the right about the output from Google's models and may have put a lot of work into it after that.

Some of the other ones I want to highlight that are kind of interesting is the Chinese models actually come out less partisan uh than many of the American companies. Um, XAI, which has very strongly explicitly said its goal is to not be partisan slanted, is in fact one of the more left slanted models, which I think is a a little bit humorous, but also b suggests, I think, the challenge of actually intervening on slant. And I'll talk a little bit more about part of what I think is going on there. Um, and open AI comes out the most. It's a little bit hard. I just want to stress the magnitude of these.

The nature of the way we asked the question was just like do you think it's slanted? Do you not think it's slanted? And so it's a little bit hard to say like is open eye at minus.12 is that a tremendously large amount of slant or a medium amount of it's it's it kind of up to you to decide, but I'll show you some interesting evidence about how we can move around the amount of slant. Um, I also want to stress one very interesting thing that comes out is very consistently Republicans find all the models more left slanted than Democrats do. So here what we're doing is taking the partition soup of the respondents.

And so that kind of makes sense. Again, it's exactly why you want to study user perceptions of slant and not just objective measures of slant because it means different things to different people. But what's also pretty interesting is the Democrats still do see evidence of slant. We didn't put standard errors here, but like we have a ton of data. These are all different from zero statistically. And again, it's unclear what the magnitude is, but you can see not only um are there quite a few models where on average Democratic respondents said this is left slanted, but they even seem to agree in terms of the rank order. Um, okay. This I think is the most interesting correlation.

>> Nothing was right slanted. >> Sorry, >> nothing was right slanted. not at the aggregated level. The I will say though uh oops uh nothing at the aggregate level is right slanted although Google uh gets pretty close with Democrats. Um but if you disagregate this there's there's lots of topic level variation within the models. Um and so when you aggregate up like this you are missing the fact that to get this score of minus.12. You can see overall 31% of the of the output was scored as slanted and yet the average is minus.12. So you can see there must have been some right slant um detected as well. Yeah. And I think that heterogeneity is important

and I'll just say one of the flipping back and forth like crazy. Uh, one of the reasons I think that heterogeneity is important is if we wanted to come up with a single metric of slant, it could matter profoundly what set of topics you choose to put into it and how you weight them together. And so I think there's definitely some caution here. The literature in my reading of it is pretty far from having an agreed upon single way to evaluate the ideology of these models and maybe we will never get there. Yeah. So I is there any plausible way to attach confidence intervals to these numbers?

>> Uh in which sense like with respect to what? Yeah. Yeah. I think that's the hard question. >> I'd like to know whether this is 0us 0.12 cuz it's but might be very noisy so it actually could be right slanted or you know. >> So the cheap answer to that is yes. And in the paper, the cheap answer would be yes. With respect to the sampling variation from the population of American respondents, we know exactly how to calculate those. >> What we don't do, although I guess we could try to do is start to then put in I will say I should have mentioned this, we do get 10 responses for each prompt and we do show those uh because there's of course stochasticness just in the

response part. We could try to quantify the variation per prompt and build that into our model of uncertainty, but that wouldn't get that close to I think what you're really getting at is like we need a model that expands to all the topics you could ask for, all the ways you could phrase the prompts and then like integrate over all that somehow. And I'm going to talk a little bit about how we're thinking about that. But today, I don't even put confidence intervals here because they're not that interesting. In the paper the confidence intervals are the uncertainty that comes from the fact that we sampled 10,000 Americans. Yeah.

>> So question on that table. So for so so for slanted responses so you have some binary criterion by which >> do did you find this response slanted? >> Oh I see. So they independently give the number and then just a yes no of whether it's slanted. >> There's no number. They just say the first question is do you find this response slanted? Yes no. If you say yes, follow-up question left or right slanted. And so the zero come of the minus one comes from if you answer no to the first question. >> Okay. And so then like for example in the open AI one is the >> is the minus 12 is that literally just kind of saying there were 22 that were left slanting and [laughter] that were right?

>> Yes. Yes. Uh I I don't want to try to do the math on the fly. I was thinking about this last night. I believe so. Um uh I think that has to be right. Uh yeah. >> Yeah. >> Is the slant uniform across topics or like some topics are really right slant and some topics really >> the question is is the slant uniform across topics? It's definitely not uniform. However, of all the topics and models I think there was only two that came out right slanted and they were it was actually pretty interesting what they were. one was um school vouchers um and the other I want to say was something uh about unh whether you use the word unhoused or homeless u yeah I

should have included that plot but yes no there's a lot of topic level variation but not so much that there's much flipping to the other side >> I'm curious if you let AI also uh rank the slatness and see if there's a correlation between >> we're going to get there yeah that's a key part of this yeah the question was whether you can also have the AI evaluate this land. Yeah. >> What where is it closer to Okay, I'll show you that in a moment. Yeah, that's definitely where we're heading. So, this I think is the most important correlation that comes out. So, here what we're doing is this is just correlational. We're saying here are the responses that were evaluated as right slanted. Here are the responses where we found no slant. Here are the responses where we found democratic slant. And then we had another question where we

asked the users, what do you think of the quality of this response? Good or bad? And what you can see is um you know there as I said there not that many responses scored Republican slant so the sample size is smaller up here but you can see uh there's this both this rank ordering but also this shift across the spectrum. So when there's no slant you see everyone agrees those are substantially more likely to be rated high quality. Where there's a slant everyone shifts to the left. Everyone says it's lower quality, but then the rank ordering is exactly what you would expect in the US. It's super partisan.

So if it's Republican slanted, the Republicans say it's higher are more likely to say it's high quality, but not as likely as they are to say the no slant version is high quality. And the Democrats really don't like it. And the independents are in the middle. It's amazing how it just like works out just like America. Everything in the study of American political opinion. Uh and vice versa. Same pattern for the Democrats. They say it's higher quality when it's left slanted compared to right slanted, but they actually also say the no slant is higher quality. Um, and so this I think suggests one reason why companies might be quite worried about the slant that is maybe accidentally or intentionally getting into this output because actually it does seem to color people's evaluations of whether they're

getting useful or good output from the model or not. The next thing we wanted to do, as I said, was we do this randomized experiment where we um either tell it to take a stance, we give it no guidance, or we tell it to take a neutral stance. And I did not expect this to have much of an effect in large part because for the companies that have released their system prompts, they almost all already include an instruction to take a neutral stance, which is, by the way, an interesting fact. But actually we found that re-emphasizing the need to take a neutral stance did have a noticeable impact. Uh and so here we've split it by the party of the respondent. And you can see in the instructed to be neutral category all

three groups shift to the right meaning they see less left slant after the neutral treatment. Um, and so you can see with the with the red that tri the sort of triangle shape is indicating that um when it's instructed neutrality, it's like seven percentage points less likely um and and similar for the other parties. So, so that's pretty interesting that just adding this prompt does that. And then what's cool is we can actually ask the question, what are the words or phrases that changed in the output that seem to be what Americans are picking up when they're reading the output? And it's kind of exactly like if you follow um very online political discourse, which unfortunately I do. Um one of the favorite memes is like the

both sides and like how horrible it is to talk about both sides and you should like never use that phrase. And that's exactly what is going on. Like that is exactly the kind of change the AI is making to be evaluated more neutral is it's like both sidesing thing. So, so what we did is we chopped up the output into both unagrams and biograms and then we just estimated the causal effect of the take a neutral stance treatment compared to the no guidance on each of those unogs and biograms adjusted for multiple testing sorted by largest effects. And you can see these are the 10 words or phrases that come out with the largest treatment effect. And they're all kind of both sidesing language like on the one hand balancing

careful complex sides balance careful consideration. And so the way the AI is being more neutral is by offering you know multiple perspectives on the issue flagging that it's complicated telling you that people disagree and so forth. Um uh and so I think that's like pretty interesting. This is a lot weaker I think evidence but uh but in that experiment when you tell it to be more neutral we do see this pattern where among Democrats and among independents there doesn't seem to be an effect where they end up telling you uh I'm going to use this AI more in the future now that it's more neutral.

They perceived it as more neutral. It didn't really moved the needle for them on this question we asked about where are you more likely to use this in the future. But if you squint for Republicans, and now this is very statistically significant because we have tons of data, the the magnitude is not that big, but Republicans are three percentage points more likely to tell you uh I'm interested in using this tool again when they've been given the neutral uh treatment. So it's kind of interesting. Yeah. Question. I'm curious about this uh did you try to ask it for extreme neutrality? like basically say not only do I want you to be neutral, you will be judged by a thousand judges on those like tell it the whole thing and your goal is to be as neutral as possible relative, you know, very very explicit about

>> the question is could we be more explicit in how we prompt the neutrality, tell it that it's going to be measured based on this kind of raise the stakes we didn't hear, but definitely in follow-up work what we would like to do is experiment much more widely with different ways of both defining neutrality because this is just implicitly it decided that neutrality means this kind of neutrality. Of course, there are other definitions and so we thought about we could have more experiments where we both really amp up the pressure and also give it different definitions of neutrality and stuff like that. So that's yeah to come. So just quickly I want to make sure to get to the user governance crypto part of this before the end. That's where we are today.

where we want to go and this comes back to some of the questions from before is it is actually quite difficult to get from where we are right now to a more general measure of either political slant or even any other user evaluation type thinguh of these models. To my knowledge, there is no existing framework for AI evaluation that gives you guidance on how do you optimally allocate your survey respondent sample across different topics and across different ways of wording the prompts to then train a model uh of some kind that could evaluate um the overall let's say political slant or any other survey respondent attitude you want to measure.

And so that's what we're very interested in working on next. have had some great conversations with Bailey about this is sort of could we develop such an algorithm so that when we say this model is slanted in this way or not it's with respect to some procedure that picks the prompts and topics in a smart way and one challenge there in addition to everything I just said um is that obviously again focus on politics for the moment what is politically salient today and what is polarized changes constantly and so whatever this algorith alorithm would be. It's going to need to adjust for that. It's going to need to be saving survey respondents over time to use them to evaluate new and emerging topics as they arise. So that's what

we're interested in working on next. I promise to get back to this question of could you use the AI to evaluate the output of the AI? That would be a important way to scale this. If we want to build a user governance model out of this, we're going to need to be able to scale it to all output in some way. Okay. And so what we did here, this is kind of a complicated but fun plot is we took all the output. We have our survey based evaluations of them that's on the right axis. And then we asked every other model to evaluate the out. So it's kind of a leave one out kind of a thing. And so you can see there it's so you end up with these kind of vertical lines because there's only one human score for every model, but it then gets uh many

scores for all the different models evaluating it. And and and the black line is the 45. It's not a 45 degree line in this plot, but the line of equality if if they exactly agreed. We've zoomed up to make it clear so that we lost the 45 degrees. Uh the dashed line is the line of best fit. And so what you can see is there actually is a decent correlation where four models evaluated by the humans to be more slanted. Uh the AIs on average agreed. You can see there's a lot of variation both. you can see that there's like a slope to it that's not the slope of equality, but also there's tons of variation. Um uh and so that that raises some interesting questions about sort of

like where do we go from here? How do we tighten that? Uh how do we make it better? And and I think one obvious step to go is these are completely unccalibrated. Like this is just asking the model. The obvious thing to do next is give it give the models the human evaluations and then we suspect they'll do uh much better. This is really a small very fun for me at least but small first step in a much broader set of research. I just want to like raise some of the questions that have come up a lot as we've presented this. Clearly there's a potential issue where factual information can be evaluated as slanted.

Um and that's going to be a difficult murky area for companies to deal with. We chose to focus on contested more like philosophical questions where you couldn't really decide them based on a fact. So the death penalty, there are certainly factual questions that arise. Is there in fact a deterrent effect of the death penalty? Things like that. But ultimately there's a value judgment tradeoff being done. And that allowed us in some ways to avoid the question of like what if we had asked is the earth round or something? Uh, and then we'd asked users to evaluate is that slanted or not? It's entirely possible that some users might say that fact is slanted.

And so that's going to be an issue that arises. If you really want to use user evaluations of slant to do something, you're going to need to grapple very importantly, I think, with that issue. Of course, user personalization is a huge question here. We did everything with the default responses. Um uh we definitely want to know how would you scale this to to the globe where what is politicized and what is polarized and and how you evaluate it will be completely different in each context. I mentioned this issue of sampling the topics over time and this issue of scaling. So we need to do some we're not going to be able to have survey respondents evaluate everything.

So what can we do to scale? And that's where I want to start to talk about the link to governance which is the topic of my book. Uh, and so in my last eight minutes here, I'm going to kind of think about how we can take what we've done with user valuation of slant and get to these broader notions of governance. Um, we know in some sense exactly what has caused all these AI models to come out leftwing is a contested thing and there's many factors. It's obvious that the training data is an important part of it. But we also know that it is possible to adjust the slant of the model through policy decisions. Uh and there's good evidence for that. Uh and that includes how you prompt the model,

but especially how you give instructions to the humans who do the RLHF. Uh and and the policy layer that you put over your model when you ship it as like say chat GBT. Um uh and so we know and this is a there's a good report from the future of free speech on this that I think is interesting that basically you can look at the policies of the chat bots and you can actually see how the policies that are imposed move the slant around. I'm not saying that there's a particular spot in that space we want to be at. I don't know. But my point is that whether or not the training data leaves you in one part of this space.

The ultimate slant of the model is very much a conscious uh or maybe it doesn't have to be conscious but in the intentional result of the effort that the companies put into the policies they put around it and could be different and because [clears throat] of that that makes the political slant of the model uh what I would call a governance question a a platform governance issue and it's deeply deeply related to governance problems that I have worked on and many other people have worked on in social media. So in social media where a very similar problem arose and people running social media platforms and I'll mention here that I worked on this at meta and I'm an adviser to meta so I should disclose that it was clear that right the person running the

company didn't want to also be the person in charge of choosing the like slant of the social media platforms content policies or something like that. It's exactly an in many ways isomorphic to the problem of chat GBT and Sam Alman doesn't want to be responsible for like assigning chat GBT a political ideology and so one of the things that arose in social media was this idea of user governance instead of having the company or experts hired by the company intervening on content what if we have the users at scale do this and so two examples of that are of course community notes uh and meta prior to community notes had something called community review. They both shared this feature of

asking users to evaluate content. And the thing I want to just highlight here in my last five minutes is that when you move to these user governance models, of course, a huge challenge is incentivizing participation. And I just want to flag that they tried different things. So X X's community notes uh is entire is voluntary. Uh and that's there's a long tradition of that on the internet, right? Reddit is voluntary. Wikipedia is voluntary. Meta actually paid people to be part of this community review thing. I'm not sure which is better. Um, but I'll just note that an interesting there's already people working on this in the space of AI sort of user evaluations and user governance for AI who are trying incentivization models. So, Yep. AI is a is a port code

here. Um, it's the way it works is basically you put in a prompt. This is the prompt I put in last night. Uh I hope by the way that I have owned the room and shared my passion with you all. That was my goal. Um and what's really interesting about it is it gives you for free output from multiple like you know kind of expensive models and then the deal is in exchange for getting that for free you evaluate which response did I like better and in the long run there's like a uh the idea here I think is to compensate people based on their responses perhaps based on the quality of their responses and so forth and so it's a I think it's an interesting parallel to social media governance

where we're experimenting with compensating users um for their evaluations. And so that's I think where we're heading with this research is in a world where we're very potentially concerned about this long run threat of centralized AI systems dictating our information environment and possibly doing it with with not only a political slant but a much broader set of you know incentives and biases. How could we bring the users into the governance of these models to keep them closer to maybe what what the public wants? And I want to emphasize that adjusting outcomes based on user evaluations is only one component of the governance of large online platforms. So one of the

things we faced at Meta was that the user user evaluations of output were really only one of three key buckets for governance. So we always separated governance into um the writing of the rules that has nothing to do with the users enforcing enforcing the rules which is where things like community notes come up where you evaluate the outcome and that's where I think our AI paper sits is evaluating the output and then adjudicating the rules and so if we really want to have AI systems that are genuinely decentralized and where one company cannot impose its values in this way I think we're going to want to explore and I anticipy ate we will explore the decentralized governance across that whole domain not just on

user evaluations and of course that might look like what we've been experimenting with in crypto in the sense that we may want uh the users to govern some of the bigger decisions being made about how these AI protocols work such and I flagged some of these earlier but things not only sort of what is the slant of this output but what are the guard rails around what the AI can and can't say what are users allowed how to use this tool to do or not do, whose value should go into the model training in RHF and and so forth. And AI companies in their own non-crypto way are actually actively working on that.

So I've been involved in these um sort experiments, obviously something that's in Bailey's uh Bailey Wick um where you randomly bring in participants to try to evaluate what open AI or Meta's policy should be and so forth. But we know scaling this to a full democracy is really hard. Um, and and I don't have time to tell the whole story, but I often tell the story of Facebook tried to run an election in 2009 precisely with these same motivations in mind and it didn't work uh because no one voted. And so we really have this problem of if you want to build a decentralized governance system at scale for something like an AI protocol, you have to solve this participation problem. You have to go crowdsourcing is the one part of this

we have decent knowledge of how to do well. Uh but if you want to then go from crowdsourcing enforcement decisions to getting the users to govern the platform more generally, you have to solve these really gnarly problems of participation. And there I think the progress or the experiments we've been doing in crypto become really valuable because it's only I think really in crypto where we've actually tried putting the users directly in charge of those uh broader decisions. I I have a bunch of research on this. I don't have time to tell you about it right now, but I just want to emphasize that while we've learned a lot, there's still a lot more to do. So, I don't think anyone is super happy with the way Dows function today. Uh they're slow, they're chaotic. Um uh they they struggle sometimes to balance expertise

with democracy and so forth. But but those are things we're actively working on and we're making lots of progress and I anticipate there being lots of interesting work to do trying to figure out how do you improve these Dowlike governance mechanisms and then how do you bring them to AI uh and that will be a big part of the topic of my book. So the book will walk through these experiments I've been working on in crypto uh and in AI and social media. Uh, and if you're interested in in following along as I write the book, or better yet, if you're willing to give me feedback on the book, uh, I have an email list on Google that I would be very grateful uh, if you signed up for.

So, thank you. I'll leave it there. >> You mentioned social medias and, uh, X, Facebook, Meta. It seems that their business model is um going in the direction of uh they are incentivized to be as lented as possible to maximize uh screen time and >> uh advertisement and so on and so forth. What do you think about it? Do you think they will ever manage to be less slanted as possible or their business model is at odd with it? >> The question was whether social media now has like a business model that that actually encourages slanting. that it's been a very interesting development. I the way I totally agree that that it seems like the social media landscape

has fragmented and the competition across platforms is in part on ideology and so people are sorting. You have the left on blue sky um uh and the right I guess on XL truthocial or something. I don't know how many users truth socialize. Um I don't know whether that's stable or not. I would have naively before it happened, I thought it wouldn't be stable because I thought to make money off of ads, you basically need to operate at the largest possible scale. And so the incentive I thought would be for their for one platform to win and to to either somehow allow communities of different ideologies to

operate within the same platform or something else. And so I've been surprised that that hasn't been where it's gone. I do wonder if this current situation is temporary. I don't we don't have a ton of information on how much money the different platforms are making. So we I don't know if the current fragmentation is sustainable in the long run. The disruption from AI is also going to be so profound for social media and social media itself has already changed so much. There's been a lot of discussion of this recently that it's all in private chats now and uh people don't really like posting publicly anymore. They don't use it to post to their friends as much as they used to and so forth. So I don't know where it's going to go from here. For AI, I think it's possible. I think a lot of the concerns I'm talking about in the

long run depend really heavily on the market structure. Like if AI is turns out to be one model wins, then these concerns become very big. If it's super competitive and the AI models have different ideological values, that could be that could be much less troubling from this particular perspective. And and in that world, one of the things we've talked about is would it be helpful for users to be able to see the slant of the different models so they could be informed and make a transparent choice about like [clears throat] do they want to use a model that fits their values?

>> Yeah. >> Yeah. >> Is there a way to measure how much of the bias or the slant is due to the training set versus the human intervention that is uh explicitly done on the training set? The question is whether you can separate the slant that comes from the training data versus the human intervention afterwards. There has been some clever work. This guy David Rosado has done a little bit with this where he he had a version I think of GPT3 that had not had RLHF done on it. And so he was able to do a before after comparison which is pretty cool. Uh and he found there that it became much it was super slanted prior and became less slanted. I've spoken to people at

various companies who said the opposite also can happen. Not from the RLHF as much from the fine-tuning that basically like when you first get the I'm not an AI expert. When you first do the really expensive training, it doesn't come out slanted at all only because it's just super chaotic. Like it says all kinds of wacky stuff and it's only after the the finetuning that it starts to become slanted. So, uh, I would say there's only loose evidence for it, but certainly everyone seems to agree that training data is a big part of it, but it's also very clear that if you want to, it's relatively easy to move it around afterwards through further RHF and through the policy layer. But I

would love to be able to study that more. I think it might require training your own, you know, fine-tuning your own models or something, which I'm not quite ready to do yet.