Mechanisms to Infer the Wisdom of the Crowd with Mallesh M. Pai | a16z crypto research talks

Transcript

welcome everyone to the last a16 z crypto research seminar of the week last but not least we have militia high uh malaysia is an economist at bryce university um he'll tell us about the wisdom of the crowds and hiring over the lease and maybe a little bit about decentralized oracles thank you thank you very much for having me uh and uh hopefully uh those of you online can hear me so uh i wanna start with a disclaimer which is the talk i'm giving is not directly related i i don't have in mind a direct set of applications for the kind of things i'm proposing um why am i presenting this well this has sort of been a hobby horse broadly

construed of mine for the last few years as a game theorist or economist understanding the economics of expertise so people have uh there are these people they call themselves experts they claim to know something more about uh the underlying unknown state of the world than the regular population and for various reasons you might be interested in evaluating their expertise aggregating their expertise today i'm going to drill down into a very specific setting but uh that that's sort of one set of questions i've been thinking a lot about over the past few years in a web 3 setting there are several settings i'm told uh reliably by people in this room over the course of this week where you might want to aggregate diffuse information so one that came up

was aggregating the opinions of backers in a defy lending setting or i'm going to carry catcher one that i thought i understood a little bit better before coming here which is oracles and smart contracts so let me tell you my caricature of it and you can tell me i'm all wrong so a smart contract as far if as an economist if i was modeling it all i'd say is it's a something that's trying to implement different outcomes based on an unknown input an input that isn't known at the time of writing down the smart contract something else is potentially going to come in and is going to be a relevant variable this input is not potentially something that's even on chain so you need these people oracles these uh other gadgets that are

going to provide the input so that the smart contract can function and if you're going to be relying on things that are off chain you want to understand the incentives of these articles what can be manipulated what can be aggregated what can you do with the information from these articles and you want to understand that so you can expand the set of contracts you write in the first place sometimes we're talking about oracle's that are reporting some state of the world that in principle is commonly known or understood off chain so the outcome of a sports event is something that everyone knows it's just that the chain doesn't know it that's not the kind of setting that uh this talk will immediately uh correspond to but in other cases even the oracles might have only noisy on

information about the state of the world that you're trying to condition on so you're trying to write a contract for example on the price of something and no particular oracle knows the price but they all know noisy versions of those prices they're sampling different amms they're sampling different exchanges what can we do how can we get uh sort of this disparate information and aggregate it in a way that's um so that's my high level pitch for why you should pay attention if you don't care about the underlying techniques okay all right and now uh all right so i'm just going to launch into a formal model this is not a high-level

overview talk uh it's it's a very specific model so the idea is that there is an unknown state of the world nobody knows it it's not something that will be revealed at any stage but there are agents experts oracles if you must there's a lot of them so we're going to rely on large number theorems who observe some information about that state they observe that information and what they observe or not what they observe but kind of the distribution of information how it's distributed among these agents is known among the agents themselves but not known to the aggregators so the question is how well

or can i identify this unknown state of the world when there are a bunch of agents who know something they know what each other they have information about what each other know but i want to design a prior free um i want to design a mechanism that does not condition on the settings of what they know uh they just uh i i want to do as well or can i do as well as a bayesian aggregator who actually knew the underlying prior who knew the underlying information structure and how well can i do relative to that okay so here's the motivating example for why the obvious thing you'd think like take the majority is uh not good so this is sort of

real this is due to a paper by pralik seong and mccoy and the question they asked people i'll keep returning to this paper but just the question they ask people is is chicago the capital of the state of illinois um yes or no and uh in case you didn't know it is not the capital of illinois um so the truth the true underlying state of the world is that chicago is not the capital of uh illinois but in lab experiments you poll a bunch of people on the street people will say yes not only will people say yes but um if you ask them how confident are you everyone who says the answer is chicago

is super confident that the answer is chicago so waiting the uh waiting for instance the votes buy some self-reported confidence score is not going to help you fix the problem okay so the question then is when we have just a bunch of experts we're not for now don't worry about them trying to manipulate the outcome they're just trying to they have information about whether or not chicago is the state of the world we're trying to chicago is the capital of illinois we're trying to learn is chicago the capital of illinois uh but majorities can be wrong can we still aggregate that information and how okay i'm going to propose a procedure

population based mean aggregation that uh fully aggregates in agents informations in the setting so if you actually gave me a large number of agents i can be very sure that this procedure will work in within the context of our model and the main innovation of this procedure is that it's going to be um is that it's going to allow not just for questions of the sort is chicago the capital of illinois but also multiple state questions so is chicago the capital of illinois or is springfield the capital of illinois or is urbana the capital of illinois like so there is a mechanism out there which i will review which is actually very clever and very interesting it's called the surprisingly popular mechanism it's

a surprisingly nice mechanism if you will that does it and once we've sort of covered the mechanism uh i i'll sort of show you also that it's not just that you can use this mechanism to aggregate the state of the world you can also use this to truthfully incentivize uh elicitation of the underlying beliefs even though you'll never learn a true state so you couldn't use a scoring rule uh straight up because you don't know the capital of chicago you're trying to aggregate that information to learn it in the first place okay so that's sort of the high level pitch if there are any questions um otherwise i'll soldier on okay so here is just a simple formal model

we're going to have a set of states of the world omega there's going to be l possible states of the world and there's going to be a bunch of agents so i'm just going to formally only consider the case of an infinite set of agents you can imagine laws of large numbers that give you good concentration and good properties even with the finite set what's known to the agents unknown to the aggregator is the following each of these agents is going to observe some signal that signal is going to be information to them about the state of the world there's some joint prior over the state of the world times the signal the agent sees conditional on the state

of the world and that's denoted by p this p is understood by the agents so agents understand at least probabilistically what information other people also have if i see a signal i know something about andy's signal yeah does the aggregator know the marginal along omega or not even that not even that so the aggregator knows all the aggregator knows is is he's willing to buy the common prior that the the agents are sort of understanding the thing and we can relax that too later but that's that's going to be the thing what agents are going to have then of course because they started with this prior p if agent i saw a signal s i

the agent i is going to update to some prior mu i and that prime ui is going to live in the space of probability distributions over and one more piece of notation i'm going to define given a prior assuming concentration holds for instance if the signals are iid across agents conditional on the state there is going to be a on there is going to be a state-by-state average population belief when the true underlying state is omega and i'm going to define that as mu bar omega again the um the principal the aggregator does not know any of these things they just know that there is ap

okay the question then is if there is such a set of agents if i knew p of course i could learn back the true underlying omega just by uh i know p if i ask all the agents for their signals i know an infinite set of signals that are drawn from some distribution i can back out what the state must be the question is what could i do if i didn't know p and i had to do something okay sorry assuming that each distinct pair of states induced distinct calculations for instance you could do that straight up if i knew the uh if i knew the p i i could do that even if it in induce different

population averages if it induced a different quartile and some you know the mean is not the only thing i could check if i knew p i could check a whole bunch of things in particular i could just check the whole distribution of posteriors and compare it against uh all right so here's the here's an example just to walk you through everything that happens and here is also the surprisingly popular mechanism of uh prelax young and mccoy explained uh within the context of a specific example so let's suppose we have the two states omega 1 is the capital of illinois omega 2 is that chicago is not the capital of illinois and of course omega 2 is the

true state we don't know it this is the district this is the specification of the priors so agents are going to see a conditionally iid signal everyone starts with the belief that chicago is likely to be uh the state of illinois at the capital of illinois with 0.7 probability uh chicago is not the capital with 0.3 and they see these signals so this is the probability of seeing sorry this is this probability this is the posterior that you have upon seeing signal one this is the posterior that you have upon seeing signal two so this is the conditional probability an agent would update to given their states conversely this is the

probability of seeing a particular signal in a particular state you can work out the population average beliefs that must be occurring in the population when a given state is the true state so if omega 1 was the true state then the population average beliefs if you work through the math it'd be five seventh and two seventh if uh two thirds uh if if chicago was not the capital then it would be two third and one third so this is the model that's known understood among the agents not known among um not known by the aggregator

and here is the surprisingly popular mechanism it's really easy to describe so we ask every agent what are your beliefs about the state of the world that's great i can calculate um i can calculate once i ask them all for their average uh all for their beliefs i can calculate the average belief by a law of large numbers the average belief in the population when the stake is omega must be the true underlying uh average mu bar of omega it's just that i don't know what the mu bar of omega should be state by state now note that this is by itself this is not

just if i knew this it doesn't tell me what the state of the world is because even if i understood the signal structure among agents for two different priors so for the prior 0.7.3 the population average belief when chicago is not the capital will be two-thirds third but if agents started with the prior 36 over 55 and 19 over 55 then two third one third would be the population average when chicago was indeed the capital so if you don't even if you have an infinite number of agents and even if all of these agents are getting a id signal without knowing the prior the aggregator is facing an identification problem

what's the sort of trick that psm introduce they say you suppose we don't just ask agents about their beliefs but we also ask at least one agent about their expected population average belief so you ask them what do you think you you don't just ask them what do you believe chicago is you know do you believe chicago is the capital of illinois with what probability you also ask at least one agent what do you think will be the average probability that all the participants in this poll will submit is there some reason we can't ask them

so that's a bit of a epistemic question of can you ask people about what their priors would have been before they saw any information so we normally don't like to assume that people can go back to what they would have felt was the underlying prior before they saw information people exist knowing the information that they know it's hard for them to back out that thing you're right i mean in some sense there's a trivial solution to this question uh just generalizing further i could just ask one of the agents tell me this whole distribution p and then thank you now i'm going to calculate this what i'm trying to propose is things that you might be able to work in practice and i'm shockingly actually going to have data at the end uh well prelex and mccoy

actually ran experiments showing that their mechanism works well in practice on a bunch of lab experiments uh it's worked well in the field i'm going to show evidence using their data that our mechanism also works well in on their practice to be fair because we haven't run out okay so what's the mechanism you ask one agent for the expected for their expectation of the population average belief so what do you think is the expected uh you're not sure what the population average belief is because you don't know what it is in society but uh you have some belief about it so tell me what your expectation of the population average belief is so for instance if the agent the agent understands that in

state one the average belief in the population would be five seven two seven the agent understands that in state two the uh average would be two third one third if they saw signal one their beliefs are 0.6.4 so of course they would report points if they were faithfully doing reporting their expectation of population average beliefs they would report 0.695.305 similarly if agent if the same agent instead saw signal 2 they would have reported this 0.705.205 uh 295 based on their weighting the the other information they have and this

that effective waiting in either case note that in the true state of the world chicago is not the capital of illinois the true population average belief is two third one third the average belief that results in the population so when you ask this agent what do you think the average should be they say well regardless of what information they see they either say that chicago should be the cap chicago is not the capital of illinois on average among the population with probability 0.305 or with probability 0.295 whereas in truth the average in the population comes out to be one-third so in both cases with respect to that agent chicago chicago

not being the capital of illinois is surprisingly popular prelax saying and mccoy mechanism just says the surprisingly popular state of the world that is where there is one state of the world out of two where the actual realized average beliefs in the population exceeds one person's elicited expectation of the population average in the population that's what you call surprisingly popular and they assert that that is the true state of the world well they don't they prove it i mean so this is this is not just uh heuristic you can give a straightforward reason for why this is

why this must be true okay it doesn't matter which agent you ask so that's a great question why is it that it doesn't matter with what agent you ask here's a theorem i i i haven't seen they have a different proof um there's a simpler uh theorem that is uh true so in a two-stage setting with two or more signals for any agent i the induced belief on omega that's a random variable in state omega first order stochastically dominates the induced belief on omega at any other state omega prime your belief on omega is a random variable that you will have some distribution of

that in state omega you'll have one distribution instead omega prime you have a different distribution of the belief you place on the state of the world being state omega the former first order stochastically dominates the latter so when you aggregate up the because you're just averaging a bunch of first-order stochastically dominant distributions uh the average belief of the true state always has to be surprisingly popular for any agent no matter whom you ask now some things i should note i uh the original preluxion in mccoy also works if you at the first order you

don't just ask people for beliefs on what is your belief that chicago is the capital of illinois you could just ask them is chicago the capital of illinois or not and elicit just the second order belief about what do you think the fraction of the votes is that works too in the original mechanism it works very well in practice for binary states i'll show you the data in a little bit or i'll show you the data right at the end because i'm a theorist but the issue with uh the surprisingly popular mechanism is that for more than two states of the world you need very strong assumptions so there is no counterpart to this first order stochastically dominant uh first order stochastic dominance relationship if you have more than two states of the world

this only works for two phases um what does that mean the like fosd like for distributions what's actual definition there what's the actual definition of first order stochastically dominant it means you put a lower probability the probability with which you put it so a distribution over probability distribution is just a probability distribution on the real line between zero and one it means a first order one distribution first order stochastically dominates another distribution if it puts lower probabilities on any lower interval so you look at any given x the probability distribution one takes a value less than x is smaller than the distribute probability distribution two

uh takes a value the random variable generated from distribution two takes a value less than x so your first order stochastic dominance just means that you are evenly likely to you are uh regardless of where you look in the distribution the first distribution is more likely to give you a higher number than the second distance well it has to catch up but it's strictly uh so the first order stochastic dominance is strictly below because it the survivor function is strictly above yeah yeah and then so even if you ask many people instead of just one person about their beliefs this would still not yeah there's no benefit in particular in this mechanism to asking multiple people

you could uh because you could of course take multiple people average them and look at what is surprisingly popular the theorem is still true but for more than two ah with more than two states um very non-knife edge examples start to break down so here is an example i'm just putting it up here i do not want to walk you through this entire set of numbers but it's an example sort of where you can just calculate again we're not doing anything we're not adding anything else so it's still a prior there's still unconditional uh there's still iid conditionally iid signals you're still calculating posteriors you are still calculating population average beliefs the trouble is that when you then run this mechanism

you could have multiple surprisingly popular states because with when you're comparing two numbers that sum to one with another two numbers that sum to one only one of them can be surprisingly popular but when you have three numbers that sum to one versus three other numbers that sum to one two of them could be surprisingly popular indeed this is the set of surprisingly popular states uh if you tried and did something like most surprisingly popular that's not going to be correct either i'm not saying that this couldn't work in practice but at least theoretically you'd have to make very strong uh diagonal dominance assumptions of basically in state one

something has a strong enough uh there's enough dominance on the diagonal to guarantee that something like most surprisingly popular works so you can't ask multiple questions you you might say tell me whether the state is omega 1 or omega 2 omega 3. note that this mechanism relies on the fact that conditional on a given state of the world the uh population average is deterministic or close to deterministic when i have two states of the world that i lump into one underlying state of the

world from the perspective of the question the population average condition on my question state of the world is random because it depends on what the actual uh resolved state of the world is right so if i ask is chicago the capital of illinois or is it urbana or uh whatever i can't resolve the question of whether it's urbana or spring conditional on the state of the world being urbana or springfield the population average distribution is going to be a stochastic variable it's it's you see what i'm saying all right so that's the mechanism that's their uh beautiful mechanism let me tell you our theorem here's the assumptions

so the assumptions uh for this for now just imagine that the signals are still conditionally iid uh we can generalize this substantially we need going back to tim's assumption we are going to assume that the probability the prior the probability distribu generating distribution is such that in any two states of the world the population average is different if you had some reason to think that population averages might be the same but you could pick some other quantile that was different you could work with that and that would be fine and the most tricky one perhaps to describe but you'll see where i'm using it because this is an embarrassingly simple mechanism to describe is that

what i want is that the support of the priors that people have in the population is a full support set it has an interior relative to the set of priors and delta else so it's not the set it's not the state like it's not the case that the state there are three states but all possible priors in the population lie along a single line okay and i'll show you some simple examples that generate that so here's the example again here's uh two states of the world chicago is capital chicago is not capital again agents are going to see a conditional eid signal these were the distributions now let me tell you the question that we're going

to ask that is slightly different or how we're going to do it that's slightly different than um uh prolex and mccoy so remember that by a law of large numbers when we know in the state of the world that chicago is not the capital of illinois the average belief in the population is going to average out to two-third one-third that we know we just when we see two-thirds one-third we don't know how to interpret it we don't know whether that's coming from because chicago is the capital or uh chicago is not the capital and like i showed you a few slides back for two different priors two third one third could have been the population average

for the same signal distribute for the same signal conditional signal information for two different priors two third one third could have been the average under both chicago and not chicago what we're going to do instead is just do the next thing that someone should have thought of and i guess we did um which is that instead of asking just one agent for their population average you have two agents now for their population average belief subject to the proviso that these are two agents with two different underlying beliefs to begin with so you ask every agent for their beliefs just like in pralik saying in mccoy but you now ask two agents what are your population average beliefs so agent one

who had reported 0.6.4 if you recall we did the math and we said this agent would naturally report 0.695.305 agent 2 who reported belief point 8.2 would do the averaging differently they would report 0.705.295 and the trouble is even though we don't know what the population average belief is state by state we know that for each agent their belief times the population average belief the their averaging has to equal their average so this is just law of total probabilities what must i be reporting for my population expected average it's my belief that the state is

state one times what the population average is in state one plus my belief that the status state two times what is the population average in state two if you have two equations and two unknowns you can recover the unknowns so you can just do the matrix inversion so once we have from two agents one who said 0.6.4 recovered that they think that the population average is 0.695.305 and this one is 0.8.2 says 0.705.295 we can infer that the only way these two agents could be making these two reports is if they believe that in state 1 it's 5 five seven two seven and in state two

the population average is two third one third then we just we now know five seven two seventh versus two third one third we compare to what we are seeing the population averages actually realizing to in the population and concluding the true state does the description of the mechanism make sense all right there is literally nothing in this mechanism beyond this uh inversion step so this would be a good time if if the definition of the mechanism is not clear all right so

more more formally what you're going to do you're going to ask each agent for their belief mu i calculate the average in the population mu hat select l different agents such that their elicited beliefs are a full rank matrix elicit from each of these beliefs their conditional population what they believe their expectation of the average posterior beliefs in the population alpha i and note that just alpha i for each agent i must be mu i times the unknown mu bar so you can just recover mu bar the vector the matrix of what the

population averages are state by state as the inverse you just want this matrix to be full rank preferably you want it to be a well conditioned full rank if you're going to do this in practice this works the theorem then is just straightforward under the assumptions the population mean based aggregation procedure recovers the true state almost surely okay it goes um you can start relaxing these assumptions so for instance i said signals are conditionally iid that is uh you're just getting a signal that is iid

drawn conditional on the state of the world the only thing we are using about conditional iidness is that the average belief in the population is a deterministic function of the state you don't have to assume conditional iid you can assume any form of limited correlation it's the only thing you need is that a lot of large numbers holds so as pick your favorite mixing property for instance vanishing correlation from far away agents uh as long as you can pick up a lot of large numbers from the appropriate textbook you will get back that the average population belief is a deterministic function of state and you can still use our theorem um or we have a theorem for that

uh the other assumption is that uh for any two states as tim pointed out we are assuming that uh the average belief in the population is different across states and this of course ensures that we can read off the true state from the recovered new bars but you didn't have to ask average beliefs you could have asked tell me your 70th percentile or tell me some other uh function off the average uh not off the average tell me some other function of the distribution of beliefs and as long as you're convinced that that is different across states and that's a deterministic thing that concentrates you can use this trick again i thought the fact that you've got a

linear system kind of was driven by using means no the linear system is from the fact that the agents when you ask them for their average expectation they they would take ex averages so you don't have to ask them for the expectation you just want any statistic that concentrates i know even in that it's true that you in the two outcome case you kind of get more from this mechanism and that you actually can compute the new bar whereas in the previous mechanism you didn't get that bar you just got the right omega in some sense you're if you believe the assumptions of the model within this mechanism you're seeing the mu bar for the state of the

world that it actually is now this mechanism will also tell you what the mu bar would have been in some other state of the world if there are settings where that is interesting to you then yes you're getting more but what you are getting more is that if you go to three states you now have something you can actually do yeah which is new bar in each of them and therefore remember surprisingly popular doesn't work for three states or doesn't is not guaranteed to work you can use it but you might get multiple surprisingly popular states and you might um like second order query is getting you one more conditional um other one was just that the convex hull

of the support has an interior relative so that that for instance if beliefs if the space of possible beliefs is the sim the three-dimensional simplex all the beliefs in the population don't lie a lie in a line that would be problematic because then i wouldn't be able to do this matrix inversion step um you don't need it's not a strong assumption it's a pretty generic assumption to just assume that uh these beliefs in a population will lie in a full uh in a full-dimensional set but one sort of cute reuse we found on a paper that did something completely different by people at least tim knows uh pretty well uh they did it for full rank extraction

uh in auctions but uh essentially if as long suppose agents this is paramount plane this is the criminal mclean generalization so deep in there somewhere they need uh they need to uh they need to prove that they can get back up to full rank and you can just reuse the same result to say that look even if you even if agents are seeing some signal that is not going to give the signal structure is by per se not going to give them to full rank but different agents see different numbers of those signals so uc l minus one draws of that so we have a non-generic signal structure but some of us might see more of those signals than others then

again the set of posteriors expands out to a full rank set um this is sort of just just the explanation of why but um yeah all right now what else can we do with this mechanism we uh could ask like i said the original uh psm paper asked agents to vote for what alternative is likely if i had two alternatives i could use our mechanism as well elicit words from all agents elicit expected vote shares and uh then uh as long as the population vote shares are different in the two states you

could still use uh pmba for the same reasons uh as before okay more interestingly uh and maybe something that you might be interested in is elicitation so so far i've just assumed that agents are going to tell you stuff truthfully when we assume that people are going to tell us stuff truthfully this works great but normally if you want to elicit beliefs normally to elicit beliefs you actually need to observe the underlying state of the world and use something like a proper proper scoring rule on the underlying state of the world because this mechanism is going to derive an underlying state of the world you could sort of

do a double scoring rule on this mechanism that then also makes it incentive compatible for everyone to report the true state so here's sort of how it would work you elicit just just the mechanism is going to run like the mechanism so you're going to elicit beliefs from all agents you're going to elicit expected population average beliefs from two agents since the population average beliefs among the all these agents the first order agents are observed rewarding the two agents with the proper scoring rule on the realized population average incentivizes them to report the their true expected the what they actually believe to be the expected population average so you've now got the second

population averages correctly now you can run a pmba you can extract the two state and all the agents you elicited their first order beliefs from you can reward their beliefs based on a proper scoring rule using the extracted state as the uh thing you score against and this way uh because i'm using an assumption that uh the set of agents is large so none of these agents can affect the underlying first order population average but given that um truthful reporting of beliefs is a uh there is therefore a bears nash equilibrium so you it's not just that

the mba is solving uh aggregation it can simultaneously solve uh solve uh incentivization so you could imagine a stronger incentive but basically if you look at any if you zoom in on any agent it's actually done i think the only place where i need a bears nash equilibrium is the it's it's um so it's not a dominant strategy because if i think everybody else is messing up then i mean i don't know what to do but

uh i it's not neces no longer and dominant strategy for b to report my belief truthfully uh so you could imagine a different mechanism i don't see in other i mean i think if you were willing to rule out just imagine in a situation where any given projection you just get like the prior scoring rule plus some additive shift which is a function

purely with other people's reports the issue is i think if suppose i think that everyone else is deviating in a way that even though my belief is 0.6.4 i think everyone else is deviating so that this mechanism is going to be screwed and report the wrong state of the world now i have an incentive to deviate and report that state of the world because that's going to maximize my score so this is the problem with uh yeah i was thinking

you realize so you don't actually observe if you then later observe the state it would be easier to pull this off yeah all right one of the things that this going back to your question about using two beliefs or asking more people for their beliefs in the baseline mechanism there's no reason to if you have l uh agents you can elicit it from them and be you know go home happy but if you think that agents are actually not hyper rational in the sense that they perfectly know exactly what's going to happen in the population they actually just have noisy estimates of

population averages then you can still get this mechanism to work by taking population averages or by by asking many people the higher order question and averaging over that if that higher order estimate is more sort of reliable for some concentration region um so the model here describes we have a model of sort of agents have just misspecified beliefs about the population average so they just have errors but those errors cancel out or in aggregate then you can just instead of doing instead of asking two agents for their population aggregates for instance you

could have two sets of la two large sets of agents for their population aggregate even though no individual agent knows the correct population aggregate the average will give you a reliable uh outcome and this is actually what we're going to use in practice in a second okay so this works great uh this allows us actually this can be generalized further so we can handle even uh some agents who are partisan and so on uh but we're still working out exactly what we can handle so i i in a in a highly optimistic move the final sentences of your abstract that was circulated to you promised you things

that you're not going to get today but i promise they will be there at some future point in the paper okay melissa is that related to like i think it's totally unrelated to your model yeah but behaviorally there's it's definitely newly popular in political survey to ask these higher order questions yeah with some kind of theory i'm not sure there's a lot of evidence here but there's some theory that you like report more honestly it's almost like a list experiment style idea but different where it's like i will tell you my real belief but if you ask me what i think other people believe i'll actually reveal something more truthful about myself yeah i think it's completely different from your model but it sounds like it could be related to this idea about partisanship no i i so i saw that uh i i

think i saw sort of the press on that last year and i don't have it gives me hope that again that these things one of the pushbacks we got when we started talking about this is yes but population averages who and you know people might be able to tell you their beliefs but who in their right minds would have any meaningful information about population averages and uh i mean that sort of thing gives me hope that there is meaningful information that people are finding so it's it's it's sort of yeah you're right it's outside my model but it's sort of it's more like evidence that we're finding that groups of experts not just know what they know but they also have some

meaningful information about what others know this paper is just taking interest to its limit and saying what could you do with that um and you can do a lot um so here is sort of uh yeah so a concern that you might have um is that this is trying to do too much with very noisy data so you're trying to invert a matrix based on elicited stuff about expected population average beliefs maybe we're just asking for too much maybe just the surprisingly popular is just more robust in some sense so how well does this mechanism work in practice um the uh authors the original authors are delightful in the sense that they both

on in the original paper promised that they would make their data available to anyone and then actually followed through on it um which unfortunately is not super common in uh the social sciences um it is also true by the way that um so i'll tell you a little bit more about the uh so here are the studies that they ran so they ran three sets of studies in study one they ran a state capital study so what did they do they told a bunch of respondents uh here is this prompt x is the capital as the capital of statewide so chicago is the capital of illinois uh birmingham is the capital of alabama 50 states they ran a bunch of surveys every uh responder had to respond with true and f uh with whether they think this is true

or false and also respond with their predicted population average in each case they also elicited a confidence so this is not just true or false this is how confident i am in that so they're not exactly running the study i would like to run but we haven't run our study yet the second study was general knowledge so student respondents were given general knowledge statements of the form you know just a fact about the world uh japan has the highest life expectancy in the world true or false or you know uh zaire uh the i don't know some some they pick an african country and name the capital do you think it's true or false

things like that uh again same thing response with true or false and predicted population average uh study three they had they collected a bunch of pictures of lesions and um they knew they had they already knew from the case files of those legions lesions whether those lesions were benign or malignant so in all three states the experimenter knows the underlying ground state the true underlying uh state of the world and they need to know this so that they can actually evaluate the mechanism uh so here it's slightly different and this is this one's going to be slightly trickier for us to handle two so they asked a bunch of dermatologists they showed them the lesion and they asked

them uh was it benign or was it malignant do you think this if you someone presented to you with this is it benign or malignant on a one to six scale one is benign six is malignant also they ask what do you think other dermatologists would say so one is like everyone will agree that it's completely benign 11 is complete everyone will agree that it's crazy malignant and scales are in between now these are scales so that's tricky but that that's that's life doing lab work um in each case we know the truth of the world we have the elicited data they compared majority and surprisingly popular we can throw in um uh our mechanism which um

so i just want to sort of highlight the numbers so there are 50 states i know this because i passed my citizenship test back um our capitals and um so majority for instance does really poorly so the majority of uh applicant the majority of elicited things for a given state matches up with the true capital of that state only 62 percent of the time surprisingly popular does really well it gets back 90 percent correctness and 90 of the time even though the majority does not know the state of the world the surprisingly popular answer is indeed the state of the world when we ran our mechanism we got close

to surprisingly popular so even though we are doing something slightly more intensive on the underlying data um we are close to surprisingly popular this is general knowledge again there were 68 general knowledge questions majority did basically a little better than a coin toss uh surprisingly popular did way better than a coin toss pmba does close to surprisingly popular the dermatology study now the majority is trained professionals you'd expect them to be good they do reasonably well 63 um surprisingly popular does significantly better 71 uh pmba doesn't quite

uh perform as well and sort of part of it as we think is the issue with dealing with the scale so we have several difficulties with this study they're not difficulties with their study but difficulties with using their study data for our things so i mean i should really just you do my own experiment and show off but instead i'm just going to tell you that it's awesome we were quite excited when we just finished doing this like last week and uh got pmba to perform close so for instance note few things one we are still in a two-state world so this is the best possible world for the surprisingly popular mechanism we of course are theoretically as good as surprisingly popular even in the two-state world but we're dealing with real-world data so this so we'd hope that in a three-state or a four-state

question uh elicitation we could do better note also that the original experiment elicited votes and not beliefs so we what we're doing when we do pmba is we are taking all the people who said true then using their confidence scores as their beliefs on the state being true averaging that out averaging their second order beliefs out so this is sort of pmba the robust version where you average in a population to try and get a better estimate and this is the one that performs the best we ran a bunch of others for instance the medi taking the median average median expected population average or taking sort of uh max

did this this one seems to perform the best and seems to perform the closest to surprisingly popular all right um there's some other theoretical niceties that you can prove in this population so going back to your question about why not ask agents the prior uh sort of one of the epistemic things in game theory is priors are just modeling devices people don't know priors people know posteriors but if they're hyper rational agents they can't just they could just not just tell you their posterior they could tell you they're posterior about everyone else's posterior they could tell you they're posterior about everyone else's posterior about their posterior you could construct an infinite hierarchy of

beliefs in theory you could just ask everyone to elicit this entire hierarchy of beliefs so two things one if a even if a finite set of agents reported their entire hierarchy of beliefs you have a mechanical procedure by which the posterior you would result in is the same as the posterior you would have gotten if you knew the prior and you just saw the signals directly so you can you don't need to know the prior per se knowing the infinite hierarchy of posteriors is sufficient even in an arbitrary correlated world even with a finite set of agents also

the converse is not true so if a you cannot guarantee that you could converge to the pooled information posterior of a sort of bayesian aggregator who knows the prior for any finite thing so in finite populations all we can promise is sort of you could do a mechanism you can do something reasonably good based on sort of your laws of large number bounds and conversely knowing higher order beliefs probably would not be super useful unless you knew the whole whole set um there's a whole bunch of literature on this i've cited sort of uh select it's it's another small uh community

that has crossed over into the ec community uh so this idea of using higher order beliefs has of course been used in aggregation it's also uh the further back idea was actually to use this for as elicitation so uh the bayesian truth serum was by the same author and based on the same uh idea uh and and that's been generalized substantially more interestingly this set of mechanisms has also been successfully used not just in the lab but also in practice so uh regal uh for some and roth for instance uh actively take uh these kind of peer prediction mechanisms to

um the field in india and use it to find out use pole micro entrepreneurs on what micro entrepreneurs think is the best which of them should be the best one to get the loan but also pull them for their higher order beliefs and use that to and they find that when they um i think my understanding of the finding is using that to allocate funds works better than like random allocation or using beliefs so uh this sort of thing seems to have some hope for actually working and that's sort of why i got interested i will stop early in the interest [Applause] the big question is the feasibility of

listening to sort of high order beliefs and practical settings and i think your point that like in some context there are these examples where we seem to be doing that yeah just thinking about like a blockchain setting like so i guess i figured i think you know if you think of like a community of people who actually like you know suppose go back to like the defy lending thing so there's a bunch of us who have to evaluate loans and i kind of know i know that i evaluate i don't know marketing projects well but i know you evaluate technical projects very well

and i get a signal when i look at the project i think the technical is weak so i now have some information potentially on how i think you will evaluate the project so in a in a community of experts who maybe knows each other a little bit or knows sort of stuff about the rest of their community or it's a repeated setting where they know each other's evaluation schemes over time uh you you'd imagine that they'd have some information about how other people might vote um so that that's one hope the other one is by the way i mean again i only have so these are entrepreneurs who knew each other they were from the same community so it worked uh this lab was a purely

you know your classical social science lab experiment bring a bunch of students uh this was dermatologists bring them into the lab they don't know each other it just it seems to work now has this can we scale this i have no idea uh it would be interesting to try and you're saying despite the fact they're in no position they're like computer posterior because they don't have the information structure yeah they i mean they don't i mean you i can't imagine that then you had a posterior but they're giving numbers that seem to meaningfully average to something that we can use um and and i guess yeah and going back it seems like it seems like there is information our people seem to have meaningful information about out there

about what they think population averages are people seem to be more willing to share that uh even than their own star than their own beliefs so uh finding ways to harness that is probably uh useful and interesting my impression is there this is like some new stuff i saw yeah on political science twitter so taking 10 grades of salt there's some people are really bad at self importing confidence levels i seem to perceive some i think average people very reasonably like i told you to answer the question so yeah i'm confident in it and they don't carry a lot of me with it but when you ask them the higher order thing they reveal that they have a lot less confidence

yeah and that's part of why i think survey people are starting to like it a lot yeah because it seems like we have all these crazy things in politics where people insist things that are pretty clearly false or true and i think it might be fake performance yeah and when they answer these higher order questions they seem to stop performing and so they'll admit that like well so i should have i should have had one more column here so because they elicited confidence they use they could actually do something called confidence weighted majority confidence rated maturity sort of robustly the sort of on average just barely outperforms majority sometimes does worse for the reasons that you're saying so that that's sort of why

basic i mean that's sort of the call for you should do something else other than majority uh and and confidence doesn't seem to help but yeah i i it is a ques it's an interesting question on how do you sort of robustly harness something that you'd be willing to um you know you'd be willing to bet money on that this works i can only tell you that this this promises this has promised [Music] i don't have a good sense um you know

maybe the only thing is sort of these were all these are all american uh so this this has tricky things to do with parsing the data because it's a one to six score and so on this is a straight up true or false question and it's um american lab candidates in us being asked questions about america versus sort of global general knowledge questions so maybe they just have a lot more information about what other people think about the country um so maybe it's a statement of their their higher order beliefs are actually more informative whereas here it's not just that i don't know the answer i also don't know really what other people know so everything is ill-conditioned when i start doing inversions yeah um is there

some like information theoretically which like this pmba is like optimal or like it's like upper bounds or no like a benchmark you'd have to so in some sense it the in the following this isn't a hard theorem but in the following sense if you buy the assumptions of the model um we know that you cannot do anything with just uh with just first order beliefs we are asking sort of l times l more information we know we can't do anything with anything less than that so we're as asking the least amount of questions you could possibly ask and get a guaranteed answer that is within the

model guaranteed to be correct so that's the sense in which this is information theoretically optimal i think but that's not a formal answer that's just saying we're asking very parsimoniously a few questions relative to uh what other information is out there that these agents have we could have done sort of you know you could if you really were brave i could ask you about your beliefs about my beliefs that your beliefs are about you know but that only happened in doctor strangelove and yeah all right thank you