Why Auction Design Matters (ft. Nobel Economist Paul Milgrom)

Transcript

Many of the assumptions that we make in economics are arbitrary for convenience, for simplicity, and they're just wrong. Pacific Bell asked me to look at this proposal by the government. What they had was just a mess [music] and ignored, you know, some obvious problems. The last time I went to the FCC, [music] I said, you know, it isn't that hard. Here's how you could do it with linked Excel spreadsheets, and I just handed them the disk. One of the areas that we will be moving into is creating [music] futures markets for compute. Here's the economic problem we're trying to solve. If it costs you $20 billion to build a data [music] center, why don't you build a $30 billion data center? Well, it's because it's very risky. Suppose [music] that next year Nvidia were to come up

with a chip that's 50% faster and uses 30% less electricity. [music] >> It's been a huge theme throughout your career that there's a practical problem that then spawns new theory that then gives rise to new practical designs. What does it feel like? How does it come about? >> are [music] two things you're conflating there, I think. One is that >> Hi everyone, and welcome to First Principles, a series by the team at a16z crypto. I'm Tim Ruffgarden. I'm the head of research at a16z crypto, and today we're talking about how prices form in complex markets and how the rules of the game can shape outcomes. Our guest today

is Paul Milgrom, the Nobel laureate in economics whose work on auction theory fundamentally changed how we understand bidding, pricing, and the role of information in markets. Among many accomplishments, he co-designed the FCC spectrum auctions that have enabled many wonderful technologies like 5G wireless, while at the same time generating over $100 billion in revenue for the US government. Paul's ideas have been extremely influential in economics, and in many cases they map directly onto the mechanism design challenges that arise in on-chain markets and decentralized finance.

Joining us is Scott Kominers, Harvard Business School professor and research partner at a16z crypto, as we talk with Paul about his foundational contributions to auction theory and what they reveal about designing better markets. Here's our conversation. All right, so Scott, we get to interview Paul Milgrom next. I'm really excited about this. What one very humbling thing to do is just sort of go back over all of the different things. Even let's just restrict to the paper writing. Like let's put aside the practical impact. Just looking at sort of the papers he's written over his career and it's just kind of landmark paper after landmark paper.

It's like, how can somebody keep this up for this long? It's just it's it's pretty mind-blowing. >> Well, and I'll tell you, not only has he written, you know, landmark papers in basically everything in economic theory, but every single thing he's ever worked on is the most exciting thing he's ever worked on. I mean, he just has this like incredible energy and like love for all of it. >> I got to ask, so um around the time he won the Nobel Prize along with Bob Wilson, you were actually a co-author uh on sort of a survey basically of of his work as as part of that. So, can you talk a little bit about the process of writing that and just what was it like?

>> was so fun. It was so much fun. So, the the Scandinavian Journal of Economics puts out a survey article on the year's economics Nobel laureate or laureates. Tommy Anderson, who who was the handling editor and and co-author and and collaborator with my my friend and and co-author Alex Tadelis, you know, sort of a network of networks, reached out and basically asked whether five of us together, all of whom had in one fashion or another worked with Paul and or Bob, Bob Wilson, uh who who won the prize jointly with Paul, uh would would work together to write a survey, and it was uh, incredible. So, uh, Alex Tadelisboyen, Piotr Dworczak, Mohammad Akbarpour, Shengwu Li, and I sat down to do this.

>> That's quite the dream team right there, so. >> Uh, I like I was astounded to be included in the list with those other four. >> [laughter] >> Like, um, it was incredible. So, first of all, like, you know, we we did our homework, right? Like, we tried, I would say, to read all of Paul and Bob's papers, discovering that even though we thought like what are the weird things is like, you know, most of them were were Milgrom and/or Wilson students. Um, all of us had like grown up in grad school like reading their work. And I always felt like I was still discovering I only knew like 25% of it.

Even on the topics I'm an expert in, right? It was like, you know, oh wait, like Milgrom introduced that idea in auctions, too? Like, who knew? Like, what? You know, and then we interviewed them. So, we had these long sit-down conversations to hear about the generation of the ideas and how they all fit together. Because one thing that it's possible to look at the body of work and not really understand without the context is they weren't just like writing papers in auction theory, right? Like, Paul Milgrom, yes, he was writing like what are now classics of economic theory on auctions and information and different forms of exchange, but they were in constant dialogue with applied problems. And a lot of the most fundamental ideas in these auction theory papers were coming out of an

either a thing he had observed in the field or a question, sort of like a thing that you needed to know in order to be able to take practical market design to the next level, and that we didn't have a theory language to speak about. And so, that, you know, precipitated papers. >> So, I'm I'm sure we'll touch on a a bunch of the things he's worked on, so maybe we should just spend a little time laying some groundwork. One thing I think would be fun to ask about is actually a very important, you know, paper in in effect finance uh, by Glosten and Milgrom >> [snorts] >> and and in particular like the idea it's almost like can you really probe the content of an efficient market hypothesis and really trying to

understand kind of in a detailed way how prices form in financial markets. >> One of the things we learned in that survey article when we were writing that survey article was that like why they got into auctions in the first place. Paul was really curious about the mechanisms and mechanics of price formation, right? Like how is it that markets actually achieve prices? >> Which I just have to say that's almost like a very obviously I'm biased I'm trained as a computer scientist but I think of that as like a super computer science-y way and almost critique of like say general equilibrium theory and like the classic economics. So it's just so cool.

>> Completely, right? Like you know, most of economics just sort of assumes that prices exist and then reason about them. Most of the way we use prices in markets very much rely on us having to have found them somehow. And a lot of economics sort of at baseline doesn't have a story about how those prices are discovered. Glosten and Milgrom proposed a really simple and incredibly intuitive answer, which is that story forgets that people have different information conditional on being the person who wants to sell versus the person who wants to buy. And so you can actually like you can get price discovery in a world where all the information is aggregated in the market because different people come to the

market with different information and that leads them to like take different sides of the transaction. >> And then if someone trades with you, you learn something, right? If someone buys something from you, you're like, "Ooh, maybe it's a little more valuable than I thought." Or if they want to sell >> aggregation in the market comes from the execution of those trades. This one paper was foundational in in information economics and market microstructure and finance. I mean like this one paper is sort of like helped create three different modern economic fields or something and that's like You know, I guess that's like, you know, an afternoon at the office for Paul Milgrom.

>> Yeah. So, obviously we have to talk to him about auctions, right? And I think what could be fun is to zoom in on kind of like the early '80s when there was a lot happening in auction theory. I mean, so you know, Vickrey was back in '61, that kind of laid out, you know, how to think about it, properties you might want, second price, first price auctions. You know, Bob Wilson, who we mentioned, tons of work in the '60s, but some really interesting things happened in the early '80s. And one by a different economist, also a Nobel Prize winner, Roger Myerson, wrote a paper optimal auction theory or optimal auctions, I forget the exact title, which asks about, you know, if you're a seller and you're selling an item, what's the way to sell it that will maximize your revenue, framed as an optimization problem, which

of course computer scientists love, right? So, computer scientists read the Myerson paper and they're like, this is exactly how I want to think about the problem. >> [laughter] >> And fun fact, economists were initially so, you know, sort of confused by this way of framing the problem that there's a whole follow-up paper that reframes it into economic language. >> [laughter] >> It's a great illustration of like how economics and computer science perspectives work together. Like, it turns out that trying to like deconstruct what's going on in an optimization can lead you to economic insights you would not have found otherwise, and those economic insights have since gone on to like frame lots of new optimization questions. It's actually like seeing early versions of that feedback that we see a lot of today.

>> Absolutely. So, this is a landmark paper, um but then simultaneously, Paul was also writing a landmark paper >> Yep. >> with Weber. So, there's this complimentary sort of, you know, equally mind-blowing paper. So, could you talk a little bit about that? What what what Why is the Milgrom and Weber paper so famous? >> One thing I'm sure Paul will talk about, like there are several different ways in which economics papers contribute to our understanding. One of them is through the specific answers and sort of like design insights and and predictions they give. Another is through the framework, the model, the way of thinking about the problem that they produce that then turns out to be useful for thinking about many other many other problems.

And you know, Milgrom and Weber gave what has become essentially the single most standard and like viable framework for reasoning about auctions with correlated values in various forms. Um what are called affiliated values. >> And just in case this sounds academic, I mean I mean Scott, how many times have we been in conversations with startups, with founders, and we literally we refer to this paper when we give advice on kind of the details of how to run for example an NFT auction. >> and in particular like, you know, so many auctions in practice. Like this is almost the normal case, right? An auction where there's some sense in which everyone's assessment of the value of the good is informative about what

the value of the good actually is. Right? >> So in other words, you can sort of piggyback on others' knowledge about >> Exactly. You can piggyback on others' knowledge. And sort of learn things from their bidding about what your beliefs about the value should be. And like that's not a thing that's relevant in every single auction, but it's relevant in so many auctions. And it's and like and you know, it might be relevant in all crypto auctions, right? Like like >> Yeah. >> you know, in crypto we're usually we're often selling a digital asset as an part of an exercise in a forming a sense of value around that asset. Like you know, you're selling early access to an NFT community. You know, first round NFT minting sale. Or you're you know, selling token units that are then going

to be used as the network token of an ecosystem and you're trying to reason about who's going to be participating, how much do we think this ecosystem is going to grow, like who's going to be using it, and how much do we think these tokens really are going to anchor a community. And all of those things you're drawing inferences from seeing other people's bids. And if you can, things from you know, inferences from who's bidding. >> Well, there's a million auction theory papers we could zoom in on. I'm sure we'll touch on some others, but clearly we're going to be talking about that one, Milgrom-Weber. We also have to talk, obviously, about Paul's work on the practical side. He wrote a famous book called Putting Auction Theory to Work. I love that title. You know, the theory is a great starting point, but then of course there's a lot of hard

work to do in the trenches. I'd say both to fine-tune how the auctions work to take into account real-world constraints, but then also just the political process of actually, you know, getting all the stakeholders to to be on board with it. And you and I both seen firsthand the work that Paul has put into this, yeah. >> We'll have to ask him some about how the conversations around the first US spectrum auctions worked. Like, you know, how did he collaborate with the Federal Communications Commission? Like, what were all the different parties involved? Because that was sort of, you know, one of the first big like public projects of market design. Sort of the modern brand of using modern economic theory jointly with computer

science and operations research and all of these different toolkits to redesign or to design from scratch a marketplace towards a huge public goal. Like, the spectrum auctions are one of the first really big examples. >> Um and I think we got to ask him about the FCC incentive auction, which is more recent, that's maybe maybe eight or nine years ago. I think unquestionably one of the biggest market design triumphs of the 21st century. >> And I teach it in my classes as like, it's market design as entrepreneurship. You know, you are building this thing.

You're imagining a major change in the world that you are going to facilitate that at best is going to happen 7 to 10 years down the line. Right? Like, you will not know whether you've succeeded. And like, market design entrepreneurship is like this. Like, you don't know whether you're going to succeed until, you know, a huge change in how people do business and how consumers go through their lives. And you have to along the way, like both have the energy and the confidence and the clarity of thought to foresee the problems, to sort of, you know, look downstream and optimize in the moment at the same time, to iterate, to explore, and like, you know, sort of build a coalition around your marketplace design that will enable you to reach that downstream goal, constantly updating

even your understanding of what the goal is. You know, and as I said, like every single thing Paul has ever worked on, he's like the most excited he's ever been about anything. It's just like an absolutely wonderful characteristic of him as a human. You know, I would see him, like, you know, he'd come in to Chicago or Harvard to give a seminar or something, like, and we we'd go for a walk and he would tell me about the new impossible problem in the design of the incentive auction. And like every single one of these impossible challenges, he was just like, "And this is going to be so fascinating to solve. Like, here's here's what we're thinking about so far. Here's just constantly treating everything that should have been a totally impossible, unsurmountable roadblock as this like huge opportunity to discover something new."

>> Yeah. Amazing attitude. Well, clearly we should ask him about what he's excited about right now [laughter] and feel that enthusiasm first hand. So, having laid that groundwork, let's talk to Paul Milgrom. I thought maybe we could start the conversation talking a bit about auction theory, which obviously has been a a really major theme throughout your career. >> Mhm. >> You know, and I think when a lot of people first encounter auctions, they think, "Oh, yeah, you know, this seems like a reasonable, practical problem and, you know, there's sort of nice solutions." And, you know, one thing I find fascinating is my understanding is your approach when you came to auction theory it was from sort of much deeper intellectual roots, coming from sort of general equilibrium theory and price theory. And that's a story I suspect a lot of our audience doesn't know. So, I wonder if you could

start by, you know, just talking a bit about what brought you to auction theory, you know, in the late '70s and what were you trying to achieve in in doing it. >> [snorts] >> So, actually, if I go to what brought me to auction theory, it wasn't any deep intellectual thing like that. I was in an MBA program at Stanford and the faculty recruited me into the graduate program because of, you know, the way I was in class. And I was advised that I should get Bob Wilson to be my PhD advisor. And so I decided I would do that by, um, writing my term paper for his class about something that interested him. So I wrote a paper about auction theory. So it didn't have any deep intellectual roots,

um, at the time. But, um, as I continued to take graduate classes at Stanford, the question about how prices emerged and why they had the properties they had, there was, um, at the time this, uh, very puzzling work by Grossman and Stiglitz which had said that people inferred information from prices, but how did the information get into prices before you transacted? And and they had these paradoxical looking results that made it seem that nobody had any incentive to gather information cuz the information was already in the prices, but if nobody gathered information, how did the information get into the prices? So looking at the process of price

formation became important. And that led to the paper I did with Larry Glosten where, you know, there's there's a process by which prices are formed. There are bid and ask prices. After the transaction, they came to reveal the information that was used by the transacting party, but not before the transaction. Before the transaction, you didn't know whether the bid price or the ask price was going to apply and it was only after the transaction that you learned which applied and so the transactions prices reflected information. The consumer could then profit from, uh, this information. If the consumer didn't have information, it wouldn't be reflected in the transactions prices. So it was a logical way to account for, uh, these paradoxes

of general equilibrium in financial markets. >> I'm glad you brought up the the with Larry Glosten cuz um it's interesting. I mean obviously So that's part of a literature called the market microstructure, which you know, is very famous in the economics and finance world, but actually with blockchain technology, one interesting thing that's happened is there's a new generation of sort of computer scientists and engineers that have now through decentralized finance gotten quite interested in the foundations there. And so I'm hearing about, you know, the Glosten-Milgrom paper now, you know, all the time from groups where historically I hadn't heard as much about it.

>> So I'd love to just actually probe a little bit deeper there. Did that kind of come about sort of hand in hand with the work you were doing just in the more pure auction theory side, call it? >> And maybe for the audience just if you can quickly review what the paradox is and what your resolution is cuz it's a really slick and simple insight and I think a lot of people listening, you know, if they're thinking about prediction markets or decentralized finance or whatever, they may not actually know the sort of fundamental paradox yet. >> Well, the paradox has to do with the intellectual history of the way things were modeled. It's only a paradox when you're when you're married to old general equilibrium theory and and in old general equilibrium theory, what happens is people can't influence prices. They think oh, the markets are

very big. Prices don't depend on what I do. And they think that prices reflect information. But how do prices come to reflect information if people can't affect them? That's the fundamental paradox. And it was reflected in formal modeling by Grossman and Stiglitz, which showed that if if you made these assumptions, you you could conclude that people had no incentive to gather information since their information was revealed in prices.

But then if they didn't have an incentive to get information, how would the how would the prices be informative? That's a very paradoxical result that comes from a very strange modeling. And that sort of opened the door at that time the way the economist thought about prices as well, they didn't think about prices. They said, you know, there's supply and demand and the point where supply and demand crosses is what the price is and we don't talk about how the price emerges. And what option theory was doing and what Bob Wilson's program was was to say, no, actually there's a process behind all of this.

And and we should look more deeply at the process. And what Larry and I did said, well, you know, at the time the way specialists ran markets is they quoted a bid price and an ask price. And it turned out in our model that a specialist didn't want to lose money. They knew that some traders were informed. So, they knew that if a trader was buying from them, that was some indication that that trader had positive information that prices should be higher. And they knew that if a trader was selling from them, that was some indication that the price was too high and prices should be lower. So, the bid price and the ask price were separated by this value of information difference.

Prior to our work, people thought the difference between the bid price and ask price reflected some costs of inventory that that were being held by the market. >> know that. >> That's wild. >> Oh, yeah. And later on, the empirical work then looked to separate how much of the spread was due to information and how much might be due to some some costs of holding inventory. It turned out a big chunk of the difference really was due to information. That's what the empirical work wound up showing. So, it was a very simple and and it came out to be mathematically quite beautiful. The people in finance had this idea, roughly speaking, that prices formed a martingale. That is that the uh

you wanted to predict the price tomorrow in expectation it was the price today. But this was in a model where there was only a single price. And we had a model which there's a bid price and the ask price, but the transaction price is formed a martingale anyway because if the guy bought, then the information that the guy was buying was already in the ask price, and so the the transaction price would reflect the information that actually came into into the market, and we got the martingale property back. Uh and some other uh things as well, things that talked about how markets break down when there's too much asymmetric information uh and a few other uh things.

>> Fascinating. This is maybe a little bit of a digression, but let me ask you here anyway. So, anyone who reads papers in the top economics journals, you know, in the '60s, then '70s, then '80s, then '90s, and then into the 21st century, I think observes this evolution of becoming more quantitative, more mathematically rigorous, more precise, and and Paul, certainly, you know, throughout your entire career from the very beginning, I'd say precision and rigor is is there in every single paper that I've that, you know, that you've ever written. I was wondering if you could sort of place your early work in the context of that evolution. Uh like, take us back to, you know, how was, you know, more fine-grained mathematical modeling viewed at that point? Was that in and of itself something relatively new to the um economist profession?

>> Well, I think, you know, actually, you can go back to the '50s and find certainly if you read Gerard Debreu's Theory of Value, it's very mathematically uh precise. He lays all the careful mathematical foundations and and seeks high levels of generality. You know, I think part of the reason when you're an economist, when you're looking at these things, sometimes when you're trying to illustrate in a model what's going on, you know that your model's a simplification. So, all you're really trying to illustrate is possibility. And when you're trying to illustrate possibility, you you don't seek maximum generality. It's not really so much as less mathematically uh mathematically rigorous as being less it's less

general. That the people are making very specific assumptions which we don't expect to be true in the world in order to illustrate how some effect would emerge. Whereas if if you start following from Arrow and Debreu and those guys, they were looking for generality. They wanted to know how widely can we conclude that a general equilibrium exists or that general equilibrium is efficient or all the other kinds of properties. When you see it being more mathematical, I think part of what you're seeing is attempts at greater generality. And you can see why as we dig deeper into ideas you would seek you know, greater generality because we

want to know, okay, how far can we push this idea? And and that's what I think came out more and more starting in the '70s and '80s and moving forward. >> How do you think about the use of math to help search for and understand ideas? Like you know, I've noticed in in working with you like your ability to see the generality is a huge part of what frames the economic sort of the economic space. How do you think about that? >> For me, um I I was always amazed that it worked this way when I got into economics.

There was always this back and forth between the the verbal descriptions and the mathematics. That is I wanted to be able to explain what I was doing to say it in in terms of economic concepts. And often the math would teach me something and say, oh, this is the condition I needed for the result to be true. Here's a mathematical condition. What does it mean in the world? >> Let me bring it back to to auction theory for a minute. In particular, I I love to chat a little bit about the you know, famous 1982 paper with Weber.

And I'd love to actually before you get into that and and just a little bit about sort of again setting the stage. And in particular, there was another extremely well-known paper that came out the year before by Roger Myerson on optimal auctions. And so I'd love to hear a little bit, you know, what was your reaction to that paper when it came out? And I would say they're very complimentary in my view at least the the Myerson and the Milgrom-Weber paper. So just a little bit about how those fit together. >> We were just sitting two doors apart at the time we were working on that paper. So the papers were growing, you know, more or less simultaneously. You know, Roger's inclusion of some common value was borrowed from the work that Bob Weber and I were doing.

Roger was thinking quite orthogonally to, you know, what I was thinking about. Mechanism design was not a a typical way to approach things then. And I was trying to understand these two completely separate threads of auction theory, the one with the common value model that that Wilson had done a lot of work on, and the Vickrey model which was the independent private values model. And I wanted to unify them and see, you know, what could be said, whether Vickrey's results still applied in the unified model, and whether whether Bob's results could be extended. There was also again a mathematical detail that the

mathematics, as you point out, was very important to me. I had an undergraduate math degree, and I knew what it meant to be careful. And the people were just writing down these first-order conditions and writing down what the solutions would look like, and I wanted to know whether these were actually equilibria. It turned out we needed affiliation for that. And then affiliation drove some of the other results as well. >> And that that same paper, in addition to, you know, this sort of generalization, you have independent values capturing the common value case and the independent private values case under sort of one general framework.

Another big takeaway for me at least from that paper was this kind of relentless focus, you know, on the auction formats that we already know matter. So, rather than just sort of formulating an optimization problem and say, let's see what we get if we solve it, but really let's try to speak to people, you know, give guidance in effect. If someone actually has to make a decision in the real world about, you know, should they run an English auction or should they run a sealed bid auction? Like, what can we as analysts sort of tell them? So, I was curious, like, was that did you have sort of practice in mind, you know, in making those decisions in that paper?

>> I didn't really. I mean, remember um I wasn't really making the choice that you've just described because mechanism design was not something we knew about at the time. Um it was totally shocking when Roger formulated the optimal auction principle, the revelation principle. There were versions of it around, but it certainly wasn't widely known at the time. And the idea that you could find an auction that was optimal among all possible mechanisms was was shocking to people. I remember I remember at the time um having a conversation with Sandy Grossman, who was very famous back then, describing that Roger had found an auction that was optimal among all auctions. And

Sandy said, "Well, what do you mean all auctions?" I said, "All." I said, you know, he said, "Well, for example, suppose that I have five people and I uh collect bids from them and I take the first and third highest and then put them in a qualifying round or a random, you know, and and then I, you know, he went through a whole series of steps and says, "Does it include that?" I said, "Yes." Um >> [laughter] >> and he he just didn't believe it, right? So, no, I I was not choosing. I was sort of driven by what I understood at the time.

>> Yes, maybe just for completeness for the audience, as far as the revelation principle, I would maybe describe it sort of fast forwarding to the equilibrium point. So, in some sense, for the purposes of identifying optimal auctions, really what matters is just sort of allocations and payments at equilibrium as a function of people's willingness to willingness to pay. But, what's what's sort of I think fascinating is Paul, we've also seen really interesting results much more recently last 5 or 10 years, including from your own students, about actually saying, "Well, wait a minute. Wait a minute. Maybe we lose a few things when we just think about the revelation version." Would you care to to you know, speak about the the sort of recent trends about really focusing squarely on indirect mechanisms and the properties you can only articulate at that point?

>> I'll start with my own piece of that because it was while we were working on the broadcast incentive auction. >> Mhm. >> And Mohammad Akbarpour and Shengwu Li were both graduate students at Stanford. So, we had to create an auction that worked when checking the feasibility of an allocation was an NP-complete problem. Um so, it was it was hard. And we kept changing the algorithm as Kevin Leyton-Brown was was developing the algorithms uh that were getting better and better at solving a larger fraction of these in in a reasonable amount of time.

And I had puzzled to my graduate students that we had always said in in economics that a mechanism is defined by its outcome function. And here we were changing the outcome function and yet um all of these things we knew even without looking even without knowing the outcome even without being able to compute even not without being able to describe the outcome function, we knew that these things were strategy-proof. In fact, it was just obvious that they were strategy-proof. And wasn't that weird? Okay. And Shengwu says that he listened and scratched his head and said, "What could obviously strategy-proof mean?" He was interested in that because he also did behavioral

economics and knew that you could have an ascending auction and a strategically equivalent second price auction, and people would play the ascending auction more accurately. They'd be more likely to pick their dominant strategy. And, you know, it in some sense it was clearer to them what a dominant strategy was. So, so he brought in this this behavioral idea, and it turned out to be quite useful. How can we design auctions that are easy for people to bid accurately in, um, that have good properties, and that requires going beyond direct mechanisms.

You have to take a look at all the details of how the mechanism works. So, it was pretty interesting. >> You've started alluding a little bit to this feedback between theory and practice and the layering of it, right? That there's a practical problem that then spawns new theory that then gives rise to new practical designs. It's been a huge theme throughout your career. Could you maybe talk us through what it again, what does it feel like? How does it come about? Do you seek it out? Does it find you? >> So, there are two things you're conflating there, I think. One is that I always thought that in writing theory there was always this this feedback between how you would express something.

I was teaching myself. I mean, I have no degrees in economics. I I studied in the business school in what was called decision sciences, and I kept trying to explain things in terms that were familiar to economists. So, moving back and forth between the mathematics, what it said, and the economics. Sometimes the economics was more general. When I expressed it in the economic terms, it was more general than the mathematics I had. So, then I'd have to go back to the mathematics and say, "Can I generalize?" And sometimes the mathematics would express things in ways that were surprising to me, and I'd say, "How would I say that using economic concepts? Can I, or does it require new concepts?" So, there was this this feedback always going on between, uh,

economic concepts and the mathematics. But, there's also a feedback, the one that that I think you're emphasizing, between theory and practice. And here it was really scary because the very first of my big practical applications, which was the original US spectrum auctions, Pacific Bell, which was the local, uh, telephone company in California back in the early 1990s, asked me to look at this proposal by the government. And I said, "Oh, I I could this is way too hard. This is, you know, for I'm just a theorist. I don't know how to how to do this kind of stuff."

And Pacific Bell said, "Well, you know, would you please look at it for us? We'll pay you for your time. Just take a look at it and tell us what you think." So, I studied the proposal that came out of the Federal Communications Commission. And I said, you know, "It's a really hard problem and I don't know what the right thing to do is, but I can do better than that." Cuz what they had was just a mess and ignored, you know, some obvious problems. So, Bob Wilson and I, um, proposed the simultaneous multiple round auction, which ended up being adopted in the US and worldwide, certainly in all the English-speaking countries, but also, you know, also in Germany and Mexico and, you know, other countries as well.

And it just took off and and it solved some of the problems. And I was very nervous when we first ran it because I knew how incomplete our analysis was. I had solved some of the problems, but I didn't really know how it would operate in in other respects. So, it was scary, but after that, lots of people came and started asking my advice. After the US, the first place was Australia that came next. And that's when I first became aware, by the way, of the computer science issues because the US had decided that it would break the country up in these pieces, you know, significant chunks of, uh, of geography and significant chunks of spectrum and sell those licenses. And people could decide what combinations of

licenses they wanted to buy. One of the Australian regulators said to me, "It's as well, you know, if people are putting together their their own combinations, why would you start with something that big? Why don't we have these what they called postage stamp licenses, which would really small areas with really thin pieces of spectrum, and let them put it together any way they wanted? You know, any combination you want you could bid for." It was obviously an inherently combinatorial problem. And uh I was horrified by the idea that you would try to push the auction that far. Realized that told me how little I understood about how the auction actually worked and when it would work, and then inspired a lot of new work on

on theory, but it came from, you know, for me it came from the regulator making a proposal that I knew wouldn't work, but I didn't know what to do about it, right? >> So, Pac Bell got you involved really just to sort of advise them potentially on bidding in this format, but then you said, "No, no, no, no. Let me let me zoom out and advise everybody how this auction should >> Oh, no, no, no, what happened? No, Pac [laughter] Bell, the the government had was deciding what rules to set up, and they had a proposal, and I can't even I could go back and try to tell you the proposal, but it was god-awful. It was really a terrible proposal.

And I told Pac Bell, "Not only is it bad it's bad for the government and it's terrible for you because you're going to end up losing the auction to uh companies that want to buy a package of all of the spectrum in the country." And And I said, "You know, we can do better for the government and better for you." So, I ended up making seven trips to Washington, D.C. to talk to the Federal Communications Commission. And um when we did the simultaneous ascending auction, this all came This is something I'm proud of, so I'll tell you this this particular story.

>> Absolutely. Absolutely. >> said, "Oh, that's way too complicated. You know, you should just do it like you were Sotheby's or Christie's or something and just sell these pieces one at a time in a particular order that they would work out from maybe what they thought was most valuable to least valuable, something like that." And this is back in 1993 that I'm doing this or 1992 or three, I guess. And I said, "Well, it isn't that complicated. Uh and we took and I had my research assistant program these little We had these little 3 and 1/2-in discs. Do you remember those?"

Um >> Mhm. >> So, we programmed a series of Excel spreadsheets. And one Excel spreadsheet was the spreadsheet on which you would submit your bid. And it would check whether your bid met the rules. And you'd put your disc in then and your bid could be uploaded by the FCC's Excel spreadsheet, which would then create a combined spreadsheet and then run around and then export the results back to an outcomes file. So, I had all this on a 3 and 1/2-in disc. And I went the last time I went to the FCC, I said, "You know, it isn't that hard. Here's how you could do it with linked Excel spreadsheets." And I just handed them the disc. And Evan Kwerel, who was the guy who was uh being relied on to do this at the FCC, took it home

and ran it on his home computer and said, "Actually, it works, right? It must be possible to do this." And uh when the when the chairman of the FCC, Reed Hundt at the time, asked, you know, "You can do whatever you want." "Uh what would it be?" He said, "Well, this is the system we'd want to use." And the chairman said, "It's pretty innovative, though." Yeah, and the turn They said, "Well, I like being innovative." Uh Reed Hundt says, "So, let's do it." So, they adopted this system. And and it spread worldwide and, you know, Bob and I got famous for it. So, there we are.

>> Yeah, I I I love this st- It's also just a great example of academic-government collaboration, right? Something that probably doesn't happen enough, but like that's one of the absolute you know, best ever examples of it. What finally convinced them? Like, did you Was it specific examples of how different parties would about the sort of low quality of an allocation you would expect to get from the say a serial sort of mechanism? I'm sure you had to pitch it a few times before they really before you could sell them on it. >> I did it and it wasn't just a matter of them believing or not believing. They hired uh Charlie Plott at Caltech to run some laboratory experiments to see if it would work. They ran some experiments with, you know, Caltech students and

they also got me to make some changes to the the activity rule. They said, "Well, what if somebody wanted to screw this up? What's the worst case? How long could it take?" So, the activity rule was to put a floor under how much bidding people could do so that the thing would converge in reasonable time in worst case. The Caltech experiments confirmed our predictions about the what what would go wrong with the alternative uh proposals that were made at the time. So, uh all that went favorably and then they hired somebody to implement that who was incompetent. Um They had software that nobody was sure would work, but they were running it in a very small auction where that the initial task

was seven licenses for paging spectrum. And uh nobody was sure it would work and so they set up a a room where there was a blackboard and they were going to If the software didn't work, they were going to run it by by hand. Then it was uh Meanwhile, you know, my reputation was at stake, right? Was this thing too complicated? Could you actually do it? But it worked like a charm and the auction won. We got arbitrage between licenses that were essentially identical, which is what we wanted to get. They all end up selling for exactly the same price. Everything came out in a perfectly reasonable way and they decided, yep, it worked in the lab, it worked according to these theorists, it

worked in the first test which was so-called small-scale test only like what was it? A hundred million dollars was our small-scale test remember. The auction four which is what we were really designing for was seven billion dollars. I mean it was a lot bigger. Um So it was a lot of money at stake. Someone got I was fine. I uh I was excited by the whole thing. >> Amazing. So that that's not the end of the story um with your work of spectrum auctions and I do want to you alluded before to the FCC incentive auction from uh I don't know 7-8 years ago. We'll get back to that. So then between these different chapters of of of your spectrum auctions work something

interesting happened which was the rise of the internet and of the web and of sort of the digital economy. And so you know, something that we witnessed at that time was auctions going digital and all of a sudden it actually having billions of auctions run every single day. And you had you know, systems like Facebook literally implementing the Vickrey-Clarke-Groves mechanism under the hood. And so I'm just curious, you know, your reflections on that time and um any personal stories you have about how auctions going digital kind of changed how you thought about auction design. >> Well, first of all about Facebook, um that was John Hegeman who was the student in the first undergraduate market design class that I taught. I taught him the Vickrey-Clarke-Groves

mechanism. >> Wow. >> And uh John was a terrific student and he entered the graduate program in economics at Stanford and Facebook lured him out of the graduate program to go take over uh that stuff. So I was uh uh a little disappointed to see him leave. And you know, of course we all know about the uh Google's misunderstanding of the second price auction and their generalized second price auction for uh for search advertising. It was all very exciting. And several of my students ended up getting involved.

Another one of my students ended up going to Yahoo where they were saying, "Suppose you did search advertising, but the ads aren't all the same size. So, how many ads you can fit on the page depends which kinds of ads you have." So, that was a combinatorial auction cuz you could have different combinations depending on what size ad and then you had bidders who were uh okay, well, in that case, how do they describe their values? How do they compare the values for four-line ad versus a five-line ad? And Google did some, you know, brilliant simplifications. One of the ideas that struck me back then was simplifying the communications. You were paying per click on search advertising and and uh actually a double click as as of 1996, I

think it was was already uh uh on a pay-per-click system for uh for advertisers. They had Google running the auction. They were predicting uh click-through rates. So, just making one value per click and predicting click-through rates and running a position auction where you would expect to get fewer clicks. They had a simplified model where the number of clicks that you'd expect to get maybe falls 30% or so with each position you move down the page. They had some uh simplified model that made the reporting very simple.

And then I remember having the discussion with Preston McAfee where you know, we started understanding that bidders had different objectives. But they don't all have to use the same language. We could have them describe their objective in different languages. And we were already seeing that that Google was converting cost-per-click information into cost-per-impression information in order to run an auction for uh for advertising slots, you didn't have to be speaking the language of the bids to communicate values, and keeping it simple was important, and that carried over into what we were doing in the incentive auction, too, where I was also very eager to make it easy because, you

know, if you're a practical person, one of the early lessons you learn is that perhaps the most important thing you can do in auctions is encourage participation. You need the bidders to come to get a good outcome. And if you make it too complicated they don't show up, right? They just don't participate. They don't know what to do. So, simplicity was also important. All those things were you know, as a practical person, were coming on my together. >> That's so fantastic. So, moving on to the FCC incentive auction, which took place over 2017 and 2018. So, you know, so early '90s was when the spectrum auction work started. This was, you know, now really almost 25 years

later, you sort of overseen and innovated many times in that interim. But, definitely there is something very new in the FCC incentive auction. So, maybe you could start just by saying, you know, why was the level of innovation required fundamentally greater than anything since 1993? >> Well, there were two main reasons. The FCC incentive auction was completely different animal from what we'd done before. In all of the auctions until then, as technology advanced, new frequencies which had been unused were being made available for new uses, and the government was selling rights to use frequencies that it controlled.

Um before the incentive auction, maybe around 2010 or so on, all the frequencies that were useful with the new technologies had been used up. The reason you couldn't use a higher frequencies for some of the uses was not that you couldn't encode signals on them, but the higher frequencies might not penetrate brick walls or into glass and steel buildings. You know, you could send signals on them, but the signals wouldn't reach the users anymore. So, we were out of the most valuable frequencies at that time. What was required was a reallocation.

You needed somebody who already had rights. In this case, it was television broadcasters to give up their rights to use frequencies, and then you needed to reconfigure them. By the way, we won an Emmy award for this, you may know that. I have uh >> I was about to say, I heard you won an Emmy, like, you know. >> I have this little Emmy statuette. Uh it was to auction off the mix that we were a winner of the technical Emmy. Um for reorganizing the television industry. So, what was going on at that time was fewer and fewer people were watching live broadcast television over the air. More was over the internet, satellites, cable. That's how people were seeing their shows. And um so, the value of radio of the spectrum

for over the air broadcast had declined. In addition, they were coding more efficiently, so the old analog 6 MHz blocks were more than you needed to code a high-definition signal in the digital format. So, the value had fallen in in over the air broadcast, but for wireless broadband, it was growing demand. This was much more intensive use of wireless data, including video over the internet was coming along. So, we needed a reallocation. And the old licenses couldn't just be turned over. If if Scott over here had a television license to serve Chicago for 6 MHz, you couldn't put a wireless data plan on that. It's very much like zoning

and and and which you can think of of zoning real estate uses. You don't want to have a factory sitting next to a house. Doesn't work very well for a lot of reasons. If your television station was broadcasting from the top of the Sears Tower in Chicago, it would send a signal that could be seen for 200 miles. Right around the top of the Sears Tower would be a very strong signal, but at the edges it would be a very weak signal. If you had an adjacent phone that was using adjacent frequencies that was near the Sears Tower, it would be like somebody shouting very loud next to you. Nobody could see the signal. If you had the phone out at the 200-mile limit, then your telephone would block out the

television signal. These things just don't go next to each other. So, the sizes of the power that was used and the power limitations, the sizes of the areas that were being served were very different for data and for television. So, the upshot of all of that was that what we wanted to do was to clear a bunch of frequencies. We ended up clearing channels 38 to 51 of all television broadcast and then reusing re rejunking up that spectrum in a way that made sense for wireless data. And then, if you had a television broadcaster who was previously broadcasting on let's say channel 42 and didn't want to go off the air, you might move him to, you know,

channel 32. But, you had to do that in a way that didn't interfere with other television broadcast. So, it was a reallocation, and this is the first hard problem, how to do the reallocation. I guess I can describe for you roughly speaking, it's a graph coloring problem, and I'll I'll why. >> Actually, right before you do that, let me quickly flag this is a great illustration of why market design is necessary, right? Like here we have a setting where some level of like first-order, you know, you talked at the beginning about general equilibrium.

It's like, "Oh, there's some stations that have licenses. It's not the optimal allocation of the license relative to new wireless applications." Like sort of a naive answer would say like, "Okay, let the market trade." But here, you know, you've already given us constraints that tell us that that's not quite something a market can do easily on its own. Now you're going to tell us that it's going to involve solving an NPR problem. >> Yes, [laughter] right. That's right. >> Actually, if I can piggyback on what what Scott said, because again, like, you know, if you just read auction theory papers, it's like, "Oh, there's some number of items for sale or to procure." Right? And you don't really think about like what that might mean or even like one thing that's amazing about this story is I feel like a huge part of the problem that you solved was even

identifying what is being procured, right? So, you know, if a television broadcaster holds a license and is broadcasting on channel 46, you might think that their license gives them the right to be broadcasting in that area on channel 46. But for you to even get started, right, you had to actually clarify what the property rights of those licenses were. Is that right? >> That is absolutely right. By the way, Evan Kwerel at the FCC, the same guy who I worked with in 1993, deserves a lot of credit for this because he understood that if you had to acquire specific licenses, you'd have little local monopolists everywhere. If if all you need to acquire is a certain amount of spectrum in an area, and then you can

move everybody into whatever remains, then you could get competition going. So, that was understood by a couple guys at the FCC, and they encouraged Congress to write the law that way. But there was even more that was related to that because when we first started on this problem, it wasn't clear that you wanted to buy and hold television stations. It might be that well, there was a little bit of interference that would be created and maybe I could buy the right to interfere with 5% of your customers. Deciding what the product was was a big deal and we spent a lot of time on that. We decided that that was would be more complicated for the bidders and computationally much harder.

The data that was required was also very different. The way we decided what was feasible was as follows. One of the very first steps was a an interference study where the FCC knew where the 2200 television stations were in the US and Canada that would be engaged in all of this. And the combination of measurement and modeling worked out their signal strength that pixelated the US into large number of pixels that were 1 km square, looked at the signal strength at the center of each of these pixels, determined whether there'd be interference between two television who was currently serving that area and would there be interference. The eventual way it was done was was this.

They would round to whole numbers of percent. So, if you interfered with 0.4% of my customers, that was zero. The criterion was that I couldn't assign two stations to the same channel if they would interfere with any, meaning more than zero, of the which means more than 0.5% really, of the customers. And you know, in in the original thing when we were thinking about total interference, we had a Venn diagram where well, what if two stations interfered with the same area and looking at the sum of the interferences, but this was really computationally simpler because we would just say you either interfere or you don't. So, now

we have a graph coloring problem. Here's the problem. Each TV station represents a node in a graph and you draw an edge connecting two stations if they can't both be assigned to the same channel without creating interference. And now you're going to assign a channel to each station or a color to each node, if you will, so that no two connected nodes are the same color, so that no two connected stations are on the same channel. And uh, we have a given set of channels and given some set of stations, you can be asked, is it possible to assign channels to the stations so that there's no interference? And that was going to be the legal requirement that we had to

meet with. So, when we were looking at which sets of stations that we could buy, in the end the the auction in effect had considered about 10,000 different combinations of stations. And for each of them we had to solve this essentially graph coloring problem to determine is it possible to assign assign um, channels to the stations without creating interference. So, the first and probably biggest challenge was um, how do we do this because as you know, graph coloring, this is an NP-complete problem. There are no algorithms that at the scale we are working would solve this, if I understand correctly

formulated that way was something like 130,000 constraints. Uh, which is too big a number to stick it in an exponent of the uh, you know, so anyway, >> Little bit. Little bit too big. Little bit too big, yeah. >> Exactly. [laughter] Like By by a few zeros, yeah. >> So, we weren't going to be able to solve all of these problems, but I became convinced from Kevin's early work that he was going to be able to solve them well and that we could design an auction in which we could assume that 99% of the problems we would we would be able to answer. Um 1% we wouldn't be able to you know, the computer would time out.

That's what we were hoping for and that's pretty much what Kevin achieved with the uh software they developed. So then we needed an auction that worked in that setting where the computer might sometimes be unable to solve the problem, you know, so there was a little bit of that. Then the other big problem is that this was a two-sided auction. We didn't have money allocated by Congress to buy these stations. We had to then sell wireless broadband licenses that would raise enough revenue to cover the cost of buying the stations plus the stations that were moved, we had to cover the retuning costs.

There were uh certain other costs that we had to cover. So it was going to be about $7 billion worth of costs and and we needed the revenues from the uh sale would exceed the costs of the purchase. So we needed two linked auctions where the whole process would be budget balanced as well and that was also novel. You know, no way of knowing what what these things were going to cost, what the revenues would come in. Computations were incredibly hard. One of the premises that I used in in um guiding the design is that we didn't expect to be able to buy a large number of CBS affiliates, for example. We thought that, you know, the mom and pop stations, your your

grandfather founded a a channel in Topeka, Kansas in 1958 and um that those were the guys who were going to have the low-value uh stations that were going to be willing to sell. So we needed those guys to participate. So the auction had to be simple enough. You know, here we're doing these horrendously complicated calculations and the auction had to look simple to the participants. So that was a challenge, right? And and so we settled on this this clock auction. If you're a participant, here's how it looks to you.

You, Tim, by the way, have looked at your station and done some evaluation. You figure it's worth around $12 million. And I say, "Tim, um And I say, "Tim, um it's first round of the auction. We might be willing to pay you as much as $100 million for your station. Would you accept it?" >> I think I know the answer to this one. I know the answer. So >> [laughter] >> Very good. The idea was to make it obvious, right? So you would say, "You know, and and we say you can say yes or no." And here are the cuts. Here are the rules. If you say no, you're out and you don't sell your station and that's fine.

If you say yes and we offer you this $100 million, you have to sell. Then I go around and say, "Tim, you know, I I've looked and you know, Scott is in your area. He's also willing to sell and you know, I got a bunch of people who are happy to sell at my opening offer. So I'm only going to offer you the most I'm going to offer you is $95 million. Would you still be willing to sell?" And you can say yes or no. And if you know, you're out. If you say yes, we might offer you $95 million or you you know, or or we might come back to you with a lower offer. So those are the rules.

And the idea is for you, it's really easy. I'm offered $100 million, $95 million. Well, you know what your answer is. If I offer you less than your $12 million, you know what your answer is to to that, too. So this was an auction that as Shen Wu would have us say now, was obviously strategy proof. That is the choices you make are obvious at every point in time. And that's written down in a technical way that defines obviousness, but it's clear I think to your listeners why the these choices were were obvious. And the way the auction proceeded on the reverse auction side, let's fix for a moment the amount of spectrum we're trying to clear. What we do is we say, "Okay, we we're going to have a certain

a of channels that we're going to use for broadcast television. Then when I go to Tim, I I ask myself, "Hmm, given the promises that I've already made to Scott and others, um is it possible that if Tim says no, I have to give him a channel?" Is it possible that I can give him a channel? That involves solving this graph coloring problem to figure out whether it's possible. And um if the answer is no, then I say, "Okay, Tim, I offer you $95 million and that's what you're getting, $95 million. But if I have room for your station, then I'll say, 'No, I'm going to actually offer you $90 million now cuz you say no, I do have somewhere to put you.'"

So that's how the uh reverse auction worked with the fixed set of channels. But then we didn't know how many channels we were going to have because how many channels we could afford to clear depended on the revenue we would raise in the forward auction, the auction where people are buying. So we had this back and forth where we would lower prices to a certain point. We didn't actually say to you when the price got to 90 million that we were going to buy your station at 90 million. We're we're just not lowering your offer yet. Um and now if we raised enough money to pay all these things, then we would offer you the 90 million. But if we didn't raise enough money, then we'd say, "Okay, well, we can't afford to clear that many channels. So we now have more channels that are available and now

there's room to place you. So now we'll offer you we'll lower your price to 85 million." So from your point of view, the price can only go down from round to round. And the whole process is designed so that the number of channels we're trying to clear um can only go down during the course of the auction. And that means for the bidders in the forward auction, if they're selling some amount of spectrum and we decide we don't have that many channels to give to those guys anymore, we reduce the number of channels and we say, "Well, we've got some more competition in the auction now because we're not going to sell as many channels, so the prices per unit go up in the forward auction and they go down in the reverse auction until we reach a

point where there is enough revenue from the forward auction to cover what we need to pay as a result of the reverse auction and then the auction ends. And And anyway, that was the design. It turned out that the bidders did consider it easy. And I've got some more stories to tell you. Can I tell you some more stories? Because >> Always. >> Absolutely. >> Because this this is coming up these days for the next applications. We were told, "Look, you're going to have a really hard time getting to participate." I mean, think about the spectrum out there. You know, we got these religious broadcasters. They're not-for-profit broadcasters. They have a mission and they're not going to sell their stations. We've got public

television stations like here in the Bay Area, KQED. And, you know, they're not-for-profit. They are >> Excellent choice of call sign. >> Excellent choice of call sign, KQED. I somehow never even realized that, even though I grew up in the Bay Area with that station. Wow. >> I hadn't realized that either. There we go. It's God's station. After the deity. [laughter] Amazing over there. >> Amazing. >> It's God's station, absolutely. So, KQED had You know, they're never going to sell their stations, we were told. Well, haha. But, it turns out KQED had three broadcast stations and it was using rather old technology and it discovered that by upgrading its technology, it

could serve its entire population using two of the stations and it sold the third one for $95 million, which he added nicely to its endowment. And the highest-paid station in the auction was a religious broadcaster in Chicago that got half a billion dollars approximately for its station said it could save a lot of souls for a half a billion dollars. And besides getting cash, it also got a VHF station which the broadcast quality isn't quite as good.

But it stayed on the air with a VHF over the air station plus a half a billion dollars. Decided that was a good deal even though it was a religious broadcaster. So um you know, money talks. These are big sums. And we were able to, you know, clear 70 megahertz of spectrum this way which is a lot for, you know, modern data services. >> And in fact many of our listeners are literally using the spectrum you >> I was about to say that literally like on their phones or whatever to listen to this podcast. >> Yeah.

>> Let me add a quick footnote in there cuz there's a there's a little it's not slight of hand because you declared it but but the trick of how you managed to get a form of price discovery with these interference in the background is really subtle and clever. So I just want to like make sure that we highlight it. Like as you said you're going to lower the price when you have an opportunity to repack a given station. But if you can't find a repacking then then you don't. And that could be because you know, the graph coloring problem has no solution or just because the the the algorithms can't find it quickly enough.

>> Right. >> And so you're discovering prices accurately in a sense except you know, up to the small failure rate of the ability to find the colorings. And so everything if I understand correctly then everything actually like comes down to the better you can make that sort of interference constraint solver the the more accurately the market can clear. >> I've heard Kevin say that there may never have been a more direct relationship between the speed of an algorithm and that's like piles of cash. Right? Because if it times out it's literally making a higher offer perhaps than you strictly needed to >> Millions of dollars higher if it times out. That's right. It times It times out when the answer would have been yes. He figures that he saved a

couple of billion dollars for the government. But they you you don't have it quite right. Um >> Okay, good. Correct. Correct. >> There's several things I want to say at this point. One is First of all, this algorithm, if you think about it for a while, is a greedy algorithm. What we're actually doing is we're putting stations on the air which have the one by one which have the highest values, roughly speaking. And as long as there's room to put them. And if there's not room to put them, then we find somebody else that, you know, to put in those stations. So, it's a greedy algorithm. And it's a greedy algorithm it we're we're solving an NP-hard problem. The actual problem of optimizing the value of the stations is one near. So, so even if we could

solve all of these questions, all these things perfectly, we still don't expect to achieve the optimum. And we did a bunch of simulations to see how good it was. It's pretty good, actually. But the uh But you know, it could have been better. We did some work afterwards where we discovered ways we could have done that better. And the other thing I wanted to do because I I told you I gave you those examples for a reason. So, we were looking at other repurchase sorts of auctions, auctions where you're repurchasing the existing rights. So, right now, one of my former students is uh looking at repurchasing water rights around the Great Salt Lake, which is drying up because too much of the stream flow is being taken. There's an

environmental disaster pending in the Great Salt Lake unless they can reduce the water use by local farmers. That also has some complexities associated with it. But the people there say the auction's too complicated. Farmers don't want to bid in an auction. They won't participate. And these stories that I have from the FCC, I think are really quite helpful in a setting like that. You know, you don't know who's going to participate. Money talks. We'll start out with a high price and we'll pay whatever we have to and guess what? There're going to be people who participate. So, we're we're hoping to be able to take variations of what we did and put them to other uses.

>> As part of this, you touched on um how you couldn't be a a mere economist to uh design the FCC incentive auction. There was no choice but to sort of grapple with the computational intractability that was inherent Yeah. Um both in the repacking problem and then as you say, even given a perfect oracle for the repacking problem, there's this NP-hard welfare maximization sort of optimization problem on top of it. I think of you as a relatively early adopter among the economics community with respect to, you know, computer scientists and and sort of the techniques.

Having now a a decade-plus, I'd say, of experience with that community, do the two communities have things to learn from each other, do you think? I mean, either in terms of, you know, bringing theory to practice or perhaps even just sort of intellectually as far as the sort of strengths and and weaknesses of each of those communities, computer science and and economics? >> So, let me start with a compliment for you, Tim. Back then, you know, the the you were on leave at Columbia, I believe. I'd asked around about people to include on the team and I wanted people who both were really good at this as computer science and also really excellent teachers and uh the two names I came down to were you and Kevin.

And um I I tried calling over you and discovered that you were on leave. That's So, that's I ended up with Kevin, who was great, by the way. Kevin, terrific, but you would have been terrific, too, I'm sure. One of the things I found when I got into this was the simple, you know, clarity with which you and Kevin both Kevin taught me about sat solving algorithms while we were sitting in a hot tub in the Sierras, right? The you know, we >> [laughter] >> we had a retreat to you know, try and I was trying to figure out we hadn't settled on an auction algorithm yet because it depended on you know, what computations we could do and we were still talking about you know, whether to buy interference rights or whether to

take a station off the air. The products hadn't been decided up and hadn't been decided and I needed to know what you could compute. Kevin explained to me well explained to me a lot taught me a lot about algorithms. It became clear to me that many of the assumptions that we make in economics are you know, the arbitrary for convenience for simplicity and they're just wrong, right? You know, we have all these convexity assumptions that make a lot of problems look very simple and sometimes they are some and I started wondering well, when are problems simple? What's really going on here and a lot of economics is about decentralization that is about the we imagine that

information is spread throughout the economy and how do you get that stuff working? That's something that I think is something that both economists and and some group of computer scientists are interested in. In data analysis too, I know that the computer scientists learned a lot about causal inference from people like Guido and Bentsen others and Susan Athey who said yeah, yes, you're very good at forecasting but forecasting is not the same as causal inference. Um so there has been communication across that boundary. I think computer scientists have been great at absorbing a lot of these ideas from economics. I think economics has been less good at absorbing ideas from computer science.

Our frameworks are pretty rigid and fragile a lot of them and it's been hard to absorb um um The incentive auction was really an exception. >> I will say though, just almost all pairs of disciplines I've ever observed when they start overlapping in interests, they kind of tend to behave with a mixture of like territorial nature and fear, and I feel like it's been kind of the opposite of that with computer science and economics. It's It's literally one of the most, you know, as you say, Paul, I mean, we should aspire toward, you know, still more applications of the impact of the FCC incentive auction, but as far as just the exchange of ideas, I actually think it's been, for the most part, extremely constructive, and one of the more positive ones I've seen in my

career between two pairs of fields. >> Well, good. Again, I think that computer scientists have done a great job of take, you know, game theory right now in the Journal of Games and Economic Behavior, which used to be a a journal that was dominated by economic publications, it's now mostly computer scientists publishing in in yeah, game theory and mechanism design. One of the things that happens in economics, as you know, is that certain ideas get rather thoroughly examined, and we move on. There's very little research in general equilibrium theory anymore. Yeah, economists worked on game theory for a while, and it exhausted most of the economic ideas that were relevant to game theory, but computer scientists have these other ideas for how to approach game theory, and so game

theory, that area has is alive again with ideas that come from computer science rather than the ones that came, you know, in previous decades from economics. >> I have to say, Paul, I mean, when you made it clear um, you know, publicly that you thought there was sort of value in these collaborations and in the computer science ideas, I mean, just by nature of your stature, that had a huge positive effect in a larger group of economists taking notice. So, I mean, the fact that you were on board early, I think it's been very helpful on the computer science side. >> Well, good. Well, thank you.

>> And you've you've also been influential through your students. Like, so many people who have worked with you have gone on to work at the intersection of the two fields in in various ways. >> Yeah. >> You just mentioned causal inference. So like you know, all I've just claimed is an association, but I'll I'll bet it's a causal treatment effect just based on my own observational data. >> [laughter] >> Perfect. >> You did mention, you know, sort of ideas and new and and you know, you talked about water rights. You always have, you know, a dozen things that are super exciting that you're thinking about. Any previews and and slash, you know, sort of things that you would tell our you know, our listeners that they should go think about, too.

>> You might have noticed on LinkedIn that the 1 Chronos and Optix partnership. Did you see that? >> Yeah, of course. >> Uh so, you know, so 1 Chronos is using a combinatorial auctions in in security exchange. Relatively simple ones cuz they happen happen they have to happen very fast. One of the areas that we will be moving into with 1 Chronos and Optix is creating futures markets for compute. Here's the economic problem we're trying to solve, okay? So if you were to um to ask Nvidia, for example, what limits their sales. Everybody's buying as many of these chips as they can afford. When

you build a data center, if it costs you $20 billion to build a data center, somebody asked, "Well, why don't you build a $30 billion data center?" Say, "Well, because I can't get that much money." is what you say, right? And why can't you get that much money? Why can't you borrow it from the bank in an environment like this? Well, it's because it's very risky. Suppose that the bank were to lend you $10 billion and you were to build a bunch of systems and next year Nvidia were to come up with a chip that's 50% faster and uses 30% less electricity.

You know, >> Right. >> what is the security worth on what you've done? And so that makes it hard for a traditional for debt financing. But if you can spread this risk if you have a futures market on the let's say rental price of compute for example, if you have a futures market on that then then those risks can be hedged. And once those risks can be hedged, you can spread the risk more widely, you can support more debt financing, you can sell more chips. So there's you know the wide demand for this. So one of the things we're working on.

>> One last question, you know, forward-looking advice for people who are interested in doing market design in many forms and shapes and sizes. What would you What would you tell them? >> Oh goodness, you know, the future is so uncertain. What I would say to my undergraduates, I want them to be rigorously technically trained. Um I I can't tell them what knowledge they're going to need in particular, but uh setting high standards of evidence and logic in their thinking by studying a discipline that demands it, which there are several.

And then you know, there's some things that are just obviously going to grow in importance. Artificial intelligence, we don't know what it's going to be, but we know it's going to be important and even more important than it is now. Learning something about that would be things to do and then beyond that, I'm you know, you place your bets, you take your chances, but being flexible and being ready to adapt to whatever is coming, that that's what the students need to be doing. >> Yeah, that should be printed on a poster so that we can >> Yeah, exactly. I was going to say >> Awesome. [laughter] I think I'm going to run that by my five-year-old tonight and see if see if see if see if it sinks in. So yeah.

>> Okay. Well, great. Thank you guys. This has been wonderful. >> you so much for taking the time, Paul. Really appreciate it. Fantastic conversation. Wow. So um amazing conversation with Paul Milgrom. >> As expected, but still above expectations. >> [laughter] >> I do want to sort of talk about my maybe favorite two moments from that interview just to reflect on it. My second favorite moment, you know, we talked a lot about sort of theory and practice and so on the back and forth there, but at one point he talked about back and forth between mathematics and economics. And I was fascinated by the way he described that. His papers do exhibit a sort of mathematical elegance and a certain aesthetic, which is a little unusual, I'd say, uh, quite unusual in in economics papers. He said,

"You know, I think I understand the economics, and then the struggle is just how do I get the mathematics to articulate what I believe I understand about the economics." But then sometimes actually the mathematics winds up teaching you that you didn't have the economics quite right and refines your understanding of the economics. So, I thought that was my second favorite moment. I thought that was I thought that was beautiful. My top favorite moment has to be the new appreciation I have for the call letters of the PBS station I grew up listening to my entire life, KQED. >> [laughter] >> Totally amazing. So.

>> Look, I I can't I can't follow Paul Milgrom and then that. We'll just say QED. >> [laughter] >> All right. Always a pleasure, Scott. Thanks so much. >> Thank you. QED.