The Explore-Exploit Dilemma: When to Try Something New vs. Stick With What Works

Transcript

Good afternoon everyone. Uh, thanks so much for having me here. My name is Alex Hutchinson. I'm a a science journalist. At least that's what I call myself these days. I've actually had a bit of a a checkered past. I started out as a as an athlete. I was a long-distance runner with the Canadian national team. Then I became a physicist. Uh, I I got a PhD. I I was a posttock with the NSA's quantum computing group in Maryland for a few years. Then I switched to journalism. I was focused on writing about the science of human performance. I wrote a book called Endure which is about the what defines the limits of endurance. Then about five years ago I made another pivot and I got really interested in the science of exploring. What is it that draws us to the unknown? What do we get

out of it? What can we do better? So, I wrote a book called The Explorer's Gene, which explores some tries to answer these questions, but also was kind of an attempt for me to understand my own zigzagging path. Why I'm always eager to dive into something new rather than sticking with the path that I'm already on. So, what I want to do today is introduce one of the key concepts from that book, which is the explore exploit dilemma. I want to get into what it is, what it tells us about the way our brains work and why it's a useful framework for thinking about decisions when your goal is to create something genuinely new. So, what is the explore exploit dilemma? If you if you read a scientific paper on the topic, they all start with the same example, which is that you're going to a restaurant. It's

a restaurant you've been to before, and you have to decide, should I order the burger I had last week, which was pretty good, or should I try something different? Should I maybe order the special, which might be the chef's masterpiece, or it might be last week's fish that they're trying to get rid of? So, we face these decisions all the time between sticking with what we know or trying something new. It could be, are you going to propose to your current partner, or are you going to keep swiping right and looking for a better option? Or are you going to invest your resources in trying to market the company, the product that made your company successful, or are you going to devote those resources to R&Ds to try and figure out what the next product is?

Now, I've given a couple talks recently to big insurance companies, and my role there is to go in and basically grab people by the scruff of the neck, shake them, and say, "You need to explore more. You can't just keep doing what you were doing yesterday and expect it to work in today's world or tomorrow's world." I think in this room, the the dynamic is a little bit different. And in fact, it may be that the advice the best advice I can give to some of you is you need to explore less. you need to stop coming up with new ideas and you know execute on the ideas you already have if you want to make it to a successful exit. But where wherever that balance lies the point is that the explore exploit dilemma is genuinely a dilemma. It's not that exploring is always the right call. It's not that

exploiting is always the right call. You need to figure out based on the context which is the right one. So how do we decide which is the right one? There there are two different ways we can think about this question. One is to look at the math of exploration. say what is it that's going to maximize the odds of a good outcome and that's obviously an important way of thinking about it but it's not the only way you can also ask about the psychology of explore exploit decisions which of these options is going to feel more appealing is going to be more engaging is going to feel more more meaningful more purposeful and so for people who founded a company I would bet that the choice wasn't just a cold mathematical calculation of what's going to maximize my chances of getting rich it's it's about the feeling of engaging

in a quest that has meaning and purpose. So we need to think about both of these. Now they do have a link. The math and the psychology are linked by evolution. People who were drawn to figure out what's over the next tail or to find new ways of getting food or resources. They were more likely to pass on their genes. And those genes are still with us today. There's a there's a dopamine receptor gene called DRD4 which influences exploratory behavior. played a key role in when how you humans migrated out of Africa and spread across the world starting about 50,000 years ago and it's still active today. I would bet that in this room there's a relatively high percentage of people with the seven repeat alil, the explorers version of the DRD4 gene. But

anyway, so the the math and psychology are both important. They generally point in the same direction but not always. So we need to think about them separately. So let's start with the math. to think about the math of explorers decisions. One way to to look at it is to think of a basically a gambling game. You walk into a casino, there's a bunch of slot machines, one-armed bandits. Each of them has a characteristic probability of paying off, but you don't know what that probability is. The only way to find out is to play each machine. So, you go in, you you try each machine a few times, you might conclude after a while that the machine on the left seems to pay off about 60% of the time, plus or minus 5%.

The one in the middle also 60% but with a a bigger error bar. And the one on the right maybe it seems worse. It's only about 50% of the time but you haven't played it very much. So the error bar is much wider. So the question is which of these machines should you put your money into? And the answer is it depends. There's no magic formula. It depends on the context. If you have two coins in your pocket, you should probably play the machine on the left, the one with the the shest bet. If you have a 100 coins in your pocket, you should explore all three machines until you're really sure which is the best machine. So time horizon matters. Volatility also matters. If the probability of each machine paying out is drifting up and down over time, that favors more exploration. You have to keep checking

each each of your options rather than just concluding you know what's best. And information also matters. I would argue I argue that we're drawn to uncertainty, but we're not drawn to uncertainty just for its own sake. If I flip a coin and ask you like, "Hey, do you want to know whether it came out heads or tails?" You'd probably say, "No, not really. It's not interesting because resolving that uncertainty doesn't teach you anything about what the result of the next coin toss would be." We're interested in uncertainty as a marker of the possibility to learn about the world. So, if we look at these error bars here, they're they're telling us how little we know about each op each option. But another way to look at it is that they're telling us how much potential there is to learn something new.

And so we should be attracted to the opportunity to learn something new. And so one way of operationalizing that is to add an uncertainty bonus to our equations. To to tilt the scale in favor of the options we know the least about. And if you work through the math, including an uncertainty bonus in your equation optimizes your chances of a good outcome. A and what's also cool is if you watch people in the lab or in the real world, we instinctively seem to add an uncertainty bonus to our equations. So, or to our decisions. So, if you look at a quick case study, 1.6 million orders from a food delivery company. As I said before, ordering food is like the canonical example of the explore exploit dilemma. So, if you crunch this data,

you come up with some fairly obvious conclusions. We we tend to like restaurants that are inexpensive, that deliver quickly, that have high ratings. Not surprising. But there are also some more surprising insights. We tend to like restaurants that are less explored. So if you have two restaurants that have similar price, deliver similar delivery time, similar ratings, and one of them's been rated five times and the other's been rated 500 times, we're more likely to order from the one that's only been rated five times, which is counterintuitive to me. I don't know that I would have guessed that. But the idea here is that the restaurant that's been rated 500 times, we know it's a 4.1 star restaurant or whatever. There's nothing to learn. But the one that's only been rated five times could be the greatest restaurant in the world and

people just haven't discovered yet. So this is the uncertainty bonus in action. Couple other insights that are worth taking from this restaurant data. One is that exploration has a cost. If you order from a restaurant that you've never ordered from before, chances are you're going to get a worse meal than if you order from one of your favorites. you're less likely to rate that meal as a five-star meal, but in the long term, if you keep exploring, you will get better and better results. So, you order from new restaurants. If it's a crappy meal, you don't order from that restaurant again. If it's a good meal, you add that to your stable of standard restaurants, and over time, you get a better and better roster of restaurants to order from. So, this is actually an an instance of a much broader truth about exploration, which is that if you

look at it in the short term or as a one-shot deal, exploration often looks like a bad call, looks like the wrong decision. But in the long term over multiple shots, exploration is what produces the best outcomes. Now, there's one deeper level of mass that we can get to. I mentioned like dopamine receptors before that explains on a superficial level why we explore. But why are we wired this way? So the grand theory of the brain right now in in cognitive neuroscience is something called predictive processing. And this is the idea that we're not just passive like video cameras looking around and taking in information from our senses. The brain's role is to simulate and predict the world and it's always trying to predict what's going to happen and it's using its senses just to

uh update its predictions and compare. And so the first rule of predictive processing is that we want to minimize surprise. That is, we want our predictions to be so good that there's as little gap as possible between what we think's happening and what's actually happening. And the problem with that commandment is that it suggests we should hate exploring. It suggests that if we want to be able to optimally predict what's going on around us, we should head into a closet, close the door, lock it, turn out the lights. Now, we have perfect predictions about what's going on in the world. Absolutely nothing. And this is called the dark room paradox. And it kind of royd the waters of cognitive neuroscience about a decade ago.

And the resolution turns out to be that we don't just want to optimize our predictions of what's happening right now. We want to optimize our predictions about what's going to happen in the future. In order to thrive, we need to be be able to anticipate what's going to happen. And that dictates that we should open the closet door, head out into the world, and learn as much as possible about it. And in fact, if you if you follow the math through, you find that this principle, the idea that we want to predict not just the present, but the future, ends up dictating that we should seek out the options that we know the least about or we should be preferentially inclined to seek those out. So the math of of predictive processing leads right back to this idea of an uncertainty bonus. Now, predictive processing also has something to say

about the the psychology of exploration. So if we go back to a guy named Wilhelm Vunt, he was one of the f fathers of modern psychology back in the 1800s and he posited a general relationship between basically between fun and effort. And it looks like this. So if something is really on the left side of the graph, if something is really simple, obvious, predictable, uh it's not really that much fun. It's not really that interesting to us. On the right side of the graph, if something is totally chaotic, very difficult, very uncertain, that's even less fun. But there's a sweet spot in the middle of the graph. And what predictive processing says is that this sweet spot corresponds to the intermediate level of uncertainty that maximizes our ability to learn about the world. And so our brains are wired to prefer this optimal

type of exploring where we can learn about the world. And in fact, it it's wired so deeply that even little babies can find it. You can do studies with eight-month-old infants and if you show them a sequence of shapes, if it's the same shape over and over again, they get bored and they look away quickly. If it's a totally random sequence of shapes, they get bored and they look away quickly. But if there's some sort of pattern in that sequence, if there's some potential to detect regularity in the world, that keep their gaze is is maintained for longer even at 8 months. So we're wired to to tune into these opportunities to learn about the world.

And the other thing to note about this this curve is that the interesting uh opportunities, they're not the easiest opportunities. They require effort. And this relates to another concept in psychology called the effort paradox, which is the idea that there are things that we value not in spite of the fact that they're hard, but because they're hard. We climb mountains. We run marathons. We order coffee tables from IKEA. We found companies. All these things are hard. And the difficulty is part of what gives them meaning in the pursuit.

So if we go back to this idea of the math and the psychology, obviously we want both. We want good outcomes and we want to feel like we're engaged in a meaningful task, a meaningful quest. So how do we reconcile these two drives? There's no magic formula that will always guarantee you make the right choice. But there is in the in the research there's a heristic that I think is worth keeping in mind and that is that you should be optimistic in the face of uncertainty. And this is actually just a restatement of an idea called the upper confidencebound algorithm. So if we go back to this uh gambling scenario, the upper confidence bound is just basically the top of the error bars.

So it's the best realistic upside we could hope from each option we're considering. And what the upper confidence bound algorithm says is that you should take the option with the highest upper confidence bound. So in this case, the option on the right. And the math then says this is basically an iterate or an instantiation of the uncertainty bonus. So the the math says that yes, you will maximize your chances of a good outcome by being optimistic in the face of uncertainty. But it also says that you will minimize your accumulated regret. Regret here is another mathematical quantity. It's basically the difference between how you hoped things would turn out and how they actually turn out. There will always be some regret, but if you're optimistic in the face of uncertainty, you'll maximize

your chances of a good outcome and minimize your regret, which I think is uh good guidance for building a company, but also for living life. So, I'll leave it there and uh thank you very much for your attention.