What We Actually Learn From Experience

Meet Anne Muro. She's been making early stage investments in Silicon Valley startups for 17 years. >> I guess the one that most people would recognize is Lyft, where I was the seed investor and led that seed uh from a $5 million valuation through when it went public. >> Lyft is now a global ride sharing company worth more than $6.5 billion as of early 2026. and's venture capital firm Floodgate Partners continues to make big bets on small companies. As their business has matured, the way they choose founders has evolved. >> Early on, I would be looking at their resume and I'd be looking at things that they done. But then a lot of times the people who are really successful hadn't done that much. >> Ann makes an educated guess at what founders might be capable of even if they're fresh out of high school. You're really evaluating the human behind that idea [music] and their capacity for learning and change and building things quickly. >> Founders know they need to learn from their mistakes and an is not that concerned when her investments fall short. >> There's the kinds of mistakes where you you invested in the wrong company. But in venture, you can only lose the dollars that you put in. You can't lose more than that. Versus when you fail to make the right decision, that's where you really lose out on the 100,000x. >> It's the big successes they don't invest in that keep VCs up at night. So, how do they learn in a way that avoids missing the unicorns? When we think about learning, we're always looking at what are the hundred baggers, which are the companies that net 100x on your seed stage investment. We'll look at other [music] companies outside of our portfolio that net that. We look at companies that are anywhere from 25 to 100x within our portfolio and we're trying to seek to figure out what are the patterns within those that are repeatable. It's a hyperco competitive industry that tries to discover entire new categories before the rest of the world catches on. >> I'll give you one example that I really love which is uh there's this philosophy book around scientific revolutions Thomas wrote and it was written quite a while ago but the theory within it can actually be quite relevant to how inflections happen in our world. How do we identify those things and then how do we map that to the current world of AI? >> It's more than simple common sense. Trying to tease out what exactly we learn from experience is an emerging research field known as correlated learning. Here in Silicon Valley, the expression that you learn from failure is very widespread and very intuitive. But the question is sort of what do you learn? How do you optimally learn from that experience? >> Today we're going to talk about [music] correlated learning and how it can help us make better decisions. We're speaking with Steve Calendarer, professor of public and private management at Stanford Graduate School of Business. [music] This is if then from Stanford GSB where we sit down with faculty and explore how their research deepens our understanding of business and leadership. I'm your host Kevin Coul. Steve is a self-described theorist. That sounds like pretty abstract work, but the models he develops can be applied to very concrete matters like coming up with a business strategy, choosing your next job, or negotiating with your mechanic. We talked to Steve about what we can learn about learning and how much we should trust experts. But we started by asking him why theorists are important. >> So there's an enormous amount of data in the world. There's facts coming up at us from left and right. And so the theorist role is to make sense of that data, make sense of those facts and put some structure on our understanding of the world. And the idea is that by putting this structure on these facts, we can not only understand the world a little better, right, but we can make predictions and suggestions about how we can make the world work better. That's the role of a theorist. >> Mhm. And in a lot of your work, you are trying to presumably translate pretty sophisticated and complicated, you know, mathematical and statistical models. If you're doing that for a lay audience, how do you do that? >> Well, that's one reason why I love being in a business school where I am forced forced myself to stand up in front of an audience of very smart but lay people and explain to them why this knowledge is useful, how it helps us understand the world better, and what we can do with it. And I love that challenge. >> Well, I'm glad to hear that you love doing that because that's what we're going to do today. And we're going to start with a paper that you did the title of which is learning in a correlated world. And it's really discussing the concept of correlated learning which is a term I'm not familiar with. What is correlated learning? So correlated learning is all around us. It's in every aspect of life. What it means is that the outcomes of different choices are correlated. That's a fancy mathematical statistical word. But what it means in practice is that what happens if you make one choice is connected, related, correlated with the outcome you get from other choices. And so what that means is we can learn from our experience. We can learn across alternatives. What you did today may have worked, it may not have worked, but it tells you something about what other choices, what other alternatives might work. And so what I do is try to study that connection, those correlations, put some mathematical structure on it to both help us understand correlated learning in practice in a variety of domains, but then help us figure out what we should do about it. How can we learn better? How can we structure our choices better [music] to make better choices? How can we design markets? How can we design political institutions in a way that allow them to learn better? What are some real world situations that this research might apply to? I >> I'll give you two examples. One is just an experience I think all of us have had which is searching for a job and learning from our experience from our past jobs. So an example I love to give is a fresh graduate comes out of college and takes a job at Goldman Sachs and they hate it. They hate the long hours. They don't like the work. They don't like that their boss yells at them occasionally and then they have to decide what to do next. If there was no correlation, they would just pick another job randomly and try that and see if that works. >> Now, when you say correlation, what are you referring to? >> In this case, it's whether this person likes these jobs or not, whether these jobs are a good fit for them. >> Okay. >> And so, to be a bit more concrete about it, if you don't like your job at Goldman Sachs, there's not much chance you're going to like your job down the street at Morgan Stanley. Sure, these financial institutions are a bit different, maybe a little idiosyncratic, but the job is going to be largely similar. >> The considerations that make that person not like the job aren't going to be different, >> aren't going to be radically different. No, the hours are going to be long, the work's going to be similar, the culture of the institution, the relationship with your boss is going to be pretty similar. And so, if you didn't like Goldman Sachs, it doesn't make much sense to go work at at Morgan Stanley, right? So, maybe you branch out a little bit. Maybe you got a hedge fund. It's going to be a bit different, right? But still pretty similar. So there the correlation isn't as high, but it's still very, very strong versus taking a job as a financial planner in Ohio that's still in finance, but it's a very different job from working on Wall Street. And so there the correlation is going to be a lot lower. And so what my modeling is trying to do is to help us understand that choice. What would be the optimal second job for our college grad to take? How do they think through these choices? and how do they strategize long-term over those choices? >> So what does the model do that leads us to that conclusion if that's the right word or that insight? >> Yeah. So the idea is to put some mathematical structure on this. And so the way we do it is we represent uh the set of alternatives in a space and this is going to get very mathematical but then there's going to be some function or some mapping from those alternatives to outcomes. So how far apart these jobs are in the space indicates how correlated their outcomes are going to be. And so we can think should you search for a job that's nearby or should you jump a long way to something very different that gives you the chance of maybe finding a job you're going to like but it itself is is sort of high risk. You really haven't learned much about the likely outcome of that job about how well that job fits to you and your characteristics. Help us understand then how those potential outcomes or possibilities opportunities are expressed in the model that would be useful for someone to take that and say hm okay I never thought about it that way. >> The mathematical answer to that is that the way I model this function or this mapping from the space of jobs the space of alternatives to outcomes is to use some mathematics that's very popular in finance which is the brownian motion. Brownian motion. Okay, what is that? >> Brownieian motion goes all the way back to a botnist 100 plus years ago who described the movement of atoms as representing a almost random process. And then 100 years ago, Albert Einstein developed the mathematical theory of a brownie in motion. So what it really is is it's a it's a description of a random walk, a process that's just wandering through the space. So what it means is because it's continuous that alternatives or things that are close together in the space have outcomes that are close by whereas [music] alternatives that are far apart in the space are going to have outcomes that are much further apart. Doesn't [music] mean they're the same, but it does mean that in expectation things that are close together are going to produce nearby outcomes. I think there's a very intuitive notion that pops up in a lot of places. the idea that nearby alternatives, nearby choices should produce [music] nearby outcomes. And by representing this function from alternatives to outcomes, [music] it captures that eternal truth or that essence [music] of decision-m under uncertainty. And it does it in a very elegant mathematical way that allows us to have [music] some structure on the problem that we can then analyze the model with this structure and say [music] something concrete. So does Brownie in motion help us corral to some degree that randomness and that uncertainty and make some sense of it? >> Absolutely. And to get again mathematical here the fact that it's again a continuous function means that if you learn the outcome of one alternative you learn one point in this mapping then you know that this mapping goes through that point and so it's going to tell you that alternatives that are nearby are going to have outcomes that are nearby. So if you think about this in the example of our college grad who worked at Goldman Sachs, he learned the outcome of that job. He knows that value. Morgan Stanley, which is very close to Goldman Sachs in the space, is going to have an outcome that's very similar or very close to Goldman Sachs. It really isn't going to be that different. It's not exactly the same, and that's the beauty of the random walk. Every alternative produces a different outcome. But this captures this sense that nearby alternatives produce nearby outcomes. And so that allows our job seeker to search. If he loved Goldman Sachs, then he'd expect a job at Morgan Stanley, you know, might be better, and he's going to love that as well. But if he hated it, then taking the job at at Morgan Stanley isn't going to be a very wise choice. He needs to jump to a job that's far apart in this space so that he can expect or hope for an outcome, a fit of that job to him that is very different and hopefully on the upside. >> Steve, is this to some degree also a mindset shift? Like what's different about this? >> I do agree. It's a mindset issue. So, it's something we all do. We all learn from our experience here in Silicon Valley. the the expression that you learn from failure is very widespread and very intuitive. And so it's a concept that's very familiar in that way. But the question is sort of what do you learn? How do you optimally learn from that experience? And so what I'm trying to add with my framework is by formalizing this idea that you can learn from your experience. Take something very intuitive, something we all believe, formalizing it through theory. So we're taking this sort of intuition that sort of exists in the ether but you know we're all sort of doing it formalizes it in a in a mathematical in an economic structure and then by building that structure we can begin to analyze what's the optimal strategy for an individual or a firm to pursue how is market competition going to play out in a world where learning across products across strategies is correlated so that's what I think is new that's what I that's what I'm trying to add >> and it turns out that you know this is very difficult to capture formally. It's not that I think economists and theorists weren't aware that correlated learning was important but it's just very very difficult to capture it formally. Well, I actually loved your very straightforward answer about what did you learn because to some degree that's an abstract notion and it reminds me of a conversation we had with Deborah Grunfeld, one of the associate deans at the GSB and she said the thing that she admires about so much research at the GSB is that it makes the invisible visible. And this seems to me [music] like one of those examples like you're taking something to use your term that's in the ether and trying to make it concrete in some fashion. Right. >> Oh yeah. I like that phrasing. Yeah. Exactly right. I think that's what we do. We make the theorists make the invisible visible. And then once we can see it clearly, we can start to sort of extract actionable insights from it. [music] >> We'll be back in a moment with more from my conversation with Steve Calendarer. We'll talk about how to evaluate experts and how correlated learning informs business strategy. [music] So your paper also talks about the trust that we extend to experts and that can be the examples you use. It could be on an executive board of some kind. It could be your local auto mechanic. Those are two very different sort of situations. What's common about them with respect to your research? >> Excellent. Yeah. So this is another vein of work that I've pushed this this model of correlated learning. what role experts play in decision-m and they're everywhere in our life. You go see a doctor, you're going to the doctor for expert advice. They have more information than than you have and they're providing you recommendations about what you should do. You go to the mechanic, the mechanic knows more than you do, or at least [laughter] she knows more than I do about cars, that's for sure. I can barely put air in the tires of my car. and I am trying to learn from uh her expertise but at the same time whether it's the doctor whether it's the mechanic I'm not so sure I I see our interests as being perfectly aligned and so there's an element of trust or [music] a doubt there as to whether they're giving advice that's in my interest or whether it's advice in their interests and so they're examples from everyday life but they come up in professional life as well you're a CEO of a firm you've got division managers those division managers know more about what's happening in their division than you do. You are trying to extract that information when you ask them for advice and recommendations [music] about what the firm should do. But you have some doubts. Are they acting in the firm's interest or are they acting in the interest of their division or their own career? The board has a similar problem with the CEO. One thing I teach about is crisis management. And I'm yet to encounter a CEO who has recommended to the board that they be fired as a result of the crisis that erupts. And so the board has a problem. the CEO has better information than they do, but the CEO has his or her own interest [music] at heart. And so this creates an enormous sort of trust problem. How can you as a decision maker get the information you need to make a good decision when it's possessed by an expert who doesn't have exactly the same interests as you? And so what we do is study this problem in the context of of correlated learning. So you're trying to make a decision. You're trying to find a good alternative. uh whether it be medical care, car repairs, whether to acquire a new division with your firm, whether to fire the CEO in a crisis or not, but the expert has their own interest. [music] And so what this sort of gets at is where does the expert get their power from? How does the expert provide that advice in a way [music] uh that sort of serves their interest, but at the same time reveals enough information for the decision maker to make a better decision? And what correlated learning adds to this problem is well when they tell me something they're revealing some information. >> Okay. So let's just bring some specificity to that because I think I'm following here. But so let's just unpack this and say the mechanic says you need to have an engine overhaul. Right? This this engine is shot. You need an engine overhaul. So you can accept that that is the repair that needs to be done or not. But if even if you accept it, right, then it's what's the price? Is there somebody else who would do a better job? There are all these other variables, right, that go in to what your model is. >> Yeah. Yeah. Absolutely. So suppose your car is creaking and it's breaking down occasionally. You know, it needs some repair. So you go to the mechanic. The mechanic comes back and says, "You need a complete engine overhaul or replacement. That will solve your problem. That will also give you a very big bill." >> Mhm. >> What do you do with that information? Well, you share some common interest with the mechanic. Both of you want the car to be fixed, but where you don't share a common interest is in how expensive the repair is. The mechanic has financial incentive to have a higher bill and you have an incentive to to not spend as much as you need to. So, do you accept that set of repairs? So, what do you do? Well, what you have learned is that you one don't need to change the brakes. You don't need to change the tires. The electrics are all working fine. So, you have learned something from the fact that the mechanic has told you what she thinks is the best repair for her. And so, then you've got some information. You've narrowed down where the problem is in your car. So, that tells me that maybe just changing the pistons, yeah, maybe even changing the oil, that's about all I can do in a car. Maybe that's enough of a repair. It's going to be a lot cheaper and it's enough of a repair to make my car work. >> And so, then I might choose that. And so that's where this correlated learning comes in. The fact that the mechanic has told me and revealed what she thinks is the optimal repair given her interest tells me something about the likely effectiveness of other repairs. And so that might mean I choose something else. But of course that feeds back into the mechanic's advice. She's thinking, well, if I tell him this, he's surely going to just change the pistons. And so her problem is what advice or what recommendation can I provide that will serve her interest but at the same time be a set of recommendations that the car owner will accept. You might think this is overanalysis of taking your car to the to the mechanic. But the idea is that this problem is all around us. It's in all aspects of our life. We are at a business school and if we think about this in a business context or someone who is a business leader an executive say >> how does knowing something about correlated learning help them at some level with decision- making understanding risk whatever that might be. Yeah, this correlated learning is super relevant to to businesses and business strategy like how should which market should you enter, what product should you develop, how should you structure your business and organize it. This is a a learning problem. It's a search problem. You're trying to find the optimal mix of elements that go into making a productive business. Learning is correlated. You try one market, you try one product, you try one organizational structure. It's going to teach you something about whether it works or not. And it's going to teach you about what other structures are most likely to work. To put it in more concrete terms, a very popular phrase in uh in startup land out here in Silicon Valley, but it's relevant to everyone in business around the world is this idea of product market fit. You know, what does product market fit means? It means you're searching for a product in the space of products and a market to compete in the space of markets where there's a real magic, a real synergy between the product you have and the market you're competing in that lead to magic happening. And that's what you're looking for. And so you're searching over both of these spaces. >> You can't search forever. Choosing randomly is unlikely to find anything good. So you want to think about that search in a structured systematic way and this is what uh correlated learning and this framework and these tools I'm developing is intended to help people decide when should you listen to the engineers when should you listen to the sales team like what is more important and how do you weigh up balance out the market and the product considerations to find that fit and that's really the job of a leader and they're doing it they're trying to solve and coordinate this learning problem in a correlated world. >> To what degree do you think this way of thinking is understood or even known? >> Yeah. Not at all really to be honest. Uh it's very intuitive the idea that you're you learning from your experience. But then that's the extent of it. There's no structured >> right tools. What did I learn? >> Yeah. What did I learn? What exactly does it tell me? How close are these alternatives in the space? like how does that affect my level of uncertainty about these other alternatives? I think it's just not known like so it's intuitive and again this is the value of contribution of theory. It's an intuitive notion that is entirely believable and we all do to some degree but it's not structured. We're not doing it well enough and so these tools help us understand this theoretical framework gives us the mindset to to do this systematically in a much better way. Every firm I see in Silicon Valley is talking about product market fit. But are they doing it systematically? I'm yet to hear of one that does it in a systematic way. >> This research is what you're spending most of your time on these days. >> Why this in particular compared to other things that you have done or could be doing >> because it fascinates me and I can't stop thinking about it. That's probably the true answer. But >> I think there there are other things I still work on other papers and projects I work on. I don't think many academics are super strategic and deliberate about what they do. We tend to follow our interest, but I do think about where is my marginal contribution going to be the greatest. >> I have something unique, new and valuable to say and I feel like I'm making progress and I'm loving doing it and and this paper we're talking about, this survey of research that on this framework I've introduced and developed and that I've done and others have done uh has been a lot of fun to write and is really encouraging that we're making progress. If Then is a podcast from Stanford Graduate School of Business. I'm your host, Kevin Coul. Our show is written and produced by Making Room and the content and design team at the GSB. Our managing producers are Michael McDow and Elizabeth Wizik Stern. Executive producers are Sorrel Husbands Denholtz and Jim Kan. sound design and additional production support by Mumble Media and H Ash. And a special thanks to an Mirao, partner at Floodgate Ventures. For more on our faculty and their research, find Stanford GSB online at gsb.stanford.edu or on social media at Stanford GSB. Thanks for listening. We'll be back with another episode soon.

Steven Callander has spent years building a mathematical framework to answer the question of how people learn from experience. “Here in Silicon Valley, the expression that you learn from failure is very widespread and very intuitive. But the question is… what do you learn? How do you optimally learn from that experience?” In this episode, Callander, the Herbert Hoover Professor of Public and Private Management and Professor of Political Economy at Stanford Graduate School of Business, explains the hidden, deceptively simple logic of correlated learning — and it may change how you think about finding the right job, the right market, or the right strategy. “It fascinates me and I can't stop thinking about it,” he says. Has theory made an impact on your life? Tell us more at ifthenpod@stanford.edu. Related Content: - Steven Callander faculty profile: https://www.gsb.stanford.edu/faculty-research/faculty/steven-callander - How to Turn Old Ideas Into Creative Solutions to Modern Problems: https://www.gsb.stanford.edu/insights/how-turn-old-ideas-creative-solutions-modern-problems - What We’re Still Learning from Silicon Valley’s Bank Collapse: https://www.gsb.stanford.edu/insights/were-still-learning-silicon-valley-banks-collapse Chapters: 00:00 Ann Miura-Ko on learning and the search for patterns in Venture capital 02:51 Introduction 05:23 What is correlated learning? 06:40 Where does this research apply in the real world? 09:28 Brownian Motion 12:45 Steven Callander’s Framework 15:25 Examples of correlated learning when seeking expert advice 20:53 Applying correlated learning 23:57 Why correlated learning research? 24:51 Conclusion If/Then, from Stanford GSB, features conversations with faculty that explore how their research deepens our understanding of business and leadership. Find out more about If/Then: https://www.gsb.stanford.edu/business-podcasts/if-then Listen on: 🔊 Apple Podcasts: https://podcasts.apple.com/us/podcast/if-then/id1725380194 🔊 Spotify: https://open.spotify.com/show/1v7V6LGUxfplMByTpVwk7h?si=38f353685fec4dd5 #gsbifthen #gsbpodcasts