Avi Goldfarb on Prediction Machines, O-Ring Tasks, and How AI is Reshaping Economics

This week, we’re joined by Avi Goldfarb, one of the leading economists of artificial intelligence and co-author of Prediction Machines. Avi has been thinking seriously about AI economics long before the ChatGPT shock, so we asked him what he thinks the earlier framework got right, what it missed, and how economists should update their beliefs now.The conversation starts with Avi’s seminal book, Prediction Machines, and the idea that AI is best understood as a drop in the cost of prediction, which is a complement to judgement. We ask what that book got right and what it got wrong. From there, we interrogate Avi on the murky boundary between prediction and judgment. We had investigated the idea that maybe judgment and prediction were not as separable as economists like to believe in our episode with Alex Imas. We also ask whether, if AI gets better at predicting human judgment, whether judgment disappears, or do humans simply “move up the stack”? And what is taste exactly? Avi says that sometimes judgment becomes predictable, but humans still matter because goals, values, organizational politics, and “what matters” are often implicit, unstable, and hard to codify. Avi shoots down Seth’s galaxy-brain suggestion that correct ontology choice — i.e., deciding what sort of natural kind a thing is, or understanding when a problem is out of context — is a uniquely separate skill (taste?), calling it just another prediction error. But he does concede that deciding how much to prepare for ‘Black Swan’ events may be an enduring role for judgment. We then revisit the O-ring theory of production and what it means for automation. We had covered Kremer’s article in a recent episode (see here) and asked Avi about his new paper, riffing on the idea at the worker level. Avi says that if tasks inside jobs are complements rather than substitutes, then automating one task may make the remaining human tasks more valuable, not less. Avi explains why workers may reallocate attention toward the tasks machines cannot yet perform (shooting down Seth’s suggestion that this is actually difficult in most jobs).The discussion also covers whether AI will augment or replace workers, whether governments should try to steer AI toward human-complementing technologies, and why that distinction may be much harder to define in practice than it sounds. Avi agrees with Andrey and Seth’s pushback on “augmentation good, automation bad” framings (e.g. friend of the show Erik Brynjolfsson’s “Turing Trap”).Then we get into forecasts: how fast AI capabilities might advance by 2030, what that means for GDP growth by 2050, whether GDP is still the right thing to forecast, and why even very powerful AI may run into bottlenecks in the real economy. We use the paper Forecasting the Economic Effects of AI to ground the discussion. We close with lightning-round topics including AI’s impact on centralization, privacy/de-anonymization, peer review, and whether academic journals still serve the function they once did.Papers, books, and ideas mentioned* Avi Goldfarb’s seminal book with Ajay Agrawal, and Joshua Gans — Prediction Machines* A black swan is the occurrence of a wildly unpredictable event, which Nassim Taleb argues, in his book by the same name, is more common than we like to think* A New Riddle of Induction — by Nelson Goodman — is the source of Seth’s thought experiment about “bleen”, a color which is green until 2029 and blue after, and green* Michael Kremer — “The O-Ring Theory of Economic Development”, covered in this episode of the pod: * Daron Acemoglu and Pascual Restrepo’s task-based models of automation, especially “The Race Between Man and Machine.”* Avi mentions David Autor and Ben Thompson on automation and skill scarcity when Seth comments that you may not be able to reallocate effort between tasks as a worker, including their paper “Expertise”* Erik Brynjolfsson in the “Turing Trap” argues that automation technologies are less good than augmenting technology* Eric Topol’s book on AI in medicine — Deep Medicine* John Markoff — Machines of Loving Grace — The source of a title for an influential essay of the same name by Dario of Anthropic. Both draw from an earlier poem about a Sci Fi utopia: https://allpoetry.com/All-Watched-Over-By-Machines-Of-Loving-Grace * Korinek and Stiglitz on AI, capital, and taxation; Lockwood and Korinek on optimal taxation and automation — We covered these topics at the end of our episode with Basil Halperin in the context of “Tax Policy at the End of History” around the 1:19:00 mark* We talk about de-anonymization, and Avi references this provocative paper from Florian Ederer * Avi brings up Bob Gordon, and his argument, famously in the book The Rise and Fall of American Growth, that the early 20th century was incredibly important for increases in US living standards, which digital technologies have not lived up to* Digital Hermits, by Jeanine Miklós-Thal, Avi Goldfarb, Avery M. Haviv & Catherine Tucker, is a paper by Avi thinking about how information spillovers, now from AI, drive some people to be more private than they would otherwise be. In our conversation, we speculate AI will make these hermits even more “hermetic”* We discuss this paper on new forecasts of AI and its impact on economic growth: Forecasting the Economic Effects of AI * Refine and AI-assisted peer review are discussed in this pod. For more, see our episode with Ben Golub, founder of Refine. This episode is sponsored by Revelio Labs — a great source of labor economics data for academics and firms. Now available on WRDS.Join our Discord community at this link: https://discord.gg/w3GSapx2d TranscriptIntroduction [00:00]Seth: Welcome to the Justified Posteriors podcast, the podcast that updates beliefs about the economics of AI and technology. I’m Seth Benzell, your loyal non-fiction machine, coming to you from Chapman University in sunny Southern California.Andrey: And I’m Andrey Fradkin, coming to you from San Francisco, California. And we are very happy that Justified Posteriors is sponsored by the fine folks at Revelio Labs. And we’re very delighted to have Avi Goldfarb, who is a leading thinker in the field of AI economics and has also been a personal mentor on the show. We’re very excited to hear his thoughts on a variety of topics. Welcome, Avi.Avi: Thanks so much and thanks for having me on the show and looking forward to it.Andrey: All right, let’s get started. I have in front of me this book that you might remember writing at some point.Seth: Gaze into the soul of the man in the bookstore.What Did Prediction Machines Get Wrong? [01:12]Andrey: Now, I just think it’s a good cover. And I had to check: when was it released? It was released in 2018. And as I was skimming through it, you know, a lot of interesting points made there are still things that we’re talking about today, almost 10 years after it was released. So let me start off with the following question. And then maybe we can work backwards more into the ideas in the book. But what do you think prediction machines got wrong?Avi: I think prediction may... I’ll start with a hard question.Seth: No softballs on Justified Posteriors.Avi: So on the specifics of which industries and when, to the extent we tried, at least I did not anticipate how quickly language and coding would become prediction problems. And when we talk about disruption and industry disruption, a lot of the examples are things like driving, and we talk about radiology. And we still have plenty of radiologists around. Self-driving cars and trucks. seem like they’re now imminent, but it certainly took a lot longer than we expected back in 2018.Andrey: So is it a fair assessment to say that the large language models, even in 2018, weren’t on your radar? I guess they weren’t on many people’s radar.The Three Ideas of Prediction Machines [02:45]Avi: Not really. We have some discussion of machine translation. So that’s in there as a huge potential use case, but the arrival of ChatGPT and how it sort of changed how we interact with machines and how we think about AI was not really there. Another way to put it is prediction machines had three ideas. So idea number one is AI can be framed as a drop in the cost of prediction. So prediction. As in filling in missing information, statistical prediction is getting better, faster and cheaper. Idea number two is that when something gets cheap, you start using it for unanticipated uses. So when arithmetic got cheap, it wasn’t just that we use computers for accounting. We started to use computers for all sorts of things that we never used to think of as arithmetic problems like imaging and mail and music. And then idea number three is what are the complements to machine prediction? And we talked about data and judgment. The book, and certainly our attention to the book in the first three or four years after it was published, was on idea number one and idea number three. So identify prediction problems in your organization, and then think about what data you need to make those predictions better, and try to understand what matters to you in terms of judgment. And that second point kind of got lost. But in the last four years, it’s become clear to me is that that second point was maybe the biggest one, which is this tool, which still under the hood is computational statistics, enables us to find all sorts of applications for computational stats that we didn’t really imagine before. Judgment and data are still gonna be useful, but that phase one, that step one, that first idea of identifying prediction problems, that’s not really how we think about using AI today. And in some sense, that... was a missing emphasis throughout the book and throughout how we thought about that book, or at least how I thought about that book for the first few years.Does Proprietary Data Still Matter? [04:59]Andrey: Very interesting. You mentioned one kind of underlying idea there, whereas you should identify the data that’s going to make your predictions better. Do you think to what extent is that now true, given that your foundation models seemingly can be very smart without having any proprietary data?Avi: Data is still central to the use of AI, the building of the models. In building a foundation model that, at least in the pre-training stage, that data is essentially interchangeable. You just need more. It doesn’t really matter what. To build a structure of language, and then you can move from there. On later stages of using that model, at least the AI companies seem to think data is valuable to the model companies. And then in terms of use cases within organizations, that’s more a matter of whether you want to delegate sort of the judgment of how to use the model and what the model should output to the vendor or whether it’s something that you need to build in-house. And depending on the organization, some of them are very happy to delegate to the foundation model provider and some of them think they need to fine tune in-house.Andrey: Well, so there are kind of two little sub ideas in there. One is you have choice. You can fine tune a worse model with your own data. And maybe that will outperform as a frontier model. I think for many cases so far, that’s been a bad bet. But there’s a different idea here. Use whatever model you want, but you design the evaluation. And then you optimize via the prompting strategy or scaffolding towards that. that benchmark for your own use case. Is designing a benchmark proprietary? Should we think of that as a proprietary data that an organization has?Seth: Is that the judgment part in the judgment prediction distinction?Vendor Choice as Delegated Judgment [07:01]Avi: Yeah, I think there’s a bunch of judgment. there’s judgment number one: which which vendor do you use? Because you’re delegating a lot of values as in like, knowing what matters to the maker of the model. And then there is judgment in how heavy-handed do you want to be to make the outputs fit your needs? And then there’s judgment on, okay, you’ve decided to be heavy-handed. What exactly does that mean? And is it, guardrails or is it really making sure that the output from the prompts every time fits your organization’s values or what matters to you?Andrey: Have you had an opportunity to kind of advise companies on this judgment decision? Like what has your experience been in these situations?Avi: At a high level, yes. I don’t want to exaggerate my experience, but the things I emphasize and the things that seem to resonate are, one, what I just said, which is recognizing when you choose a vendor, you are delegating your understanding of what matters to that vendor. And then two, that means before you start thinking about choosing a vendor, you need to know what matters to you. So think through, you know, before you go talk to somebody, you should know what your KPIs are and what outcomes you want to see. Because otherwise, once you talk to them, they’ll convince you that their outcomes are the ones you want to see. and so it’s this, I talked to, someone who is running an AI at a... Let’s call it a big healthcare organization. And his job used to be, like five years ago, his job was building tools. He’s like, my job isn’t building tools anymore. There are all sorts of vendors building AI tools for healthcare. Okay. And what my job is now is every week, 20 or more people come in and say, I have a solution for you. And he chooses one or two of them.Seth: Kind of seems like a good job for an AI.Avi: Well, maybe, maybe not. But he understands the individuals, the people, guess, in theory that could happen, but the individuals in his organization, what they’re willing to accept, what they don’t. Which decisions they like to have control over, which ones they’re comfortable delegating. For the ones they like to have control over, he has a sense of what might be negotiable and what might not be. He knows where the power structures are and what things might change. Therefore face resistance from people who have the power to resist. He knows those things that might not face resistance from people because the people don’t have power to resist, but they’re going to be really, really unhappy about it. It’s going to bad for the organization. And so there’s all these things that I guess in principle an AI could do, but we’re a long way away, I think, from that.Can Prediction Eat Judgment? [10:16]Seth: So let me let me just push down that line a little bit longer is the way to think about this sort of prediction and judgment distinction is is that like as the models get better the Prediction is like eating more and more of the stack right? You know we give the information about our organizational structure to the AI and then maybe it can make a couple more of these decisions for us And you could either imagine that asymptoting to, you know, in 20 years, AI does everything, or you could imagine there are higher and higher levels of judgment that humans keep on getting promoted to. Are one of those two ways the way that you think about it?Avi: Yes, Andrea Pratt has a note in our first Economics of AI volume that covers that exact idea. I think actually it’s a comment on our paper or the model behind the Prediction Machines book. it’s, well, in principle, with enough data, you can learn to predict judgment. And so you move up the stack. So absolutely. There are some limits to that. There’s limits on you may never get enough data. on that kind of judgment. Judgment can change over time. To the extent that ultimately you’re trying to predict your tastes, then they can change over time. And there’s some limits on causal inference and the impossibility of seeing the counterfactual, which creates a need for a model.Andrey: But humans have that problem too.Avi: Yeah, yeah, yeah, no, I agree. But in the need for a model. So then the question is, well, how come LLMs and some of these models seem to be pretty good at doing that? And in the process of prediction, I suspect -- though I don’t know rigorous work on this, so I’m being cautious --Seth: That’s what this podcast is for.Avi: this is building some kind of model of the world that is embedded in the training data, like the language.Taste, Values, and Human Wants [12:16]Seth: So let’s go back to the one of the examples you gave, which is this idea of taste, right? Because I’ve had so many conversations with other economists about this idea that, well, taste will save us as a scientist, right? Because the AI won’t have taste. I have some ideas about what taste might mean, but can you be a little bit more precise about what you think taste means and why it’s something worth saving?Avi: So, okay, let’s operate under the assumption that whatever we want to call the machines, their goals are to help humans. Okay, not all humans. And we can debate about which humans, but like ultimately.Seth: Well, the Anthropic Constitution says, you know, safety first, the idealized anthropic researcher, then the guy that then then like virtue and then like the customer in some order like that.Avi: I’m gonna, all that matters for the point I’m about to make is that it’s not about the machine’s needs. So in that case, at the very limit, humans have wants and needs and those wants and needs, the machines need us, our judgment to know what our wants and needs are.Seth: So taste literally as in, this tastes good to me, I want more of this food.Avi: That would be one specific example of it. Absolutely. Okay. Now, I think we’re a long way from that limit, but that’s what I would argue the limit is.Seth: That’s the Bailey, right? So now let’s go out to the motte.Avi: So then it’s more like, okay, what matters to a set of humans, a group, an organization? What can we codify? If you can codify it and say, like, this is your goal, you’re not quite at that limit, but pretty close to it, then the machines can try to optimize on a goal. Goals have so much that are implicit. And so the machine would have to be able to infer the implicit part. Maybe it can, maybe it can’t, I don’t know. And then you can sort of ratchet back all the way to where we are now, which is you still need to tell your agent what you want. You still need to check on it every once in a while and guide it in the right direction. Prompting still has a role.Ontology, Umbrellas, and Context Shifts [14:45]Seth: Here’s another way of thinking about taste. And I’m curious whether you think this is in one of the categories you already listed or a new idea or you wouldn’t call this taste, which has to do something like with the idea of your ontology that is kind of built into the system, right? It’s your way of sort of dividing the world up into parts and maybe a good tastemaker or a good judger might have a more refined or more adaptable ontology. than the prediction machine. So I’ll give you an example of what I mean. have a couple of examples in mind, but one example I have is, you know, historically in the data, it’s always been the case that if lots of people show up with umbrellas, it means that you can predict that it’s raining. But then we have these Hong Kong protests and in the Hong Kong protests, they’re the umbrella protests and people bring umbrellas to show that they’re protesting, right? And it seems like a human would do better at adapting to like the completely new context for why you would need umbrellas than, you know, a pre-trained system that was only on historical data. So you can say that that’s like a context switch problem. Is that one of your ideas of taste or is that more of a judgment that’s not a taste?Avi: Honestly, that seems like a prediction failure to me.Seth: Right. That’s just we don’t have data on the context that we’ve moved to. The job is to understand when the context has changed, maybe.Avi: The judgment, I would say the judgment is like, what’s the consequential decision that’s going to be a function of, look outside and I see a lot of people in umbrellas. Yeah. What am going to do? And.Seth: You know, I should water my plants. Should I water my plants?Avi: No, I water my plants. Okay. So I look outside, a lot of people are carrying umbrellas and I think, no, I don’t need to water my plants. Okay. And then it turns out it’s a protest. It’s a little bit of weird context, but going with your example.Seth: It’s gotta be a weird context. That’s the reason that the AI is going to make the wrong decision because it’s out of context.Avi: the, the automated sprinkler doesn’t go on and, my plants die. Right. Okay. So, the judgment is, is it then worth it for me to invest more either in my prediction technology or to actually go outside and look and to see if there’s rain, to overcome that downside. So what you described as an error in prediction, there’s ways to reduce that error in prediction. The judgment is whether it’s worth the bother to reduce that error in prediction or to create some kind of insurance system where you would say, you know what, I’m gonna water the sprinklers. I’m just gonna run the sprinklers anyway. That’s how I think about judgment. It’s sort of what goes wrong when your prediction fails or it’s one important aspect of judgment.Seth: Sorry, can I give you an even more abstract?Andrey: Wait, wait, wait. No. I actually disagree with the premise of the example in many ways. I think a reasoning model would be able to handle the situation, especially with internet access, substantially better than many humans already, because you can call an API to get the weather forecast if you’re unsure. You can read the news. You can use reasoning traces. There’s this kind of implicit assumption in your question that like, we’re just using a raw pre-trained model and like asking it to like, if you, like, if you had a gun to your head, what would you do? You know, and not use any reasoning.Seth: Okay, but I can tell you a story, right? The weather API was always reliable in the data, but now there’s been a government takeover and I don’t trust the new government and you shouldn’t trust the API weather data anymore, right?Avi: So Andrey, I actually agree with, like, that seems unrealistic, but I think the idea is what you’re describing is how many resources you wanna put toward making it right, and I would view that as judgment.Andrey: But I guess the model has that judgment, maybe. Already. Already. Yeah, that’s kind of goes out like the stack of when judgment problems become prediction problems, I guess.Avi: But then there’s going to be... well, there’s going to be some places where the model is imperfect. Okay. Yes. Still a prediction tool. It might be better than human. Actually, it doesn’t matter if it’s better than human. But to the extent the model is imperfect, how do you want to behave? Like, let’s say the model is right 99.99 % of the time. Does your behavior change at that versus 99.9999 % of the time, even if the human benchmark is 50? And that ultimately is going to is going to be essential to judgment. We do this with self-driving cars. The models aren’t perfect, but they’re better than human. And yet, I still drove to work today, partly because that’s the law in Canada.Andrey: Do you think there’s hope? I mean, maybe this is kind of too much in the weeds versus the abstract idea, but sometimes people implicitly assume that they’re anchoring on the current technology where there’s an instance of an LMM that does something. But we might be able to design systems of LLMs that are interacting with each other to cover some of these. shortcomings that we can think of. I mean, at a conceptual level, maybe it’s the same thing anyway...Avi: So maybe another way to think through these trade-offs is to talk about whose judgment, okay? Which is Seth’s example was about, or my example was about my judgment, know, the individual’s judgment and should they listen or not. Andre, I think what you’re describing is the model builder’s judgment on which things is it worth investing in making the model better and when is it okay not? Like they have choices on sort of rate and direction. And those require some understanding of what they think is going to matter in terms of the use cases, the model. And on that, yes, there is a limit where a small number of players have extraordinary power because AI scales their judgment because they embedded into the models. But I do think. then there is still a human or set of humans responsible. It’s not like, the AI did it. It’s humans making those kinds of decisions. And I understand, like, at the limit, that actually gets quite nuanced, especially once we have models with continuous learning. But that’s how I think about that problem.Grue, Bleen, and Black Swans [21:41]Seth: All right Andre, can I ask my riddle of induction question? Andrey: Do you need me to induce it?Seth: You already know where I’m going with this. I’m curious if Avi knows where I’m going with this, but this goes back to the question of maybe where taste comes in is having a better or a more human ontology than the machine. All right. Have you ever heard of grue and bleen, Avi? These are colors that are different than blue and green. No? Okay, awesome. So briefly, we have this conceptual category, which is a thing that’s green. And a thing that’s green, we think that if you don’t do anything to it, it should be green indefinitely, right?Avi: Okay, yeah.Seth: All right. There’s this other thing that’s called bleen and things that are bleen are green until the year 2029. And after 2029, they turn blue. Right. Here’s the issue is that bleen and green things are observationally identical until 2029. Right. Yeah. So an inhuman, bad at forming natural kinds, ontology of an AI might decide that something is bleen instead of thinking it’s green. Right? And a human’s role might be to say, no, that’s a bad definition of a natural kind. That’s a bad ontology. And that would be a role of either taste or judgment. Do you buy that? Is this way too abstract?Avi: I think what you’re describing is a failure of prediction. I don’t think that’s taste or judgment. The taste or judgment is if you or a machine aren’t sure if something is bleen or green, do you care?Seth: Okay. Well here’s the thing, you didn’t even have the concept of bleen until I told you about bleen, right?Avi: So this is just the difference, I think, between known unknowns and unknown unknowns. So in Prediction Machines, we have a whole chapter framed on Rumsfeld and his discussion of known unknowns and unknown unknowns. Look, sometimes you don’t have a prior on it, and it’s an unknown unknown. That doesn’t mean that it’s not a prediction failure. It was just off the support of your data, and you didn’t know what to do about it. And I think that happens all the time.Seth: Sometimes you find a black swan.Avi: Yes, exactly. And so like, there might be places where humans are better at that kind of prediction than machines. There might be places where both humans and machines are really awful at that kind of prediction. And if that’s the case, then you want to have robust systems to anticipate those kinds of things. And that’s where judgment comes in. Like, if you’re wrong about the existence of a black swan, you know, does that change anybody’s behavior? I think the answer is no, because black swans and white swans aren’t actually that different from each other. But if there were other examples, like financial crises, where he uses the metaphor of the black swan, then absolutely there are meaningful differences. And you shouldAndrey: Financial crises.Seth: All right, so you’re saying that jobs that will survive TAI number 7 should be Black Swan, anticipator.Andrey: Not an anticipator. Actually Seth, this is actually kind of the key point. The point is, anticipator of whether Black Swan affects your utility enough that you should plan for it.O-Ring Complementarities and Automation [25:22]Andrey: I think next it will be awesome to talk about automation and some O-rings. Actually, the previous episode we did, we reread Michael Kremer’s classic O-ring paper because it’s been so inspirational for so many. It’s a great paper. They don’t write them like this anymore.Seth: It’s so fun to read. They don’t like to do macro like that anymore, unfortunately.Andrey: So we were wondering, so you have your own spin on the O-Ring paper. Maybe you’ll tell, you can tell us a little bit about that.Avi: Paper makes a pretty simple point. There may be two simple points. First one is that when you think about tasks within a job, they’re not interchangeable and substitutable. So it’s not just like, okay, a machine comes in and takes tasks. Sometimes tasks are complements. Now that isn’t, I’m gonna a little cautious. We talk about that in our O-Ring automation paper. It’s not necessarily a new idea. It’s implicit in the constant elasticity models. you can have a Leontief production function.Seth: We’re talking about the Daron-style task-based models. But if you actually read the papers everything immediately goes Cobb-Douglas. It’s always immediately weird. All the tasks are substitutes and then Cobb-Douglas over all the tasks.Avi: Yes, but it’s possible to, within the canonical model, to have that. So our point number one is tasks can be complements. And I just wanted to be cautious because I don’t want to claim that that’s necessarily our idea. But it’s an emphasis maybe that the existing literature hasn’t had. And then the second is, well, once you have tasks that are complements, if a machine starts doing some of those tasks, human can move their attention to the other tasks that are not yet automated. And when that happens, the human gets better at those tasks, which then makes automation of those remaining tasks even harder because the machine has to be better than now the human who’s spending all of their time focused on the remaining few tasks.Skills Versus Tasks [27:40]Seth: So let’s pause right there because I have a couple of questions right there immediately. So one way to think about automating part of your job is you’ve automated part of your job and now I can reallocate to the stuff that’s not automated. also another way to think about tasks within a job that are complementary is to think about them as sort of like innate skills or abilities. So think about the job of being a basketball player. The job of being a basketball player involves being tall and being agile. If you somehow automated being tall, I can’t reallocate my skill points into being agile, right? If we think about my performance as more as a combination of my skills, then automating part of it or taking part of it away, it’s not necessarily obvious to me that I can get better at the thing that’s not automated.Avi: The way we, okay, so first the way the literature usually thinks about jobs is generally at the task level, not the skill level. Okay. So a worker does a bunch of tasks. Okay. Those tasks require skills, but the worker does a bunch of tasks and the A machine comes along and can do the task and not the skill. So I’m not sure what it means for a machine to be tall. What it means for a machine to slam down.Seth: Well, let’s think about being a doctor. Let’s assume you might imagine being a doctor involves bedside manner and judgment about and diagnosis right it’s not clear to me that if you automate my diagnosis I can reallocate more effort into bedside manner some people are just level five at that and some people are level one at thatAI Doctors and the Future of Medical Work [29:25]Avi: It is obvious to me that there’s a bunch of tasks in a doctor’s workflow. Some of them involve diagnosis. Some of them involve talking to patients and making the patients feel better. And within those, there are skills in being good at filling in the missing information of what’s wrong with the patient and skills of making the patient feel comfortable. And actually, for some of those tasks, you might even need both. A machine comes along and automates the diagnosis skills. Okay. That means medical professionals are going to be spending more time on the other skills. This is actually an Eric Topol’s deep medicine book. I’m not sure if you’ve read it. It’s, it’s like a pre-ChatGPT, but like how AI might transform medicine. And that is his core thesis. The idea is that AI is going to make healthcare human again, because doctors are going to spend less time looking at screens and focused on diagnosis and more time. interacting with patients and making patients feel better. So in that sense, we get the automation of the diagnosis task and some of the computer tasks that should exactly lead to reallocation toward the human part. But then you brought up something else, which is, do our current doctors, if they spend that much more time interacting with patients, are they the right people for this job? Or alternatively, could we have a different set of medical professionals who we could train because now the machine can do some of those tasks who would be way better than our current doctors at the remaining tasks? I suspect if the machines get good enough at diagnosis and identifying appropriate treatments, there is an enormous opportunity for a new kind of medical professional who is focused on essentially interacting with patients.Seth: Yeah, so you’re making the occupational reorganization point and that’s that’s obviously essential and we’re going come back to that in the second. Yeah, I just I’m just pointing out that maybe maybe my example of basketball wasn’t so good. Maybe my medical example wasn’t so good. But I bet you I could pick out some domains where the elasticity of task output to effort is very inelastic.Avi: Okay, trying to think. You’ve switched from skills to task and that makes me much, much happier.Seth: Well, I mean, you would only need to worry about skills is if you were inelastic to effort, right? Then it’s just the skill.Rare Skills, Common Skills, and Wages [32:04]Avi: So there’s the new Autor and Thompson paper on automation, which I think gets at some of the things you’re talking about, which is if the things the machine does are relatively rare skills, like are tasks that involve relatively rare skills, to be precise, then what happens is we get entry into that profession. More people can do it and very likely wages go down. And if the machine things that the machine does are things that many people can do, they require less specialized skill, then the remaining humans in that job will, there’ll be fewer of them and they’ll likely be higher paid.Seth: Right, think that’s right, but I think maybe a missing component here is within the job already, what is the correlation in abilities between people who are good at the automatable and non- automatable part of the task, right?Avi: Yeah, but I think that’s the statement about that. Like in the short run, we’ll get the Autor and Thompson results. And in the long run, we’ll get a reallocation of jobs, right? There’s a system of professions and the system of professions will change.Are Tasks More Complementary Than Cobb-Douglas? [33:23]Seth: In the long run, you get the reorganization of jobs. Maybe one other thing I want to talk about before we get into reorganization of jobs is just this question about, tasks more complimentary or less complimentary than Cobb Douglas? Do you have a sense of that with tasks within a job? I mean, it seems like would vary a lot, a lot from occupation to occupation. I think we all have this intuition that they should have some kind of complementarity. That’s why they’re a job in the first place. That’s why they’re bundled. But you might bundle them and they still might just be, you know, gross substitutes that have a little bit of complementarity.Avi: I suspect there’s a lot of heterogeneity across jobs and I don’t think we have good data on that yet because sometimes we haven’t been looking because our model is substitute model and so our papers are fundamentally focused on the substitute.Seth: And I think this is an example of somehow the theory is sometimes a little bit downstream of the data, right? We just have so little data on people reallocating effort across tasks within a job that of course it makes sense to aggregate up to just add up all of the tasks done by all of the workers. That’s kind of, that’s my guess of why Acemoglu gets there.Avi: So of the task papers, the Eloundou et al., Dan Rock’s paper, is incredibly careful on every page.Seth: This is not an automation measure. Do not use this to measure automation.Avi: This could be a complement, it could be a substitute. These are just jobs that change. So like kudos to them, the four of them for being super, super careful. Nevertheless, when that paper is cited both in the academic literature and in the press, that idea seems to get lost. I’m not exactly sure why, maybe that’s because of the model.Seth: Question people want to answer, right? The people don’t want to know what job’s going to change. People want to know what job should I get, right? And so...Avi: Well, okay, but if it’s a question people want to answer, then the complements matter just as much as the substitute. I wonder if the answer that people want to know, like the answer that people want, and then they just...Andrey: I actually think it’s I think take has always been that just most people are pretty, they’re very sophisticated users of this data, but a lot of people don’t have a sophisticated economics model. And therefore to them, it’s just obvious that what’s going to happen is the machines are going to take our jobs. As a result, that’s just, they don’t have a more nuanced model of economic activity and therefore that’s how they interpret it. Now there are more sophisticated readers, think, we know some of them, where they’re just really just think that AI is going to be able to do everything in a very short period of time and then it all kind of becomes moot. You know, if you think that every single task can be done by an AI.Why the Impact of AI Was Ambiguous in Earlier Work [36:15]Seth: Yeah. Well, I guess this kind of brings us to your 2019 Journal of Economics paper, which is about where you guys kind of where you kind of throw your hands up. That’s not that’s a positive part and say there’s an ambiguous impact. So I guess I want to push you there on is the ambiguous impact because. We just don’t know all of the relevant elasticities, right? We need to know the elasticity within tasks within a job. We need to know elasticity across jobs within an organization, the elasticity across sectors of demand. And if we could put all of those together, we would be able to answer the question. Or is it more ambiguous than even that?Avi: No, I think you need to understand when that paper was written in order to understand the paper, which is in 2019 or late 2018 when we were writing it, we had no concept of anything but a task- based model with substitutes. Okay, maybe that was on us. We should have. But Acemoglu and Otter and Rastrepo were the dominant- Paradigm. ... working in literature, especially Acemoglu.Seth: Are you saying our ontology was limited?Avi: I’m not exactly sure what you mean by that, but...Andrey: You forgot about the O-ring which was the black swan of papers.Avi: Yeah, yeah. So like, we did.Seth: I mean in Kremer, I mean, presumably you looked at Kremer again before writing your paper. You can almost see he’s almost there. He’s almost at, and this is within workers too. He doesn’t exactly say it.Avi: Exactly. So when we wrote that paper, we were thinking task-based substitution. That was the model that we had. And actually, in the process of writing that paper, in some sense, we learned what was wrong with that model and ended up with, we just don’t know. And part of that is, we wrote it in 2018, 2019. We were looking for new tasks from AI. So this is before ChatGPT, like four years before ChatGPT. So new tasks hadn’t really come up yet. All we had was identifying space junk and treatment for complex disease, which actually wasn’t our idea. It was Tim Taylor’s idea, our editor.Andrey: Well, you already had AlphaFold, right?Avi: Yeah, but it’s not clear what the new task is because of AlphaFold. Yeah, fair enough. In terms of... So, and actually that paper in some sense directly led to our work on system change and GPTs, because Tim Bresnahan pulled me aside that summer at the Summer Institute and told me he hated our GPT paper. I’ve told you guys this before. Because it was a task-based model and that’s not how meaningful change happens. That then led to all this work on trying to understand, well, if it’s not a task-based model, how does the system change?Andrey: Okay. And we’ve covered that to Bresnahan paper on this podcast.Reorganizing Jobs Around AI [39:22]Seth: I guess let’s talk about reorganization of tasks. Obviously that seems to be, that’s the best case answer. The best case answer is you split off the, I guess from the perspective of a firm trying to boost productivity, maybe not necessarily from a worker’s perspective. From the firm’s perspective, you want to slice off the automatable thing, let that rip, and then figure out what you have to leave behind for humans. Is there any good research about... How do you do that? What industries are better than that at others? Like, what’s the next research frontier on that question?Avi: I think you just defined it. there are two. One is like within the firm, how do we think about where the complements are and what’s left for humans and how does that vary across organizations? The second part, and Alex Emas has highlighted this recently, is it also depends on elasticity demand for the...Seth: products.Avi: Like, you know, even if within an organization workers reallocate and they become hard to automate because they’re more productive, but then the organization is producing more, well, someone has to want that more or else then, you know, at least that organization or its competitors are going to to business.Seth: Well it’s factor, well its price will come down, know there’s a kind of a nebulous connection between price and profitability.Avi: Right. Price goes down. It’s got to go down like, well, quantity has to go up enough that we still need the workers.Andrey: There might be a paradox in there that’s not really a paradox. The misnamed Jevons paradox.Avi: Maybe.Should We Want Less Automation? [41:05]Andrey: Following up on this idea, think several prominent economists have called for a government push or ideological push to make AI that complements humans rather than substitutes for humans.Seth: Friend of the show, Erik Brynjolfsson has written about the Turing Trap. Is the Turing Trap misnamed? Is it not a trap? Should we embrace the Turing?Avi: Okay, so this is our science paper.Seth: Let’s get the hot takes. This is where we brought you on.Avi: Do want more automation? Yeah, so Eric has said it. Doron has said it. There’s lots of policy. We should complement humans, not replace them. And John Markoff is a journalist. He has this book called Machines of Loving Grace, same title as Amodei’s essay, essay, but older book. It is about the history of computing.Seth: When you’re a tech billionaire, you’re allowed to use cool phrases unsighted. I’ve noted this.Augmenters, Automaters, and Inequality [42:10]Avi: Well, they’re both referencing a poem. And in Markov’s book, there’s these two streams of computer science. There’s the, I forget exactly how he labels them, but essentially there’s the augmenters and the automaters. And at least from my perspective, the augmenters seem like the heroes of his story. And the automators who start to become prominent as this book is getting written around 2014-2015Seth: They’re trying to trap us. They’re trapping us.Avi: But we also know that the rise of computing the internet massively increased inequality. They generated enormous wealth, but they massively increased inequality. And I hypothesize that the reason for that is, yes, they were augmenting what humans do, but they weren’t augmenting what all humans do. They were augmenting what a set of humans who are good at abstract thinking do. And those people were already doing pretty well. And so in the process of augmenting humans, right, because no human can do what the internet does or what a computer can do, they augmented folks at the top and left others with relatively stagnant incomes.Seth: Is this story there really at the task level? The way I think about that inequality story is that it’s kind of at the firm level, right? It’s we’ve now put the corner store into competition with Amazon and so Amazon wins and whatever Amazon takes as input wins.Avi: There’s a bunch of different pieces. The one I’m emphasizing is like the Autor, Katz, and Kearney framework, which is about skills.Andrey: I mean, it has to be both, right? There’s a set, right? Like, the humans who are now able to market their unique skills match with the firms that are larger, but you kind of need both to create the inequality or some of the humans become superstars without like needing the firm in first place, right?Avi: I think in principle you could get within firm inequality without getting across firm inequality. We ended up getting both.Seth: Yeah, both. Both happened.Andrey: Fair enough.Avi: but as I’m thinking like Autor, Katz, and Kearney with computing and then Shane Greenstein, Chris Foreman and I have some work on sort of the internet inequality, same kind of idea. so on the other hand, automation technology, if it’s automating things that folks at the top do, could superpower everybody else. Okay. And this is a could, cause we hasn’t really happened. So what we hypothesize, so the question, the paper is called, Do We Want Less Automation? And our answer isn’t no. Our answer is, here are reasons why it’s not obvious. Okay? It’s very economist-like. And the essence of it is, we were just talking about this medical example. Well, if what doctors are paid for is 10 years of post-secondary schooling, that essentially is about prediction, diagnosis and treatment. Then someone potentially with two to four years of post-secondary schooling who was much better at managing patient stress and all these other things, training like a social worker, combined with a diagnosis machine could be super hard. And so their productivity goes up. And there’s a bunch of industries where What people at the top do seems a lot like filling in missing information.Are Intellectuals Giving Biased Advice About AI? [45:58]Seth: One might even cynically say that these thought leaders who have been so augmented by the internet are maybe not giving the populace the best advice.Avi: Maybe. So I had an undergrad RA write an essay for me. She’s a philosophy major. you know, a couple summers ago, it’s Amelia Agarwal. I feel like I should call her out.Seth: Love undergraduate research on the pod.Avi: Yeah, the opening of her essay was, part of her assignment was to read and hear about all these people who said AI is going to automate work. And so I’m going to have to have leisure, like essentially. And she’s like, that doesn’t strike me as bad. And then she dug into it and her framing was essentially the people whose identity was driven by their, you know, intellectual abilities, public intellectuals are exactly the people most threatened by AI. And so anyway.Andrey: You know, it’s very interesting. I actually disagree. Yeah, I think lots of intellectuals are threatened by AI but not public intellectuals and that’s because humans are going to want other humans to communicate to them in many ways. So, the role of the public intellectual is not going to go away. The role of the maybe the scientist toiling away on their research. That is in my opinion much more a threat. if you’re... one might even deduce that Seth and I have started this podcast as a hedge for that world.Seth: Well, what I say is as the price of writing papers goes down, the return to reading papers goes up. But maybe this goes back to the taste idea, right? Which is one way you might think of taste is a public intellectual doesn’t let’s let’s be cynical for a minute. The public intellectual, the public art critic doesn’t actually know art better than anybody else, but they serve a role as a coordination mechanism. Right. Everybody trusts Andrey. So when Andrey points at the thing and says it’s good, everybody converges to that. And then maybe that’s one notion of taste that will be preserved.Avi: Yes, and so you started in science and moved to art. There’s probably differences between them, but in the sciences, there’s a question, or a scholar’s, what’s our goal? What are we trying to accomplish? And I think different disciplines have different goals. And depending on the goal, the role of the human curator changes. If the goal is so that humans understand the world, and have sort of a consistent model, then there’s a real role for a curator. If the goal is to build a better spaceship, then maybe there’s not such a role for a curator. And so I haven’t been following that literature, so I don’t know really what the formal academic take on what I just described is.Can Policy Steer AI Toward Augmentation? [49:27]Andrey: Yeah, I agree. I haven’t seen much formalization. So listeners, if you know of any, send it along. Yeah, I mean, I sorry, I just want to make a final point is that I think I like your criticism of this augmentation idea. But to me, there’s like a much deeper criticism, which is there’s there’s just kind of a whiff of central planning involved in it. like, how how do you know? What technologies are going to automate versus augment. Like this is very hard to predict in my mind. And to think that the government is going to like somehow implement a system of taxes on technologies that are augmentation versus substitution, it’s ridiculous in my opinion.Avi: So I was taking as given that you can understand what is automation and what’s augmentation. I agree it’s a very hard challenge. There, I think the narrative, I’m gonna be careful. I think the argument is if even without choosing winners, we might be able to tax capital relative to labor or something like that. in order to push things in a particular direction. I think that’s it.Andrey: Yeah, that’s the most plausible.Seth: That’s pretty plausible, but when you actually hear versions of the Turing Trap articulated, it’s really like go and burn down the houses of the people who want to automate you.Avi: Okay. So Korinek and Stiglitz have a chapter that’s really about tax and capital that’s in our economics of AI book. And I think like the Acemoglu Johnson argument is really about tax and capital. I’m not enough of a macro economist to have a strong opinion about one way or the other, but that I agree seems moreSeth: Right, and then there’s a deeper, deeper argument there about whether or not you want to tax capital, right? There’s the old Chamley-Judd result about, well, know, labor is inelastic and capital is elastic, so really you don’t want to tax it. There’s obviously international considerations about if you have a fully automated technology, isn’t that just going to locate itself in the lowest tax jurisdiction? And so it might be very hard to tax capital. And then of course the Iván Werning follow-up research kind of complicating the original Chamley-Judd results. So this gets in the weeds really fast.Andrey: And it’s also very blunt in many ways, right? A lot of capital is not about automation. it’s a... I don’t know.Avi: Yeah, and there’s all sorts of questions in public finance and how that all plays out to like the there’s under the names Trammell and Korinek. I think it’s Trammell. No, it’s not.Andrey: That’s Lockwood.Avi: Lockwood and Korinek, thank you. have a relevant paper there.AI Growth Scenarios Through 2030 [52:36]Andrey: Next topic. Yeah. So there was a very well-circulated survey of economists about their expectations of economic growth in different AI scenarios.Seth: Now Avi, I understand you have intentionally not read this so as to have an unbiased take, so you will not be contaminated by the opinions of everyone else. Is that right?Avi: That is absolutely right.Andrey: Excellent. You’re definitely not in the same university as many of the authors.Avi: I probably will, but we’ll see.Andrey: All right. So the first conceit is that there are three scenarios for AI progress that they want us to consider. The first one is slow progress, where by the end of 2030, the AI can do PhD student level assistance, half of eight hour long coding tasks, passable stories and songs. Robotics navigate homes with some help. So that’s kind of the slow. Moderate is you have semi-autonomous labs, five-day coding tasks, high-quality novels and hit songs. Robotics can perform basic tasks. And then rapid progress outperforms top humans in research coding and leadership, award-winning creative works, nearly all physical tasks. So those are the three scenarios by 2030. So the first question is, how do you allocate the probabilities between slow, moderate, and rapid by 2030?Avi: So, okay, so with the exception of the statement about hit songs and award-winning, those are all about the models and not about the outcomes. So I’m going to ignore the hit song and award-winning part because I think that’s...Andrey: It’s of the quality of the quality that could win it.Avi: Okay, because at a high level, what I think is the technology is going to accelerate rapidly, but there are all sorts of meaningful barriers to widespread diffusion and having an impact on the economy. and sometimes I think we’re already in the slow and for aspects of the medium versus the fast, I feel like I should call it 50-50 because I’m skeptical of the like, I’m skeptical of the robotics stuff, but the five day coding task seems very, likely. And so just.Andrey: Yeah, there’s some other things. CEO level agency, you know, like is is one of the criteria.Seth: I don’t know whether or not they can run a vending machine.Avi: But don’t like part of it. So much of what a CEO does is like is charisma and creating followers, right? And I’m not sure that’s a mission.Seth: Is it charisma judgment task? Is it charisma judgment?Avi: It’s a skill. I’m not sure it’s a prediction or judgment. It’s more like an action.Andrey: Yeah. But okay, fair enough. Just to give you like a sense of where economists came in and they took this in the fall, 39 % that were still in slow by 2030, 47 % that were in moderate and 14 % then were in rapid. So you are more bullish than a typical economist.Avi: I’m more bullish. I probably shouldn’t have said zero for slow. In retrospect, I was just going to be something five to 10 or something like that.GDP Growth by 2050 [56:22]Andrey: Okay, great. Now, and I think this is the question that really there was a lot of controversy about. So, the question was, by 2050, what is the annual change in GDP on average?Avi: GDP or GDP per capita.Andrey: This is GDP.Avi: I like I have to make a population assumption. somewhere between two and 3%.Andrey: All right. You are well within the economists’ answer here: 2.5%.Avi: duplicate. And so we’ll be a little above that.Andrey: So 0.5%, that’s all we get. okay. Extra from AI over and above.Avi: Well, no, I don’t think you want to say that because the reason we have 2 % is because of innovation in past.Andrey: Okay, so fair. I agree, I completely agree with you.Avi: Like it’s possible, especially with, you know, it’s possible we would have gotten zero.Seth: 5 % better than historical rate of technological growth.Avi: Yes, something like that.Andrey: Now, what if you were for sure, what if you for sure knew we were in the fast scenario by 2030? How would that like change your predictions?Seth: It’s hard to get to above three.Avi: Like, yeah, I just think there’s a lot of bottlenecks in the economy. I think that, and we’re going to figure out what they are.Seth: We’re gonna find out fast and that guy is gonna be rich.Avi: Yes.Andrey: So you’re once again, like a very down the median economist.Avi: On growth. Yeah, okay.Seth: Can I ask you, you think that’s mostly about bottlenecks? You don’t think that’s mostly about people taking leisure?Avi: I think it’s mostly about bottlenecks.What Are the Bottlenecks? [58:36]Seth: So gun to your head, what’s the biggest bottleneck in that high growth robots are awesome scenario.Avi: I feel like my best answer is we’ll find out.Andrey: Okay. I guess the pushback that folks gave is this is a scenario where by 2030 robots can do nearly all home and industrial tasks and faster than humans, right? So you might say, well, manufacturing and physical tasks are a tiny, not tiny, but they’re not that big of a portion of the GDP already. maybe-Avi: be essentially zero is the point. If they’re that efficient and that cheap, then they won’t mean like, I guess it depends on how we calculate the deflator. agriculture is way more productive. GDP hasn’t grown by that much.Andrey: But what if we have, you know, you know, robot doctors that can do, you know, like,Avi: Great, then medicine will be cheap. It’ll be less of GDP.Andrey: I guess, all right, so here’s a hypothetical. Here’s a hypothetical. Let’s say we had a cure for cancer as a result of this, which is very plausible in the rapid scenario, and that we also, at least in principle, have the technologies to administer it through robots very efficiently because we are in a world of just true abundance. My sense is that people would value that medical care extremely highly. And if one were to properly deflate the existing cost of cancer treatment, wouldn’t that imply a very large GDP effect? Now you can say maybe we’re not going to calculate that correctly.GDP, Consumer Surplus, and Health Breakthroughs [1:00:25]Avi: Now I feel like I’m going to, you know, it’s sort of the Bob Gordon sense. I don’t think we deflated antibiotics properly. I don’t think we deflated flush toilets properly. So if you’re talking about consumer surplus, then maybe consumer surplus will be found, especially, you know, to the extent that it’s health outcomes, then huge increase in consumer surplus, much more than the argument that we’ve had for digital. Because the that debate on whether digital really made us better compared to what was happening in the 20th century, I reasonable people can be on both sides of that debate. what you’re describing, is can’t secure people living wonderfully and healthy to 100, there might be some limits to how long, but that would be wonderful and great for consumer surplus. But if that happens, I guess it might and it’s that easy, it might become so cheap that it’s it’s like agriculture. Because food is pretty essential too. And food is so cheap that we don’t worry about it so much anymore.Seth: Inelastically demanded. think people will elastically demand years of life in a way that they won’t elastically demand calories, right?Avi: Potentially.Seth: You think people will get sick of it. I thought you were to go to maybe you’ll recall in Doron’s simple macro economics of AI, a favorite paper of this podcast. He actually predicts that actually consumer surplus might raise by less than is implied by the GDP growth rate, because we’ll invent evil jobs like social media manipulator. Do you are you still convinced that consumer surplus growth will be faster than GDP growth evolves? Or are you open to this idea of the invention of evil tasks?Avi: I feel like we are not in my expertise.Seth: Turn it up.Andrey: Seth is really trying to get the hot takes.Avi: I don’t like to judge what particular products, a particular.Seth: Well, you can’t judge, you can’t predict.Avi: Yeah, you know, what am I in a-Andrey: Then you become a economist.Avi: Actually, let me give... So I think it’s reasonable for people to say some roles, some jobs, some products are better than others. I don’t think that has a meaningful role in GDP calculation. And I also worry if in our consumer surplus calculations, we economists say some things are better and some things are worse because then... So much of it is just obviously to the taste of the...Seth: It’s such a normative can of worms, right? GDP we can measure, consumer surplus. I mean, we do things at the Stanford Digital Economy Lab around trying to do willingness to accept experiments, but obviously those are highly limited too.Avi: So consumer surplus as in figuring out the area under the demand curve, that’s the kind of task I think we’re good at. It’s within our domain. whether the demand curve is morally right or wrong, that’s not something I’m going to be finding out this day.Andrey: I wanted to just like close off that loop a little bit by just saying that you just gave me an answer that said that for our evaluation of how good of a world we’re gonna get in 2050, GDP is no longer the correct sufficient statistic, which obviously makes me question like why is this such a bench? Why are people so interested in forecasting GDP in 2050 if we think it’s going to get pretty uncoupled with consumer surplus in these scenarios?Avi: Well, I’m not sure it’s more or less uncoupled than it has been in the past. I think reasonable people can disagree on that. I think the debate between Bob Gordon and Erik Brynjolfsson or Bob Gordon and others over the years is sort of is really informative about how hard it is to say, you know, what’s better versus today versus the past. What happened in the early 20th century is pretty amazing. okay, that’s point one. Point two is it’s not obvious to me that GDP like GDP tells you your national capacity. That’s what it tells you.Seth: That’s useful for things like wars and public finance.Avi: If I remember my first year econ, haven’t taught first year econ for a long time. That was the idea. What’s the industrial capacity of the country? Or what’s the economic capacity of the country? It turns out it’s highly correlated, as I understand it, with lots of welfare measures. You guys know this. And so we use it for that. Once you start deviating, then... then that’s fine, but you’re now embedding a whole other set of values. At least with GDP, we know what the values are. It’s not it’s not value laden, but we at least know what the values are that we’re embedding in that measure.Andrey: But guess I’m not sure we know, just in many conversations with economists, this question of deflators has come up and most of us haven’t spent much time thinking about what actually goes into that and how well that’s done and how relative to different goods. So I agree with you that we’ve been recommending that people use this because it’s very correlated with welfare, but you know.Avi: So, yes, and the NBER productivity group in many ways was focused on questions about how do we measure innovation and progress and a lot of that, some of the early work that came out of it was explicitly about this question. it’s not that people haven’t thought about it and that there’s not a whole community that grew out of that. Now admittedly, we don’t have that many, you know. papers about deflators and inflators anymore. But Shane, when he was running the program, digital, almost always had somebody on the program focused on measurement of prices over time in the digital world. So just to say at least it’s on his radar and it was part of what Sloan Foundation was excited about why they originally started funding the Digital Economics Group.Sponsor Break: Revelio Labs [1:06:56]Seth: This chance to contemplate your posteriors is sponsored by Revelio Labs. Revelio Labs is a leading provider of labor economics data and data services for companies, academics and independent researchers. Andrey and I have been working in economics of AI for a long time and we can confirm just how useful Revelio’s data is. Revelio’s team combines comprehensive micro-level data on employee professional profiles, job postings and employee sentiment with standardizations, mappings, and enrichments available, all to make that data useful without making your modeling decisions for you. The data can be flexibly aggregated to company, market, or industry and be used to study questions ranging from career trajectories to occupational transformation to the returns to skills and the impact of AI on labor demand for tasks. Can’t imagine anyone be interested in those. And Revelio data is available on RWRDS. So if you’re an academic with a good library, you might already have access. And if you don’t, you can reach out to their excellent economics team and they’ll hook you up. Will AI Centralize or Decentralize Decision-Making? [1:08:16]Seth: All right, okay, we’re gonna give you a topic. We want your hot take. So will AI centralize or decentralize decision making in the economy?Avi: Yes.Andrey: It was good though.Avi: Like, so, I don’t know, this is no longer lightning round. But for an ultimate hit thing, have that interesting paper saying why it’s gonna centralize and their argument is good. And the exact same arguments they have also say that it could empower people on the periphery. And the answer is almost surely both are gonna happen. There’s gonna be some people who figure out how to scale themselves and their judgment and gain enormous power. And at the same time, others who are able to do things they couldn’t do before, just like we saw with online platforms where there’s been both the centralization of power and the ability of niche players toSeth: Here’s the part that I thought that that dialogue missed, which I recommend to all of our readers to look at because it’s fascinating, is the argument that AI will centralize us is that AI is going to help these centralized decision-makers understand the complexity of what’s going on. But what if AI makes us weirder faster than AI conceptualizes the weirdness that it’s creating? What if we just get super duper weird? That would make it very hard to centralize.Avi: Yeah, I think that’s a version of my argument, which is that the people on the periphery can, know, individuals can use it to make themselves more productive, better, happier, whatever their goal might be.Seth: more, more, less, less controllable. What do LLMs imply for privacy regulation in economics?Avi: first answer was nothing. There’s lots of ways to worry and think about privacy and privacy does matter. First answer is not obvious how it matters now differently than it did five years ago.Digital Hermits and De-Anonymization [1:10:11]Seth: I the idea is that it will be...Andrey: De-anonymization.Avi: Yeah. So yeah, so that’s where I said my first answer. then, okay, well, to the extent that okay, here, here we go. Catherine Tucker, Jean-Michel Lachetal and Avery Haviv and I have a paper called Digital Hermits. okay. And the idea of that paper is again, I’m really bad at these hot that all. okay, the the idea of that paper is right now you might be willing to give your grocery preferences to whatever company. but you might not want the company to know your IQ or your religion or something else, your union status or something like that. Okay. And in a world with bad prediction tools, you can give your grocery information and not the other information. But if some other people are giving both, then over time, you can’t even give your grocery information if you want to protect your religion or IQ. So. In the equilibrium there, we end up with one or two groups. We get hermits who don’t give any information and everyone else who gives all their information just gives up. So what you’re describing with LLMs is a version of the prediction mapping from, just writing something to now having all sorts of extra information about you that we might not want to put you. And so like being able to connect different pieces of information.Seth: make the hermit hermit-ier, right?Avi: They’ll make the hermits hermit-ier and create demand to the extent that privacy is a value and it’s now harder to protect. There’ll be demand for laws thatSeth: It’ll make the hermits more hermetic, I should say.Andrey: I think it could be a function of abuse, right? Obviously, I haven’t studied privacy as much as you, but I think when this data gets abused, there’s a lot of demand for laws, retribution, and protection. But when it’s an abstract value, but it’s not getting visibly abused, it seems like it’s less of an issue. this data is used for personalized advertising. Yes, some people have a negative reaction to that. In the end, in the grand scheme of things, it’s not that bad. But if now someone is finding out, you know, all this private information about you specifically and, you know, that information, let’s say can be, you know, someone, you know, leaks it or talks about it online or tells your employer or whatever, you know.Avi: So Right. Yes, there’s going to be a decline in online anonymity. Actually, like, I if you remember Catherine Tucker’s discussion of Florian Ederer’s paper on de-anonymizing econ job rumors at NBER. That paper is about fundamentally about something else. But her discussion was, okay, this is the world we’re moving to. Maybe because of quantum, maybe because of LLMs, it’s gonna be very hard to post things anonymously. And so once that happens, once things you say digitally you expect to be known, how does that change behavior? And then there’s like, I guess your original question was, how does it affect privacy regulation? LLMs are gonna do two things. And I don’t know what the equilibrium is gonna land, which is, I don’t know why you keep doing this.Seth: I’m counting this is be thing number one and then I’ll you thing number two.Avi: So thing number one is what we just described, which is demand for privacy regulation goes up because there’s new risks and people do value privacy. The other hand is there’s new opportunities to use data and benefit from your data. I can sort of think about that’s what agents are going to enable you to do. And so there is also an increase in demand for regulations that enable data to flow. And where that plays out country by country, continent by continent, who knows? But like, just like with digital, we saw both the increase in the benefit and the increase in the cost of data flows. I think we’re going to see another wave of that.AI and Peer Review [1:14:23]Andrey: Follow on question, peer review. You were the editor of marketing science for a long time. Narrow question is, what does this imply for anonymity of peer review? And a broader question is, effects of AI on peer review more broadly.Avi: So yeah, I was, was a senior editor, marketing science. Actually, I haven’t thought about that peer review anonymity point, but absolutely in principle. it’s disguisable. I think there’s a solution to this. I just don’t know that we want it. Like running your review to have ChatGPT or Claude or whatever you want rewrite it so that it doesn’t sound like you with all your points. Seems at least on the language matching will work. Not on the idea matching, but that’s already revealed. Like a whole bunch of people tell their author to... Although actually as an editor, learned that it’s not as... Often it’s not the author that’s asking for those citations. It’s like their advisor. Okay. Like there’s a lot of that, but still. So I think that’s manageable. Certainly on the one way, like I don’t think we’ve hadAndrey: Yeah, at least.Avi: Double-blind peer review for 20 years, at least in the econ side of marketing and then econ. The pre-prints are out there. The pre-prints are well distributed.Andrey: Yeah.Seth: So it just be public? mean, so that would be the other direction. Is that it just opens public reviews.Avi: I think if reviews are public, we’ll all just collude. I think those be mass collusion. I shouldn’t say we’ll all. I would prefer to think that I won’t collude. But I think that’s just an invitation.Seth: Go ahead. You don’t think that there could be a disciplining of that when somebody reads your review and says this is, this is nonsense?Avi: I think the benefit of having your reviewers for sure know that you said good things about their paper, it’s going to be hard to overcome. There’s a question about whether the whole system makes sense or not.Andrey: Well, that’s kind of what I was getting to next. I, you know, I do some advising for Refine and I’m a big fan of their product. And it’s pretty clear to me that Refine is doing a better job of peer review than the vast majority of peer review outside of very, very select venues. And it’s only going to get better. And so the question is, given these capabilities, what should it look like in the future?What Are Journals For Now? [1:17:05]Avi: So, okay, I’m gonna propose something. But I’m gonna start with I don’t know. Here is one out there idea, which is, it’s not obvious to me what purpose the journals serve. When I talk to scholars, especially junior scholars, I don’t think people read the journals. They may be happy that some paper they knew appears four years later, but it’s not like they get the AER and open it and read it. You know, people in my vintage, or at least some of us,Seth: wall of JEPs under here as you can see.Andrey: One AER, Avi, is the one you gave me with my own paper. Thank you very much.Avi: A couple of years ago, I paid for like three years of AERs for them to deliver to me and then they refunded my money. Guess they stopped. Because they don’t print them anymore. So like, that just doesn’t seem like how knowledge is discovered anymore. even sort of like what I... Okay, so then what’s the purpose of the journal if it’s just to verify what matters or to verify accuracy, refined can do it. And then like, do we have the whole peer review system for? If it’s to not just verify accuracy, but also refine papers in a way that’s consistent with peers’ tastes, and especially with the editors’ tastes, then the revision process is important. And if it’s about the editors’ curated tastes, then there’s probably a much easier way to do that, which is they post their PhD syllabus.Avi: Like I wonder if what’s going to happen. Also like this, yes, there’s a lot of papers out there and submitted and there’s a lot of authors, but there’s just too much over the course of a year for anybody to keep track of what’s even in the AER, like one journal. Nevermind trying to keep track of marketing science and management science and all the others. Okay. I wonder if there’s going to be a curated set of people who, I don’t know who chooses them. who are essentially the tastemakers and maybe they’re editors, but maybe they’re just people who like say, hey, I like this paper. Justified posterior. I was going to say, that’s one role that you guys have. It’s this weird thing that people now in business schools can come out for tenure with eight, 10 papers in what are ostensibly A journals and no one’s heard of them because yeah, they published the papers, but they weren’t out there.Avi: They didn’t get onto syllabi or whatever else. those cases are hard, because on the one hand, they were told they needed to publish X papers, and they published X plus four papers. And the other, the point is to contribute to knowledge. And they’re there for somebody to discover eventually. But then maybe the LLM could just write the paper when you need it.Andrey: Currently we’re writing for the LLMs anyway, we know who the readers are of our paper.Closing [1:20:17]Seth: I think that’s a great place to leave it. Avi, this has been an amazing discussion. Thank you so much for making the time.Avi: Yeah. Great talking to you. Take care.Andrey: Thank you.Seth: All right, and you folks out there, please join our hopin’ Discord community. (https://discord.gg/w3GSapx2d) Like, review, and subscribe, and keep your posteriors justified! This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit empiricrafting.substack.com