Big Data's Limits in Financial Markets (S2, E8)

Send a text

Data drives nearly every aspect of modern life, from the algorithms suggesting what you should watch tonight to the autonomous vehicles navigating city streets. Yet in the world of finance—where you might expect data to reign supreme—the relationship between information and decision-making is surprisingly complicated (and relatively new).

Professor Mike Gallmeyer pulls back the curtain on this fascinating paradox, revealing why financial markets present unique challenges for data-driven approaches. While Tesla collects millions of data points daily to perfect self-driving technology, investors working with a century of stock market returns have barely over a thousand data points to analyze. This fundamental limitation—what Gallmeyer calls the difference between "big data" and finance's "small data" reality—creates profound implications for how we should think about investment decisions.

The conversation delves into the historical evolution of financial data, from the pre-1960s era when decisions relied heavily on intuition and "soft information," through the development of the CRISP database at the University of Chicago, to today's sophisticated algorithmic trading systems. Gallmeyer explains how market participants continuously adapt to new information sources, creating an ever-evolving landscape where yesterday's winning strategy becomes tomorrow's conventional wisdom. This endogenous change within financial markets makes them fundamentally different from systems where data collection leads to steady, predictable improvement.

For anyone fascinated by markets, data science, or the intersection of human judgment and quantitative analysis, this episode offers valuable perspective on the promises and limitations of data-driven decision making. Whether you're managing your retirement portfolio or simply curious about how markets function, you'll gain insights into why certain problems remain resistant to even our most sophisticated analytical tools—and where human judgment still provides irreplaceable value.

Show Notes:

• Dimson, Marsh, & Staunton, Global Investment Returns Yearbook 2025
• Kim, Muhn, et al., Financial Statement Analysis with Large Language Models (2024)
• New York Fed Staff Nowcast
• Federal Reserve Bank of Atlanta, GDPNow

Thanks for listening! Please be sure to review the podcast or send your comments to me by email at [email protected]. And tell your friends!