How Well Do LLMs Predict Human Behavior? A Measure of their Pretrained Knowledge

This research paper introduces the equivalent sample size (ESS) as a novel metric to quantify the predictive value of Large Language Models (LLMs) compared to traditional human-provided data. The authors define ESS as the specific amount of domain-specific training data a machine learning algorithm requires to match the accuracy of a pretrained, fixed LLM. To estimate this value, they developed a statistical inference procedure utilizing block-out cross-validation to compare LLM performance against error curves of models like Random Forests and Lasso. Applying this method to the Panel Study of Income Dynamics, the study reveals that LLMs effectively substitute for hundreds of observations in tasks like predicting homeownership but provide negligible value for others, such as forecasting smoking behavior. Ultimately, the framework offers a standardized way for researchers to determine when an LLM can serve as a reliable surrogate for human data versus when traditional data collection remains essential.