The Illusion of Learning from Observational Data: An Empirical Bayes Perspective

This paper addresses the "illusion of learning" in causal inference, where combining observational data with randomized experiments fails to improve accuracy because the bias distribution of observational studies is unknown. The authors demonstrate that while standard empirical Bayes methods often fail to resolve this, the inclusion of calibration studies—observational research on interventions with known zero effects—allows researchers to identify and adjust for systematic bias. By learning the mean and variance of these biases through calibration, researchers can use shrinkage estimators to meaningfully combine diverse data sources. The proposed calibrated empirical Bayes procedure achieves consistent causal recovery and reduces estimation risk as the number of studies increases. This framework is validated through simulations and a real-world application involving water-usage field experiments. Ultimately, the research provides a statistically rigorous method to unlock the value of large-scale observational data to supplement expensive or limited randomized trials.