Malte Jansen/ Aishvarya Aravindan Rajagopal: The Reproducibility and Robustness of Secondary Analyses in Educational Research: The Role of Publication Bias and Researcher Degrees of Freedom

Many educational researchers conduct secondary data analysis using large-scale school assessment studies that usually include various variables based on representative samples. To access such data, researchers must often apply by submitting a research proposal. Our project aims to examine the reproducibility and robustness of secondary data analyses from a research data center that offers over 70 educational studies for secondary analyses. This approach provides us with a unique database of data usage applications. In these applications, researchers describe their central questions, hypotheses, and planned analytic approach. Between 2008 and 2020, around 600 data applications from over 900 researchers resulted in around 180 publications. Based on this data and an additional survey of applicants about their project results, we will examine which data applications result in publications. Second, we will reproduce the published results of a selected subsample of data applications by using the information given by researchers in the publication. This direct reproduction of study results might provide hints on improving transparent descriptions of the research process. Third, we will explore the heterogeneity in effect estimates introduced by different analytical strategies and datasets using robustness checks on a smaller subsample of publications. The talk will overview the project and present the first results.