Human Expertise in Algorithmic Prediction

This research introduces a framework for integrating human expertise into algorithmic predictions, specifically focusing on instances where algorithms deem inputs "indistinguishable." The authors propose a method for selectively incorporating human judgment in these cases, demonstrating its proven ability to enhance the performance of any feasible algorithmic predictor. Empirical studies, including X-ray classification and visual prediction tasks, reveal that even when algorithms generally outperform humans, human input significantly improves predictions on specific, identifiable instances, which can constitute a substantial portion of the data. Furthermore, the paper explores how this framework can lead to algorithms that are robust to varying levels of user compliance, providing near-optimal predictions even when users selectively defer to the algorithm. Ultimately, the work advocates for human-AI collaboration to mitigate algorithmic monoculture by leveraging diverse human perspectives in prediction tasks.