Clio: Privacy-Preserving Insights into Real-World AI Use, by Jonathan H. Westover PhD

Abstract: This paper presents Clio (Claude insights and observations), a privacy-preserving platform that uses AI assistants to analyze and surface aggregated usage patterns across millions of conversations without requiring human reviewers to read raw user data. The system addresses a critical gap in understanding how AI assistants are used in practice while maintaining robust privacy protections through multiple layers of safeguards. We validate Clio's accuracy through extensive evaluations, demonstrating 94% accuracy in reconstructing ground-truth topic distributions and achieving undetectable levels of private information in final outputs through empirical privacy auditing. Applied to one million Claude.ai conversations, Clio reveals that coding, writing, and research tasks dominate usage, with significant cross-language variations—for example, Japanese conversations discuss elder care at higher rates than other languages. We demonstrate Clio's utility for safety purposes by identifying coordinated abuse attempts, monitoring for unknown risks during high-stakes periods like capability launches and elections, and improving existing safety classifiers. By enabling scalable analysis of real-world AI usage while preserving privacy, Clio provides an empirical foundation for AI safety and governance. Learn more about your ad choices. Visit megaphone.fm/adchoices