Building General User Models from Computer Use

This episode explores the UIST 2025 paper "Creating General User Models from Computer Use," which proposes building a persistent user model from raw computer traces such as screenshots, UI text, message context, and app switching. It explains how the system stores confidence-weighted natural-language propositions about a person’s preferences, knowledge, goals, and current situation, aiming to support cross-application assistants that can help proactively rather than waiting for explicit requests. The discussion situates the idea against earlier recommender systems, Bayesian user modeling, and newer LLM memory architectures, arguing that the paper is most interesting as a synthesis of HCI user modeling and retrieval-and-revision style AI memory. Listeners would find it compelling because it gets concrete about both the upside of more context-aware assistants and the hard problems underneath them, including noisy behavioral data, narrow evidence relative to broad claims, and the privacy risks of inferring things users never said aloud. Sources: 1. Creating General User Models from Computer Use — Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyi Yang, Michael S. Bernstein, 2025 http://arxiv.org/abs/2505.10831 2. User Modeling via Stereotypes — Elaine Rich, 1979 https://scholar.google.com/scholar?q=User+Modeling+via+Stereotypes 3. The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users — Eric J. Horvitz, John S. Breese, David Heckerman, David Hovel, Koos Rommelse, 1998 https://scholar.google.com/scholar?q=The+Lumiere+Project%3A+Bayesian+User+Modeling+for+Inferring+the+Goals+and+Needs+of+Software+Users 4. Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions — Gediminas Adomavicius, Alexander Tuzhilin, 2005 https://scholar.google.com/scholar?q=Toward+the+Next+Generation+of+Recommender+Systems%3A+A+Survey+of+the+State-of-the-Art+and+Possible+Extensions 5. User Modeling and User Profiling: A Comprehensive Survey — Erasmo Purificato, Ludovico Boratto, Ernesto William De Luca, 2024 https://scholar.google.com/scholar?q=User+Modeling+and+User+Profiling%3A+A+Comprehensive+Survey 6. Generative Agents: Interactive Simulacra of Human Behavior — Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, Michael S. Bernstein, 2023 https://scholar.google.com/scholar?q=Generative+Agents%3A+Interactive+Simulacra+of+Human+Behavior 7. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023 https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems 8. Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration — Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, Diyi Yang, 2024 https://scholar.google.com/scholar?q=Collaborative+Gym%3A+A+Framework+for+Enabling+and+Evaluating+Human-Agent+Collaboration 9. Supporting Physical Activity Behavior Change with LLM-Based Conversational Agents — Matthew Jörke, Shardul Sapkota, Lyndsea Warkenthien, Niklas Vainio, Paul Schmiedmayer, Emma Brunskill, James Landay, 2024 https://scholar.google.com/scholar?q=Supporting+Physical+Activity+Behavior+Change+with+LLM-Based+Conversational+Agents 10. TaskTracer: a desktop environment to support multi-tasking knowledge workers — Anton N. Dragunov, Thomas G. Dietterich, Kevin Johnsrude, Matthew McLaughlin, Lida Li, Jonathan L. Herlocker, 2005 https://scholar.google.com/scholar?q=TaskTracer%3A+a+desktop+environment+to+support+multi-tasking+knowledge+workers 11. UI-TARS: Pioneering Automated GUI Interaction with Native Agents — Yujia Qin et al., 2025 https://arxiv.org/abs/2501.12326 12. Need Help? Designing Proactive AI Assistants for Programming — Valerie Chen et al., 2024/2025 https://arxiv.org/abs/2410.04596 13. Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistants — Deepak Nathani et al., 2026 https://arxiv.org/abs/2604.00842 14. ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulation — Jiho Kim et al., 2025/2026 https://arxiv.org/abs/2509.21730 15. AI Post Transformers: Learning Latent Action World Models from Video — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-09-learning-latent-action-world-models-from-1570a4.mp3 16. AI Post Transformers: PaperBench: Can AI Replicate AI Research? — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-06-17-paperbench-can-ai-replicate-ai-research-862944.mp3 17. AI Post Transformers: Reasoning Theater and Unfaithful Chain-of-Thought — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-05-reasoning-theater-and-unfaithful-chain-o-a4507e.mp3 18. AI Post Transformers: Neural Computers as Learned Latent Runtimes — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-11-neural-computers-as-learned-latent-runti-9fa282.mp3 Interactive Visualization: Building General User Models from Computer Use