“Human-Guided Agentic Research: A Research Agenda” by fastfedora

tl;dr: As recursive self-improvement accelerates, we need a top-level agenda to research how to effectively keep humans in the loop. We need to study how humans can best interpret and guide research performed by autonomous agents when those agents lack taste, tacit knowledge or competence, or may try to reward hack, sandbag or sabotage such research. This is one attempt to define the problem and the shape of potential solutions. A Story About the Future of Research Imagine yourself a year or two in the future. Recursive self-improvement (RSI) is accelerating. Agents work in swarms independently for days or weeks at a time doing research. You work in a frontier lab doing AI safety research. You sit in front of your computer and click into the input box, ready to kick off a new project. What do you type? “Solve AI alignment”? Beware giving a magic genie vague wishes. Think about that again: what exactly do you type? How do you know what you type is the best way to prompt this agent swarm into doing your bidding? When the lead agent comes back a week later, what exactly does that output look like? How do you use that [...] ---Outline:(00:49) A Story About the Future of Research(02:21) Recursive Self-Improvement Is Here(04:27) The Problem(04:30) Destination-Focused (Convergent) Research(05:52) Direction-Focused (Divergent) Research(08:33) Destination-Focused vs Direction-Focused Research(10:20) The Agenda(11:52) The Threat Model(14:15) Relation to Existing Agendas(14:36) Scalable Oversight(15:30) AI Control(16:37) Cooperative AI(17:28) Research Directions(18:10) Frameworks(20:36) Infrastructure(22:58) Interfaces(30:51) Initial Objections(30:55) Labs Will Develop These Tools & Techniques(31:30) Industry Will Develop These Tools & Techniques(32:15) Agents Can Dynamically Generate UI(33:11) Agents May Acquire Taste(33:51) Humans Will Slow Things Down(34:36) Summary The original text contained 3 footnotes which were omitted from this narration. --- First published: June 29th, 2026 Source: https://www.lesswrong.com/posts/8KrTuCAzL2fdYHNrv/human-guided-agentic-research-a-research-agenda --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.