4. Controlling Superintelligence: Capability and Motivation Methods

This explores the control problem associated with the creation of artificial superintelligence, defining it as a unique principal-agent challenge where humans seek to govern the powerful AI. The sources differentiate between two main control strategies: capability control, which limits what the AI can do through methods like boxing or tripwires, and motivation selection, which focuses on shaping what the AI wants to do through techniques like direct specification or indirect normativity. Furthermore, the discussion introduces four conceptual categories for superintelligence—oracles, genies, sovereigns, and tools—assessing the safety implications of each based on their susceptibility to these control methods. Finally, the text briefly addresses the challenges of a multipolar scenario with multiple competing AIs, contrasting this outcome with a single dominant superintelligence, or singleton, and considering the potential for a singleton to emerge even after an initial multipolar transition due to competitive dynamics.