[HUMAN VOICE] "Meaning & Agency" by Abram Demski

Support ongoing human narrations of LessWrong's curated posts:www.patreon.com/LWCuratedThe goal of this post is to clarify a few concepts relating to AI Alignment under a common framework. The main concepts to be clarified:Optimization. Specifically, this will be a type of Vingean agency. It will split into Selection vs Control variants.Reference (the relationship which holds between map and territory; aka semantics, aka meaning). Specifically, this will be a teleosemantic theory.The main new concepts employed will be endorsement and legitimacy. TLDR: Endorsement of a process is when you would take its conclusions for your own, if you knew them. Legitimacy relates to endorsement in the same way that good relates to utility. (IE utility/endorsement are generic mathematical theories of agency; good/legitimate refer to the specific thing we care about.) We perceive agency when something is better at doing something than us; we endorse some aspect of its reasoning or activity. (Endorse as a way of achieving its goals, if not necessarily our own.)We perceive meaning (semantics/reference) in cases where something has been optimized for accuracy -- that is, the goal we endorse a conclusion with respect to is some notion of accuracy of representation.This write-up owes a large debt to many conversations with Sahil, although the views expressed here are my own.Source:https://www.lesswrong.com/posts/bnnhypM5MXBHAATLw/meaning-and-agencyNarrated for LessWrong by Perrin Walker.Share feedback on this narration.[Curated Post] ✓