AF - Concave Utility Question by Scott Garrabrant

<a href="https://www.alignmentforum.org/posts/uJnR4YmG5Kq9FfTey/concave-utility-question">Link to original article</a><br/><br/>Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concave Utility Question, published by Scott Garrabrant on April 15, 2023 on The AI Alignment Forum. This post will just be a concrete math question. I am interested in this question because I have recently come tor reject the independence axiom of VNM, and am thus playing with some weaker versions. Let Ω be a finite set of deterministic outcomes. Let L be the space of all lotteries over these outcomes, and let ⪰ be a relation on L. We write A∼B if A ⪰ B and B ⪰ A. We write A≻B if A⪰B but not A∼B. Here are some axioms we can assume about ⪰: A1. For all A,B∈L, either A⪰B or B⪰A (or both). A2. For all A,B,C∈L, if A⪰B, and B⪰C, then A⪰C. A3. For all A,B,C∈L, if A⪰B, and B⪰C, then there exists a p∈[0,1] such that B∼pA+(1−p)C. A4. For all A,B∈L, and p∈[0,1] if A⪰B, then pA+(1−p)B⪰B. A5. For all A,B∈L, and p∈[0,1], if p>0 and B⪰pA+(1−p)B, then B⪰A. Here is one bonus axiom: B1. For all A,B,C∈L, and p∈[0,1], A⪰B if and only if pA+(1−p)C⪰pB+(1−p)C. (Note that B1 is stronger than both A4 and A5) Finally, here are some conclusions of successively increasing strength: C1. There exists a function u:L[0,1] such that A⪰B if and only if u(A)≥u(B). C2. Further, we require u is quasi-concave. C3. Further, we require u is continuous. C4. Further, we require u is concave. C5. Further, we require u is linear. The standard VNM utility theorem can be thought of as saying A1, A2, A3, and B1 together imply C5. Here is the main question I am curious about: Q1: Do A1, A2, A3, A4, and A5 together imply C4? [ANSWER: NO] (If no, how can we salvage C4, by adding or changing some axioms?) Here are some sub-questions that would constitute significant partial progress, and that I think are interesting in their own right: Q2: Do A1, A2, A3, and A4 together imply C3? [ANSWER: NO] Q3: Do C3 and A5 together imply C4? [ANSWER: NO] (Feel free to give answers that are only partial progress, and use this space to think out loud or discuss anything else related to weaker versions of VNM.) EDIT: AlexMennen actually resolved the question in the negative as stated, but my curiosity is not resolved, since his argument is violating continuity, and I really care about concavity. My updated main question is now: Q4: Do A1, A2, A3, A4, and A5 together imply that there exists a concave function u:L[0,1] such that A⪰B if and only if u(A)≥u(B)? [ANSWER: NO] (i.e. We do not require u to be continuous.) This modification also implies interest in the subquestion: Q5: Do A1, A2, A3, and A4 together imply C2? EDIT 2: Here is another bonus axiom: B2. For all A,B∈L, if A≻B, then there exists some C∈L such that A≻C≻B. (Really, we don't need to assume C is already in L. We just need it to be possible to add a C, and extend our preferences in a way that satisfies the other axioms, and A3 will imply that such a lottery was already in L. We might want to replace this with a cleaner axiom later.) Q6: Do A1, A2, A3, A5, and B2 together imply C4? [ANSWER: NO] EDIT 3: We now have negative answers to everything other than Q5, which I still think is pretty interesting. We could also weaken Q5 to include other axioms, like A5 and B2. Weakening the conclusion doesn't help, since it is easy to get C2 from C1 and A4. I would still really like some axioms that get us all the way to a concave function, but I doubt there will be any simple ones. Concavity feels like it really needs more structure that does not translate well to a preference relation. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

First published

04/15/2023

Genres:

education

Listen to this episode

0:00 / 0:00

Summary

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concave Utility Question, published by Scott Garrabrant on April 15, 2023 on The AI Alignment Forum. This post will just be a concrete math question. I am interested in this question because I have recently come tor reject the independence axiom of VNM, and am thus playing with some weaker versions. Let Ω be a finite set of deterministic outcomes. Let L be the space of all lotteries over these outcomes, and let ⪰ be a relation on L. We write A∼B if A ⪰ B and B ⪰ A. We write A≻B if A⪰B but not A∼B. Here are some axioms we can assume about ⪰: A1. For all A,B∈L, either A⪰B or B⪰A (or both). A2. For all A,B,C∈L, if A⪰B, and B⪰C, then A⪰C. A3. For all A,B,C∈L, if A⪰B, and B⪰C, then there exists a p∈[0,1] such that B∼pA+(1−p)C. A4. For all A,B∈L, and p∈[0,1] if A⪰B, then pA+(1−p)B⪰B. A5. For all A,B∈L, and p∈[0,1], if p>0 and B⪰pA+(1−p)B, then B⪰A. Here is one bonus axiom: B1. For all A,B,C∈L, and p∈[0,1], A⪰B if and only if pA+(1−p)C⪰pB+(1−p)C. (Note that B1 is stronger than both A4 and A5) Finally, here are some conclusions of successively increasing strength: C1. There exists a function u:L[0,1] such that A⪰B if and only if u(A)≥u(B). C2. Further, we require u is quasi-concave. C3. Further, we require u is continuous. C4. Further, we require u is concave. C5. Further, we require u is linear. The standard VNM utility theorem can be thought of as saying A1, A2, A3, and B1 together imply C5. Here is the main question I am curious about: Q1: Do A1, A2, A3, A4, and A5 together imply C4? [ANSWER: NO] (If no, how can we salvage C4, by adding or changing some axioms?) Here are some sub-questions that would constitute significant partial progress, and that I think are interesting in their own right: Q2: Do A1, A2, A3, and A4 together imply C3? [ANSWER: NO] Q3: Do C3 and A5 together imply C4? [ANSWER: NO] (Feel free to give answers that are only partial progress, and use this space to think out loud or discuss anything else related to weaker versions of VNM.) EDIT: AlexMennen actually resolved the question in the negative as stated, but my curiosity is not resolved, since his argument is violating continuity, and I really care about concavity. My updated main question is now: Q4: Do A1, A2, A3, A4, and A5 together imply that there exists a concave function u:L[0,1] such that A⪰B if and only if u(A)≥u(B)? [ANSWER: NO] (i.e. We do not require u to be continuous.) This modification also implies interest in the subquestion: Q5: Do A1, A2, A3, and A4 together imply C2? EDIT 2: Here is another bonus axiom: B2. For all A,B∈L, if A≻B, then there exists some C∈L such that A≻C≻B. (Really, we don't need to assume C is already in L. We just need it to be possible to add a C, and extend our preferences in a way that satisfies the other axioms, and A3 will imply that such a lottery was already in L. We might want to replace this with a cleaner axiom later.) Q6: Do A1, A2, A3, A5, and B2 together imply C4? [ANSWER: NO] EDIT 3: We now have negative answers to everything other than Q5, which I still think is pretty interesting. We could also weaken Q5 to include other axioms, like A5 and B2. Weakening the conclusion doesn't help, since it is easy to get C2 from C1 and A4. I would still really like some axioms that get us all the way to a concave function, but I doubt there will be any simple ones. Concavity feels like it really needs more structure that does not translate well to a preference relation. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Duration

4 hours and 33 minutes

Parent Podcast

The Nonlinear Library: Alignment Forum Daily

View Podcast

Share this episode

Similar Episodes

AMA: Paul Christiano, alignment researcher by Paul Christiano

Release Date: 12/06/2021

Description: Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AMA: Paul Christiano, alignment researcher, published by Paul Christiano on the AI Alignment Forum. I'll be running an Ask Me Anything on this post from Friday (April 30) to Saturday (May 1). If you want to ask something just post a top-level comment; I'll spend at least a day answering questions. You can find some background about me here. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Explicit: No

Details

What is the alternative to intent alignment called? Q by Richard Ngo

Release Date: 11/17/2021

Description: Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is the alternative to intent alignment called? Q, published by Richard Ngo on the AI Alignment Forum. Paul defines intent alignment of an AI A to a human H as the criterion that A is trying to do what H wants it to do. What term do people use for the definition of alignment in which A is trying to achieve H's goals (whether or not H intends for A to achieve H's goals)? Secondly, this seems to basically map on to the distinction between an aligned genie and an aligned sovereign. Is this a fair characterisation? (Intent alignment definition from) Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Explicit: No

Details

AI alignment landscape by Paul Christiano

Release Date: 11/19/2021

Description: Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI alignment landscape, published byPaul Christiano on the AI Alignment Forum. Here (link) is a talk I gave at EA Global 2019, where I describe how intent alignment fits into the broader landscape of “making AI go well,” and how my work fits into intent alignment. This is particularly helpful if you want to understand what I’m doing, but may also be useful more broadly. I often find myself wishing people were clearer about some of these distinctions. Here is the main overview slide from the talk: The highlighted boxes are where I spend most of my time. Here are the full slides from the talk. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Explicit: No

Details

Would an option to publish to AF users only be a useful feature?Q by Richard Ngo

Release Date: 11/17/2021

Description: Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Would an option to publish to AF users only be a useful feature?Q , published by Richard Ngo on the AI Alignment Forum. Right now there are quite a few private safety docs floating around. There's evidently demand for a privacy setting lower than "only people I personally approve", but higher than "anyone on the internet gets to see it". But this means that safety researchers might not see relevant arguments and information. And as the field grows, passing on access to such documents on a personal basis will become even less efficient. My guess is that in most cases, the authors of these documents don't have a problem with other safety researchers seeing them, as long as everyone agrees not to distribute them more widely. One solution could be to have a checkbox for new posts which makes them only visible to verified Alignment Forum users. Would people use this? Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Explicit: No

Details