4 - Risks from Learned Optimization with Evan Hubinger

Today, I'll be talking to Evan Heavinger about risks from learned optimization. Evan is a research fellow at the Machine Intelligence Research Institute, was previously an intern at AI, working on theoretical AI safety research with Paul Christiana. He's done software engineering work at Google, Yolcan, and also designed the functional programming language. We're going to be talking about the paper risks from learned optimization in advanced machine learning systems.

The authors are Evan Heavinger, Chris Van Merweig, Flodie Merriek, Yor Schausa, and Scott Garabrand. So Evan, welcome to the actor. Thank you for having me, Daniel. Yeah, so the first thing I wanted to say about this paper is it has a nice, a glossary at the end with like what various terms mean.

And I think this is like sort of unusual in machine learning papers. So just straight off of that, I wanted to firstly thank you for that, and let potential readers of the paper know that it's maybe easier to read than I think. Yeah, thank you. Yeah, we were talking on that.

I think one of the things that we sort of set out to do in the paper was define a lot of things, and create a lot of terminology. And so in creating a lot of terminology, it's useful to have a reference with that terminology. And so we figured that would be helpful for people trying to go back and look at all the terms that we're using. In terms of all the time, it's not necessarily the case that we use in our paper, but we use in the future, or people might use it differently.

But we wanted to have a clear reference for we're going to use a lot of words. Here are the words, like here's what they mean, here's how we're finding them, and here's how we're using them. So at least if you're looking at the paper, you could read any context to understand what it's saying, and what the terms mean. Yeah, that makes sense.

So just a heads up for this for the listeners. In this episode, I'm mostly going to be asking you questions about the paper, rather than walking through the paper. That being said, just to start off with, I was wondering if you could briefly summarize like what's the paper about and what's it trying to do? Yeah, for sure.

So essentially the paper is looking at trying to understand what happens when you train a machine learning system, and you get some model. And that model is itself doing optimization. It is itself trying to do some search, it's trying to accomplish some objective. And it's really important to distinguish.

And this is one of the main distinctions that the paper tries to draw. So to draw is the distinction between the loss function that you train your model on, and the actual objective that your model learns to sort of try to follow an internal. And so we make a bunch of distinctions, we try to find these terms. We're specifically talking about these sorts of two align problems.

The outer line problem, which is the problem of aligning the human goals that they're trying to get to do with the loss function. And the inner line problem, which is how do you align the actual objective that your model might end up pursuing with the loss function? And we call this sort of actual objective, the MESA objective. And we call a sort of model, which is itself doing optimization on MESA optimizer.

What's MESA, by the way? Yes, that's a very good question. So MESA is prefix, and it's sort of the opposite of MESA. So MESA is pre-versum above.

MESA is sort of pre-versum below. The idea is that if you can think about sort of meta-oppensation in the context of machine learning, meta-oppensation is when I have some sort of optimizer, eight, iterating set process, sort of training process, whatever. And I create another optimizer above that optimizer, a meta-oppimizer, sort of optimize over it. And if you want to look at the opposite of this, we want to look at the situation in which I have a training process.

I have a sort of loss function, and I have a sort of gradient set process. What happens when that gradient set process itself sort of becomes a meta-oppimizer? What happens when the thing which is optimizing over the model, the actual sort of training, or the work, is self-developing in the optimization procedure? And so we want to call that MESA optimization, where it's sort of the opposite of meta-oppensation instead of having an optimization level above, where it's sort of expecting to have an optimization level below.

Cool. So one question I have about this paper is, like you're thinking about optimization. Like where's optimization happening? You know, what's it doing?

What's optimization? Yes, that's a great question. And I think this is one of the sorts of things that it's notoriously difficult to find. And there's a lot of sort of definitions that have existed out there, different people trying to define the different ways.

You have things like Daniel Dennis, like TensorFlow stance, you have things like how this would be something that I sort of put some time from going into this. In risk mode optimization, we sort of define optimization in a very sort of mechanistic way, where we're like, look, a system is an optimizer, if it is internally doing some sort of search according to some objective, searching through a space of possible plans, strategies, policies, actions, whatever. And the argument for this is, well, once you have a model which is doing search, it has the large space of things, and it's trying to look over those things and select them according to some criteria, that criteria has become a objective of sorts. Because now, it is selecting the way in which it's output is produced, has a large amount of sort of optimization power, a large amount of sort of selection, going into choosing which things satisfy this criteria.

And so we really want to be the case that wherever that criteria that exists there is one that you know what they'd be like. And furthermore, the sort of thing which we argue is we should actually expect, that you will learn models which have to sort of search, because search generalizes really well, it's a relatively simple procedure. And then finally, also, search how many sort of mechanistic definition lets us sort of make a lot of arguments and sort of get to a point where we can understand things in a way where if we just adopted a sort of behavioral definition, like intentional stance, for example, we would be able to do. Because we want to be able to say things like, how likely is it when I run a machine learning process, when I do a training process, I will end up with a model that is sort of an optimizer.

And it's very difficult to say that if you don't know what an optimizer looks like internally. Because I can say, well, how likely is it that I'll get something which is well described as an optimizer. And that's a much more difficult question to answer, because you don't know sort of, you know, the process of reading is that the process of machine learning, it's like something internal, it's like on, you know, you know, we have all of these sort of implicit vector biases, which might be pushing you in one direction, and then you have this sort of loss function which is trying to ensure that the behavior of the model has, you know, figured properties. And to be able to answer whether or not those vector biases are going to push you in one question, or you need to know things like how simple is the, you know, the thing which you're looking for according to those vector biases, how likely is it that, you know, the process of reading the sentence or you end up with something set aside.

And I think the sort of point where we end up was, not that there's something wrong necessarily with this sort of behavioral stance of optimization, but just that it's very difficult to actually understand, from a concrete perspective, the extent to which you will get optimizers when you're teaching learning, if you're only thinking of optimizers behaviorally, because you don't really know to what extent those optimizers will be simple, what extent will they be favored by reading the sentence. So by having a mechanism definition, we can actually sort of investigate those questions and much more concrete. So one thing that strikes me there is like, if I were thinking about what's easiest to learn, about what, you know, what neural networks will be pretty spigrading descent, I would think like, well, you know, grading descent, like it's mostly selecting a behavior, right? It's like, okay, is this, how can I tweak this neural network to produce like behavior that's more like what I want, or you know, outputs or whatever is appropriate to the relevant problem.

And famously, we currently like, don't, can't say very much about like, how the internals of neural networks work. Hopefully that's changing, but not that much. So it strikes me that it might actually be easier to say, like, well, behaviorally, we know we're going to get things to behave, to optimize an objective function, because like, you know, that's the thing that is optimal for the objective function. So I was wondering like, like, it seems like in your response, you put a lot of weight on like, well, you know, networks will select simple, or optimization will select simple networks or simple algorithms.

And I'm wondering like, why do you have that kind of emphasis instead of emphasis on like, selecting for behavior? Yes, that's a really great question. So I can answer this comes from, so first of all, I totally agree. And I think that in fact, a problem that the way in which we currently do machine learning is so focused on behavior incentives.

One of the things that in my research, I actually sort of have spent a lot of time looking into and trying to figure out is how we can properly do mechanistic incentives, how it can create ways to train neural networks that are focused on making them more transparent, or making them, you know, able to sort of be satisfied by some sort of internal properties in my work. That being said, the reason that you can't, I think it's very difficult to sort of analyze this question behaviorally is because fundamentally, we're looking at a problem of unidentifiably. If you only look at behavior, you can't distinguish between the model, which is internally like doing some opposition process, it has some objective and like, produces the sort of like, particular results versus sort of model, which doesn't have any sort of internal objective. It's just sort of produces the same, but it's still produces the same results on the training process, which has a bunch of like, heuristics.

But those two models have very different generalization behavior. And it's the generalization behavior that we care about. We're looking for situations in which, and we sort of talk about this a little bit, there's this, this is posed by a plan that they had, that sort of talks about this sort of 2D robustness picture, where the thing that we're fundamentally concerned about is the situation where you have a model, and the model is very robust in the sense of competitiveness, sort of it's very sort of capability robust. It has the ability to pursue whatever objective is pursuing very effectively, off-discovery and on-discovery.

But it's not very objective robust. So it learns some sort of proxy-effective making, it works sort of pretty well in the training distribution, but it doesn't work very well off-discovery. And the problem is that if you're just looking at the behavior on the training distribution, you can't really distinguish between a model which like will be objective robust, which like has learned a proxy, which like will work well off-discovery, and a model which has learned a proxy, which won't work well off-discovery. Because if you're just looking at the training distribution, and you're just looking at a behavior on the training distribution, you can't tell because those might just look identical on the training distribution.

As a really comfy example, this sort of amazing example that I often like to get. So you have a situation where I'm training a model on a training distribution as a bunch of sort of mazes, and I want it to get to the end of the maze. I put a little green arrow at the end of the maze, and instead of as a model for getting to the end, I train how much it'd be relatively small mazes. And then I deploy the model in a new situation where I have larger mazes, and I put the green arrow at some random place in the middle.

And there's a bunch of things that can happen. So it could be the case that in this sort of new deployment environment, the model is sort of capability robust, and it really knows how to solve mazes. But it can also not be capability robust, and it just cancels all the bigger mazes. But if it is capability robust, then there's sort of two options.

It's either objective robust, and it's one of the correct processes, which is getting to the end. So it makes it, it navigates the end of maze. Or it could be objective non-rebus, and it learned to be incorrect proxy, which gets the green arrow. So it navigates the green arrow.

And if you're only looking at behavior on the training distribution, you can't distinguish between the model which is trying to get to the green arrow and the model which is trying to get to the end of maze, because the training distribution, those are sort of not identifiable. You can't sort of distinguish between them by the behavior. And so if you want to ask the question, how liking is it, then we will end up with models which generalize in one way versus models which are generalized in another way. We have to look at the other thing, then just the behavior incentives, which is causing models to be all into one category of the other, and that is the inductive biases.

The reason that your training process would produce a model which is going to generalize in one way, or generalize in another way, is by having the same training performance, is all the way to come out to what is incentivized by the inductive biases of that training process. And to understand what's incentivized by the inductive biases of training process, we have to look at the models of terms. Because the inductive biases aren't just about the, the, what's happening here at this point, the inductive biases, all of the behavior on the training distribution is the same. So the only difference is what are the inductive biases incentivizing in terms of the internal level of the model?

Are the inductive biases going to incentivize the source of models to have the properties or models of other properties? And fundamentally that's what we have to look at if you want to start answering the question of what you're going to get in a standard or setting, because that's the sort of thing we're just going to determine what option you end up with when you're in this sort of un-identifiable position. Yeah, so it sounds like, like one thing that you notice reading this paper is that it seems very focused on the case of distributional shift, right? Like, like you're interested in generalization, but not in the like generalization balance sense of like, you know, from the training distribution to the training distributions, but simply I'm literally seeing, but you know, some kind of like different world.

And like, why, like, why can't we just train on the distribution that we're going to deployment? Why is this a problem? Yes, that's a really good question. So I think there's a couple of things about here.

So first, one thing that's worth noting is that I think a lot of times, I mean, so people read the paper and be like, well, what is going on with this sort of deployment distribution? Sometimes we just do online learning. Sometimes we just do like, you know, situations where we're just going to keep training. We're just going to keep going.

Why can't we sort of understand? I think basically my argument is that those situations are not so dissimilarizing my thing. So let's, you know, for example, consider online learning. I'm going to try and just keep training it on, you know, every data point as it comes in and keep sort of pushing in different directions.

Well, let's consider the situation where you have your joint online learning and the modeling captures some new data. We still have to ask the question, like, for this new data point, will it generalize properly? Or will it not? And we can answer the question, really generalize properly, but you're not honest with the data point.

By looking at all the previous training that we've done was training the environment of sorts. And this new data point is a deployment environment of sorts where, you know, something potentially has changed from the previous data points to this new data point. And we want to ask the question, will it generalize properly, is this new data point? Now, fun and there's a generalization problem is not changed by the fact that after it produces some action on the data point, we'll go back and do some additional training on it.

If there's still a question on how we're going to generalize this new data point, and so you can think of online learning set-up situation where we're trying to keep training in on all the data, as basically all they're doing is they're adding additional feedback. They're making it so that if the model does do something quarterly, maybe we have the ability to sort of have the training process and interact that. But fun and that's always the case, even if you're doing a deployment where you're not doing online learning. If you notice that the model is doing something bad, there are still feedback mechanisms you can try to shut it down.

You can try to look at what's happening and retrain it up. And you can always feedback mechanisms that exist. And if we sort of take a higher level of sap and think about what is the real problem that we're trying to solve, the real problem we're trying to solve is situations in machine learning models take actions that are so bad that they're under coverable. That are sort of natural feedback mechanisms that we establish don't have the ability to sort of account for and correct for the sort of mistake that is in it.

And so you know an example of a situation like this, Paul talks a lot about this, Paul Giano talks a lot about this sort of unrecoverability. One example would be Paul has this post what failure looks like, and in the way that looks like also the post of the situation, we could imagine for example a sort of cascade of problems where you know one of the things that are happening in the future. And so you know one of the problems goes off the rails because it has some proxy and a certain authenticity. And this goes to other models go off the rails and you sort of get a cascade of sort of problems that creates unrecoverable situations.

You know you end up with a society where we put models in charge of all these different things and then you know as one goes off the rails it pushes others off the rails because as soon as you get one sort of model doing something problematic, that now puts everyone else into situation where they have to deal with that problematic thing. And that could itself be off the rails from the things we just had to do previously which increases the probability because other models fail and you get that sort of cascade. I'm not saying this is the only, I think this is one possible failure mode, but if you can see that it's sort of one possible failure mode that is demonstrating the way in which like the real problem is situations in which our feedback mechanism is an assumption. And you will always have situations where your feedback mechanism is potentially an issue, you know, does something truly sort of catastrophic.

And so what we're trying to do is doing something so abstracting the first ones. And to do that you have to pay attention to the generalization because every time it looks a new data point, you're always relying on generalization behavior to sort of do the correct thing on that. So, so when you say generalization behavior, you mean like, you know, it's going to encounter a situation and I guess I'm not sure why like just normal ML isn't enough like generalization bounds exist, you know, like why can't we just, you know, train on distribution, deploy, you get different samples from your distribution so it's a little bit different, but like generalization bounds say that like, you know, it's just not going to be so bad. Why isn't that like the end of the story.

I think a lot of these generalization bounds rely on a particular assumption, which I think is quite questionable, which is the idea such that the assumption that in fact, like there, the one institution and the like the training solution can actually create like take ID samples that there's like them can create like a particular balance between like the like divergence between the distributions or like things like this. In practice, all these sorts of assumptions just I think basically just double and there's ways in which like there's a lot of sort of like obviously we're taking a hold, you know, there's always things that are sort of changing about our environment. And so, you know, you can imagine, you know, just the fact that like the data advances the fact that like, you know, various things are happening in the world, you might not be able to anticipate. And obviously there are ways to try to address these problems in the standard practice of ship learning.

So, you know, a very standard thing to do is to do is to sort of train. So you might have a situation where you're like, well, we expect that the distribution is going to vary in a much different ways. We can try to sample some of the ways to expect distribution are vary in a very interesting sense. But there can still be a lot of problems that happen even that situation.

So one of the things that we talk about in the paper, I think it sort of becomes particularly problematic. So, when you start getting into situations where you've really done a good job of doing out of zero training and like tried to train away a lot of the sort of naive sort of civil ways in which you might end up with, you know, a green arrow optimizer versus end of the base optimizer. It's something that we call the second line. And the second line is a sort of particularly perdicious case of a sort of mesa optimizer which has a base objective that is sort of different from the ideal law section.

And the particular case of the second line is a situation in which we have a model and that model that has some proxy objective, you know, it's trying to get the green arrow there. But it realizes that it's inside of a training process and that that training process is going to select it for getting to the end of the base rather than getting to the green arrow. And so it realizes that the correct instrumental strategy to like be able to get the green arrow in the future is to currently try to get into the base so that it will like stay around the training process and will be modified to like go to the green arrow instead. And then eventually at some point maybe it gets the ability to like take some action like maybe you just avoid it.

And now what is in deployment, it like can go to green arrow a bunch of times and you won't like correct for it. Or it otherwise gets into some situation where the correction mechanisms that you have or no longer sufficient to prevent it from being able to do like the true process. And this is particularly concerning because if you're in a definite line of situation, relax, so the sort of adversarial training set no longer really works because if you think about, you know, a definite line system, it's going to be actively looking for any sort of possible sort of differences between the train distribution and the point distribution to let know that now it's a situation where it sort of do it once and it doesn't have to be as very also corrected feedback. And so there's a lot of examples of sorts of things that you know, a sort of deceptive model I'm looking for.

Paul Greschon has a sort of example that is kind of sort of fun. And it's not, you know, necessarily realistic but it just goes to show the sort of way in which this can happen which is this RCA 24-8 example. So RCA 24-8 example goes like this. You can imagine, so RCA 24-8 is this really large super prime.

There's this sort of challenge to try to factor RCA 24-8. Sorry, it's a super prime. It's like a product of two large primes. I think, yeah.

I'm using these for a lot of the wrong. I'm pretty sure I put it up. It's a product of two very large primes. But finding that factor is a big factor.

Finding the like two very large primes that make up this very large number is very challenging and computational difficult. And currently it's sort of beyond the computational abilities of like modern computers and like, I don't know, there's a challenge to try to factor it and nobody has yet to do so. But it's the sort of thing that like probably will have enough compute that somebody will eventually get around to like being able to factor it in the future. And so you can add to the situation.

You have this deceptive model and it's like, well, I don't know what I'm going to be in the point. But some point probably I'll be in a employment situation where like I have lots of power and I can like just sort of do whatever I want. And so I'll choose to look for a factorization of RCA 24-8. And if I see such a factorization then I can be pretty sure that I'm no longer in training.

Because the people training me don't have to compute available to factor RCA 24-8. But probably at some point somebody's gonna factor it. And so I can be pretty sure that if I see an act of RCA 24-8 that I'm no longer in training, I can be pretty sure that eventually I will see such a factorization. And so it's sort of a general sort of pointing this sort of general problem of when you try to do atmosphere training, you're putting yourself in this sort of situation where you have to be able to, it's sort of this P versus NP thing.

Where like you're putting yourself in a situation where you have to be able to actually like generate all the sort of possible counter-tamples. Whereas the model is a much easier decision. It only has to be able to do the check. It only has to be able to check whether it is in such a situation.

And there's just a lot of things where it's easier to check than it is to generate. So a factorization is an example of that. It's a lot easier to check that you have a factorization than it is to generate such a factorization. And this is sort of a general thing that comes from the fact that there are different complexity classes that vary based on, you know, like we can think about NP as being the complexity class of things which are usable.

And P as being the complexity class of things which are relatively easy to actually generate. And then these are, I don't know, we think that these are the same. So there's like lots of problems like this. Even if we're not thinking about things as rigorous mathematically as RCA 24-8, there's just a lot of things about the world that are easier to check than they are to generate.

And so that means that if you have a model which is actively looking for differences, your procedure of let's just try to generate a bunch of differences is no longer about the work. Yeah, I think this is, so there's a lot in that answer that I can get back to. One thing that it does remind me of is, yeah, people do talk about adversarial training. And it always strikes me that like, it seems like if you want to really make that work, you need much better interpretability tools than you have at the moment precisely to check this kind of thing.

Like what is the model checking? Like you're really finding the worst case scenario and you don't know a lot about the model, right? So I guess I wonder if you think that's right or do you have any thoughts about that? And then that's exactly right.

So I have a post that talks about relaxed adversarial training, which is terminology that comes from all the shadow. And basically the idea there is, well, how can we improve this adversarial training setup? Well, instead of doing adversarial training on the model's inputs, we can try to do it on the models in terms. We can try to sort of generate descriptions of situations in which the model might fail, try to generate sort of like possible sort of like activations for which the model might fail.

And these problems might be significantly easier than actually trying to generate an input out of some office. Cool. So let's watch and get back to you. I want to go back to basics a little bit, just about my optimizers.

So one of the examples you use in the post is like you can sort of think, at least for the sake of example, you can think of evolution as like some optimizer. And evolution is pretty as humans and humans are themselves optimizers for such a obsession. And I wanted to get a little bit more detail about that. So I'm a human.

And a lot of the time I'm like almost on what are I like? Or I don't like, for instance, I'll be sort of acting on some instinct, right? Like a lot of noise happens and I kind of jump. Sometimes I'll be doing what I previously decided to do or like maybe what somebody told me to do.

So for instance, we just arranged this interview and I know I'm supposed to be asking questions and like, you know, because we already penciled that into the calendar. That's what I'm doing. And often what I'm doing is like very local means and reasoning. So I'll be like, oh, I would like some food.

I'm hungry. So I guess I would like some food now. So I guess I'm going to cook some food. Like that kind of thing, which is different from what you might imagine of like, oh, I'm going to like plan.

I'm going to think of like all the possible sequences of actions I could take. And I'm going to search for like which one maximizes like how rich I end up in the future or, you know, some altruistic objectives maybe if you want to think nicely for me or something. So am I a masochomizer? And if I am, what's my masochative?

Yes, that's a really good question. So I mean, so I'm happy to answer this. I will like just warn. I think that my first look at him is that it's like, it's easy to go straight.

I think thinking too much about the sort of evolution of humans example, I'm tapping with humans and like forget that like humans are not machine learning models and like they're like a real difference is there. But I will try to answer the question. So I think that so if you think about it for most of a computational perspective, it's just not very efficient to try to spend a bunch of compute for every single action you're going to choose to try to compare it to like all of these different possible regimes. But that doesn't mean you're not doing search.

It doesn't mean you're not doing planning, right? So you can think about, you know, what ways of what you're doing might eat. Well, maybe you have a bunch of heuristics, but you change those risks over time. And the way you change those risks is by doing sort of careful search and planning something.

I don't know exactly what the correct way is described with human brain and like what algorithm internally runs and like how it operates. But I think it's a good guess to say and pretty reasonable to say it is clearly doing search is clearly doing planning is clearly at least sometimes, you know, like you're saying, it's computationally, you know, intractable to try to sort of do some master search every time, you know, you think about, you know, you're going to move a muscle or better. But it doesn't mean that you can't like on for particular actions particular strategies develop long term strategies think about sort of possibilities for like, you know, consider any different strategies to select them according to like whether they satisfy whatever you're trying to achieve. And, and that also means the second question which is what is the basic objective, what is it that you might be trying to achieve.

This is obviously very difficult to answer and people trying to answer the question what are like human values for forever essentially. I don't, I don't think I have the answer and I don't think that I don't like expect us to get the answer to this sort of thing. But I nevertheless think that like there is a meaningful sense in which you can say that humans clearly have goals and not just in a behavioral sense, in like a mechanistic sense. I think there's a sense in which you can be like, look, humans select strategies, they look at different possible things which they could do and they choose those strategies which like perform well, which like have good properties of those sorts of things they're looking for, you know, they try to choose strategies which, you know, put them in a good position and try to be strategies which like make the world a better place.

They try to be strategies which like, you know, people have lots of things that they use to select the different sorts of strategies that they pursue, different sorts of heuristics that use the different sorts of, you know, plans and policies that they don't. And so I think it is meaningful to say and correct to say that humans are based off of this form. Are they essential example? I don't know.

I think that they're like relatively non-central. I think that like they're not the like example I would want to like have first and foremost of like, based off of the station. But they're a meaningful example because they are a like the only example that we have of like a training process of like evolution being able to produce a like highly intelligent capable system. So they're useful to think about certainly though like we should also expect there to be differences.

Yeah, that makes sense. Yeah, that makes sense. Yeah, that makes sense. I guess I'm just asking to guess a sense of like what a mess optimizer really is.

And I guess your answer is interesting because it sort of suggests that like if I like, you know, do my start of your plan or whatever and like to search through some strategies and then like execute like I'm like even when I'm an execution lawyer, I'm still an optimizer. I'm still an optimizer, which is interesting because it sort of suggests that if like one of my clients is like, okay, I'm gonna play guard and then I like, you know, open up a game and then I start doing planning about like what means are gonna win. Am I a mess optimizer when I'm doing that. I mean, so my guess would be like not really you're still probably using exactly the same optimization machinery to do go as you are to like plan about whether you're going to go and so in some sense, it's not that you did.

So like for a basic optimization situation in which you internally did an additional search over possible like optimization strategies and possible like models to do opposition and then like if you're releasing the exact same optimization you're just one optimizer and you're like, you're optimizing for different points. Okay, that gives me a bit of a better sense of what I mean by the concept. So a few more questions about this. So one thing you do, I believe in the introduction of the paper is you distinguish between meso objectives and behavioral objectives.

I think that it's supposed to be that my meso objective is the objective that's like an internally represented in my brain. And my behavioral objective is the objective you could recover by asking yourself, okay, looking at my behavior, what am I optimizing for? In what situations would these things be different? Yeah, so that's a great question.

So I think that in most situations when you have a meaningful based objective, we would hope that your definitions are such that the behavioral objective would basically be that way subjective. But there's certainly exist models that don't have a meaningful meso objective. And so the behavioral objective is different in some sense of broader because you can look at a model which is an internally concerning search and you can still potentially ascribe to it some sort of objective that you could look like it looks like it is. What's the example of such a model?

I mean, you can imagine something like, you know, maybe you're looking at like, I think for example, a lot of like current reports are going on with them, especially my reverse learning models, my potential is this. So category, if you think about, I don't know, it's difficult to speculate on like exactly what the intervals of things are, but like guess would be you can do about something like opening I5 for example, that like it certainly looks like these listeners what I think is. Sure. Open I5 was sort of project by opening I2SOL-DODA to where they built agents that are able to sort of compete on a relatively sort of professional level that they would do.

And I think if you think about some of these agents, these look at the Open I5 agents, I think that my guess would be that Open I5 agents don't have like very coherent sort of internally represented objectives. They probably have a lot of search mistakes. Potentially there's some sort of search going on, I guess would be not very much such that he wouldn't really be able to ascribe like a very coherent base objective. But nevertheless, we could look at it and be like, there's clearly a sort of coherent behavioral objective in terms of trying to, you know, we ended up doing it.

And we can understand why that behavioral objective exists because if we think about opening I5 as being a sort of grab out of characteristics and all these various things, those risks were selected by the sort of base optimizer, which is a gradient process to sort of achieve that behavioral level. And one of the things that we're sort of trying to hammer home in a personal conversation is that there's a meaningful distinction between that base objective, the sort of loss function that the trade process is operating for. And if your model actually itself learns to objective without a objective, and so it's easy because of the fact that in many situations the behavior objective looks so close to the loss function to sort of conflate the ideas and assume that, you know, well, you trained Open I5 to, you know, windowed games, it looks like it's trying to windowed games, so it must actually be trying to windowed games. But that last step is potentially fraught with errors.

And so that's one of those sort of points that we're trying to make. Great. Yeah, perhaps a more concrete example would have thrown us at being an example of a system that has a behavioral objective, but not necessarily a major objective or an internally represented objective. Yeah, I think the third step is a little bit of edge-hases.

It's like, well, the third step is even really how the behavior objective, I think it kind of does, right? I mean, you know, it's trying to like stabilize temperature is on level, but it's so sort of simple that, I know, what is your definition of behavior objective? I don't know. Probably.

Yeah. So I want to ask another question about this, this sort of interplay between this idea of like applying and internally represented objectives. So this is, I guess this is a bit of a little bit of a troll question. Sorry.

But suppose I have my program, right? I have this variable called my underscore utility and, but like I'm writing a program to navigate amaze quickly. That's what I'm doing in the program. That's a variable called my underscore utility, and it's assigned to the longest path function.

And I also have a variable called my underscore maximizer that I'm going to try to maximize utility with, but I'm actually a really bad programmer. So why, so why I signed to my maximizer was the argument function. So the function that like, it takes another function and like minimizes that function. So that's my maximizer.

It's not a great maximizer. And then the act function is my maximizer of my utility. So here, the sort of way this program is structured is if you read the variable names, you know, is there's this like objective, which is to get the longest path and the way you maximize that objective is really bad. And you actually use the like minimization function.

Yeah, according to you, what is the, so this is a maze of my supervisor because I did some optimization come up with this program. According to you, what's the maze of objective of this? Yeah, so I think that this is something really important, which is that if you're trying to look at a trained model and understand what that model's objective is, you have to like have transparency tools or some means of being able to look inside the model that like use it or real understanding of what is doing. So you can like describe an inflow vector.

You can add a situation. So I can clearly answer it as well. The base objective should be the negative of the thing which you actually specified as, because you know, you have some very well-managed maximize. But you're maximizing it.

So you know, the real objective of you were sort of trying to extract a like objective, which is sort of meaningful to us. And to be just so listeners can don't have to keep that on their heads. I think this is like fine. It's the shortest path, even though my underscore utility is assigned to one spot.

But you were saying, right? And so I think that what goes to show though is it goes to show that if you had sort of a level of trivial use rules, for example, then enabled you to see the variable names, but not the implementation, you might be significantly led astray in terms of what you expect to be a sort of model. So it goes to show that there's like difficulty in really properly extracting what the sort of model is doing, because the model, you know, is necessarily built to be easy to extract that information. It might have sort of weird, you know, red herrings or things that are like implemented in a strange way that you might not expect.

And so if you're in the business of trying to figure out what a model to be subjected to is by looking at it, you have to run into these sorts of columns, right? You know, things might not be implemented in what you expect. You might sort of have to think that you expect to be an maximizing or something. And so you have to deal with these sorts of columns.

You're trying to look at models that you got there. The question that I really want to get at here is, this is obviously quite an example, but typically optimizers, like at least optimizers that I write, they're not necessarily perfect, right? They don't find the thing that like they're trying to maximize some function when I'm writing it, but the thing doesn't actually maximize the function. So I guess like what I'm part of what I'm getting at is like to what extent, if there's going to be any difference between like your Mesa objective and your behavioral objective, like to what extent, it seems like either you've got to pick the variable name that has the word utility in it, or you've got to say like, well, the thing that actually got optimized was the objective, or like you might know that there's a third way, but I don't know what it is.

So do you think there's a third way? And if so, what is it? Yeah, so I guess the way I was trying this is if I'm actually we're just trying to get good mechanistic descriptions. And you know, what we say, Mesa Optimizer, I mean Mesa Optimizer to refer to the sort of model, which has something you know, look at as a relatively coherent objective and like opposition process.

Obviously, not the models have it, right? So there's lots of models which like don't really look like they have a coherent opposition process objective. But we can still try to get a nice mechanistic description about that sort of model we're doing. When we talk about Mesa Optimizer's referring to models which have this sort of mechanistic description, which look like they have an objective and they have search processes trying to sort of stack for that objective.

I think that the sort of way in which we sort of are getting around these sorts of mobility problems like that would be, well, like we don't claim that this works for all models, lots of models just don't fall into the category of Mesa operators. And then also we're like not trying to like come to a conclusion about what a model's objective might be only by the behavior which is asking if the model is structured such that it has a relatively coherent opposition process and objective, then what is that objective? It's more noting that this obviously sort of doesn't cover the full scope of possible models and it doesn't necessarily cover the full scope of possible models that we are concerned about. But I do think that covers a large variety of models.

And we make a lot of arguments in the paper for why you should expect this sort of optimization tricker because it sort of generalizes as well. It performs a lot of simplicity measures. It has the sort of ability to sort of compress really complex policies into relatively simple search procedures. And so there's reasons that you might expect that you go into in the paper, this sort of model to be the sort of thing that you might find.

But it's certainly not the case that it's always going to be something that you might like. Okay, that makes sense. So I guess the final question I want to ask to get a sense of like what's going on with this optimization. How often are neural networks that we train as optimizers?

Yes, that's a really good question. I have no idea. I mean, I think this is sort of thing that is like an open research problem. My guess would be that most of the systems that we train currently don't have this sort of intro structure.

There might be some systems which we talked a little bit in the original conversation paper about some of the sorts of related work that might have sort of these sorts of properties. So one of the examples that we talked about is the sort of RL squared paper from OpenAI, where in that paper they sort of demonstrate that you can create a sort of reinforcement learning system which sort of learns how to do reinforcement learning by sort of passing it through many different reinforcement learning environments and asking it to sort of along the way figure out what's going on in each time rather than explicitly training on like that specific reinforcement learning environment. This sort of thing seems like closest to sort of thing that we're pointing at in terms of like existing research. But of course the problem is that our definition is one which is mechanistic and requires understanding of is it actually doing search, which is not really something you know because we don't have the ability to really look inside of our models and know whether they're searching out.

So is it happening with current models? I don't know. I think that like we certainly see like some of the sorts of failures in terms of like, you know, sometimes we see models which try to pursue proxies to the objectives which are trained under. But those proxies aren't necessarily the sorts of ones which are like very objective robust.

They might not be sort of a really good coherence search process which generalizes well and is able to sort of pursue that proxy intelligently and coherently even off the screen. Which I'm basically what we care about because if you think about the sort of two-degree robust picture, if you have a situation where a model is very sort of a capability robust is really good at doing computations, it's able to sort of take really so coherence capable actions. But it's doing so for a certain non-rebusted active, an objective that is not the one which we actually want to achieve. You end up in a sort of bad situation because you have a very capable model which is doing the wrong thing.

We certainly see some amount of objective non-rebusted. But in some sense, we sort of also see that along with capability non-rebusted. Like a lot of cases when you sort of look at models of distribution, they just sort of are bad, but they don't do like really coherent things. And so we're in a situation where like these sorts of notions haven't yet come apart completely because we haven't yet sort of gotten good enough capability robustness.

And good enough at producing models for capable of doing really coherent optimization on an off-dismution to be able to get to a situation where they have potentially objective which might be meaningful. Okay, cool. So I now want to move on a little bit to how we might produce mess optimizers. So one reason you give for why or when mess optimization might happen is when there's a lot of computation available at inference time or runtime relative to how much computation is divided to training the system.

And I guess this is because like the optimization has to happen at some point and if it doesn't, if it can happen at like inference time, like when the model is running, then you don't do it so much when you're training. So I guess I'm wondering, do you expect this situation of like a lot of computation being used at inference time relative to training? And if so, why? And also how big a deal is this do you think?

Yeah, so I guess I'm not sure I 100% agree with that description. So I think pretty much I would say is that the more computer model has available to it in front time, the more likely it is to be a mess optimizer. I think the additional question of how do you compute available at inference time versus the view of the training time to compare to each other is a relatively different. I think we're complicated.

So for example, I think that in fact, the more computer you have available at training time, the sort of better you are at selecting for models which are sort of very conformally according to your devices. So, yeah, so this is a little bit complicated. So one story for sort of why I need this. So you think about something like double set.

So double set is a phenomenon which is sort of demonstrated to exist in certain systems where what happens is you start training your machine learning model and initially you sort of get this sort of standard learning curve where the sort of training error goes down and the sort of test error goes down and then goes back up as you start over. And then sort of sort of, you know, inside of double set is well if you keep training passes your training error, your sort of your test error actually starts to go back down again and potentially get lower than it ever was. And the reason for this is that if you keep training, if you put enough compute into just training past the point of convergence, you end up in a situation where well, you're selecting for the model which performs best according to the inductive biases of your sort of machine learning system which fits it out. Whereas if you only train to the point where it gets your training error then stop immediately.

There's a sense in which there's sort of, well, there's only sort of potentially one model. It's just the sort of first model that was able to find which fit the training error fit to the training data. Whereas if you keep training, it's selecting for many different possible models which find the training data and finding the sort of simple one. And one of the arguments that we make for why you might expect me to make some opposition is that they sometimes are relatively simple.

Search is a pretty sort of simple procedure which is not that complicated to some point. But has the ability to sort of in fact do tons of different sort of complex behavior at inferences. It's a way of taking these sort of really complex policies that depend on a lot of different things happening at a certain environment and compressing them into this sort of very simple search policy. And so if you're in a situation where you're putting a ton of compute into training your model, then you might expect that sort of according to something like this double sense story that what's happening is the more compute you put the sort of more your training process is able to select the sort of simplest model which fits the data as opposed to just the sort of first model that defines this data.

And the simplest model that fits the data, you know, the argue is more likely to be a mesophisor. And so in that sense, I would actually argue that sort of the more training could you do, the more likely you already get mesophisors and the more different time you have, the more likely you get mesophisors. And the reason for the inference time compute is the sort of similar to the reason you were saying, which is the inference time compute enables you to do more search and make search more powerful. Search does require compute.

You have to sort of do this. It takes a sort of compute to be able to do the search, but it has the benefit of this sort of compression of, yeah, you have to do a bunch of compute for performing the search. But as a result, you get the ability to have a much more complex policy that's able to be sort of really adapted to all those searches. And so in that sense, the sort of more compute on both ends, I would expect to lead to sort of more likely to the mesophisor.

It is worth noting there's some sort of so-called yes here. So like in practice, we generally don't train sort of that far past convergence. I think that there's also still a lot of evidence that like, especially if you're doing things where, for example, you like to do very early stopping, where you only do like a very small number of great sets after a certain initialization. There's a sense of which that also is very sort of strong amount of ducted biases because you're so close to initialization that like inductive biases are relatively easy, sort of simple to initialize, which I take up a large volume of sort of initialization space, but those sorts of things should be very likely to be fine.

And so you still have a sort of very strong ducted biases, but actually it's a different ducted biases. You know, we don't have a lot of great research here. But I think it's the case that like when you're doing a lot of sort of training compute on like very large models, if you're doing those really, really stopping or training like much past convergence, you're only got situations where the sort of inductive biases matter a lot and you're sucking for sort of relative simple models, which I think sort of leads to something for search and something for optimization and something for myself. Okay, so that makes sense.

So yeah, this gets me to another question. One of the arguments that you make the paper is this idea that like, well, mesophisors are just like more simple. You know, they're more compressible. They're like easier to describe.

And therefore like they're going to be favored by inductive biases or by things that prefer simplicity. So this presumably depends on the language that you're using to describe things. So I'm wondering in what languages do you think it's the case that optimizers are simple. In particular, I'm interested in the sort of quote unquote languages of like the LT norm of your neural network and the quote unquote language again of the size of your of say a multilay Perceptual in our CNN, you know, how many layers does.

Yes, that's a really good question. So I mean, the answer is I don't know. So I don't know the extent to which you know these things hard faculty specify in like sort of concrete machine learning settings like how hard is it to actually like if you were to sort of, you know, try to write an sort of neural network way type is cool that we need to do and so how much space does it take out in the sort of girl that require. I don't know.

My guess is that it's not that hard and it's really depends on your texture. But we can start if you start just a very similar texture. We think about sort of, you know, very simple sort of, you know, multilayered approach. Can you go to opposition?

I think you can. So you can do sorts of things where you can like, you know, I think that has a sort of like pretty important here because it gives you the ability to sort of work on results over time, create like, you know, many different possibilities and then refine those possibilities as you're sort of going through. But I do think it's possible to influence sort of like bare bones sort of search style things in sort of just thinking about a whole other set of setup. It certainly gets easier when you sort of look at sort of more complicated setups.

If you think about something like LSTM, like our current network, you have the ability to sort of work on like a plan sort of over time in a sort of similar way to a human mind like you were describing earlier, or even like make plans and so excuse them. And especially if you think about something like Alpha zero or even more, especially something like Mu zero, which explicitly has like a sort of world model that tries to search over and to, you know, MCTS sort of lots of research over that to find this, you know, side objective. And so you're learning some objective and then you're explicitly doing search over that objective. Now, and we talk about this sort of situation of like hard coded searching in the paper a lot.

Things get a little bit tricky because you might also have sort of based off of what is going on inside any of the individual models and the situation that you might have. And up with that, or sort of well, you might end up in a situation with a sort of search, you've given it to be hard coded as opposite of search, but you really need to do a default problem. So it's a little bit complicated, but certainly I think that like architectures, current architectures are capable of influencing search and being able to do search. There's any additional question how simple it is.

I think that so, you know, unfortunately, we don't have really great measures of sub-simplicity for neural networks. You have sort of, you know, very simple measures like neural complexity and there's evidence indicated like how long or complexity is like, at least related to the sort of the type of biases of neural networks. So there's a recent research from one of the co-authors and some of the research was U.R. skulls, who's one of the co-authors on this conversation trying to sort of analyze the sort of what exactly is the prior of neural networks and initialization.

And there's like, there's some theoretical results there that indicate that like, how long or complexity actually can serve as like a real sort of bounds there. But of course, the modern complexity is not the people who are not where a modern complexity is a measure of a description like this based on finding the sort of simplest term machine, which is equal to sort of reproducing that. And so we can look at these sorts of theoretical measures like a modern complexity. It seems like search has relative to the local modern complexity.

But of course, the actual dot devices of machine learning and then greatly decide to complicate it and we don't fully understand them. You know, there is other research. So there's also some research that looks at trying to understand the sort of the biases of gradient descent trying to, you know, it seems like gradient descent generally favors sort of these very large basins. And there's sort of evidence, these sorts of very large basins seem to match on to sort of relatively simple functions.

There's also sort of a sub-liquid type of hypothesis, which emphasizes that sort of if you train a very, very large neural network, it's going to have a bunch of different sub-networks. Some of those subnetworks add initialization, what happens is sort of encode the sort of already be relatively close to the sort of goal that you're going to get to. And then the gradient descent process will sort of be amplifying some subnetworks and sort of de-empatifying other subnetworks. And if you sort of buy into some type of hypothesis like this, then you're sort of an inspiration for a well.

There's a question of how likely are sort of subnetworks inputting something like optimization, like it's a good initialization. Obviously, the sort of more the larger goal that network is, the more sort of possible subnetworks initialization there are that exist. You know, so combinatorial explosions sort of way, such that as you get to much larger and larger networks, the sort of combinatorial explosion of subnetworks, sort of seems like it would eventually cover even things sort of, you know, potentially quite complicated. But unfortunately, I can't just not that much I can say, because our understanding of the inductive biases of neural networks just isn't great.

We have like some things which we can talk about, simplicity measures and stuff. But unfortunately, at the end of the day, I can be like, well, it certainly seems like search would be relatively simple and likely to learn. But like, I don't have like really strong results for that because we just don't have a great understanding of exactly how these devices operate. It does sound like one thing, though, is that like you mentioned, if you want to do search in something like a multilayer perceptron, you mentioned that it helps to just have a bunch of layers, so there are fine options and such.

So it kind of sounds like you think that if we were selecting on network depth, that would be a kind of simplicity that would disfavor search, rather than favoring it. Does that sound right to you? That doesn't right. So the way which I would describe that is in some ways sort of looking at me for some time, it's sort of thinking about sort of more like a speed prior.

So if you think about a speed prior or something sort of like, yeah, so speed prior is sort of trying to select on models which has sort of like small time from sort of initialization to sort of completion. So like the actual amount of time takes around the sort of function. If you think about sort of depth versus width, if you have a very wide network, but it's not very deep, then it might be actually very fast to run because you just sort of run through that initial step. It might be sort of computationally expensive, but because you can paralyze the sort of width-wise, the actual sort of individual set is sort of relatively sort of relatively fast in a way in which you would sort of be favored by a speed prior.

And so one of the things we do claim in the paper is that these sorts of speed prior like incentives, this incentivized based operation. Because based on computation does require the extent that sort of it requires you to be able to have these sorts of multi-level like long sort of relatively long sort of computations. And if you sort of restrict your model to not be able to do these sorts of relatively long infertile computations, then it's probably going to be less likely to be able to do to be able to be able to be able to make some computations. So certainly in that sense, I think that there is a sort of distinction there between like sort of a simplicity prior and sort of a middle circuit speed prior thing where the speed prior gets different incentives.

And there's a distinction where if you sort of are really selecting on depth, you're sort of doing something more like a speed prior which would change that. What do you think of the important difference here is between time and space? Because it sounds like you're saying, like you have a lot of time to compute, then you can be an optimizer, but if you just have a lot of space, then you can be an optimizer. Like what's the, why is that the same thing?

Doesn't special relativity tell us that these things are the same thing? Well, okay, so here's a really simple sort of analogy to think about this, is let's say I have a look at table as a model versus like an optimizer. But what's a table that is equivalent to the optimizer, it's going to be really large because it has to be explicitly encoded, all of these different things that it's going to do. But it's also going to be really fast because you just look at the look at table, you figure out what it's going to do and you do that action.

Why is it fast? It doesn't take exponentially long to figure out where, like, you go through the look at table and there's like a billion entries that go really fast. I mean, no, worst case, right? Eventually, the original one is expected to go of one.

I think this is actually like, in fact, the case, if you look at like, there is analysis of like looking at sort of middle circuits and speed priors and stuff trying to understand what would happen in certain situations. And in fact, I think look at tables like, you know, are really good because they are, they're in fact, like, if you have all the information already encoded, yes, you need to like have some, you'd be able to generate some key to be able to look it up. But there are like mechanisms that are able to do that relatively quickly, you know, a hash out being one of them, even just looking it up in like a binary search tree or something, it's not going to be that bad potentially. And so if you have a look at table, it can be relatively fast, but it takes up a lot of space.

Whereas if you have an optimization system, it has to be relatively slow because it has to actually come up with the possibility of like all the different possible options and like then selects one of them. And that process is going to take some, it takes some, it takes some time, actually, like look through different options and select them. And you know, you have to restakes to try to narrow down the field of possible things that you're going to look for, but you still need to look, you're still probably going to need to look at a bunch of different things to be able to select the sort of the correct one. So we can think about in this situation, you know, the optimizer has a really sort of large, sort of, it has a larger, so it can for a time to keep, it has a smaller description, because you don't have to explicitly write down in the description of the softizer all of the different actions that's going to take for all of these different things.

It just needs to have a sort of general procedure of well, let's look at an objective, let's do some search and select something. Whereas you think about the table has to explicitly write all these things down, then it can get out of this. You know, and this sort of thing, this sort of trade off between the space and time, you know, occurs a lot in sort of, you know, computation. You know, a lot of times you think about, you know, you'll take a function and just sort of memoize it, which is a procedure of, well, I have this function, I call it a bunch of different arguments.

I can make it faster if I'm willing to spend a bunch of space storing all the different possible results that I generally call it. And you know, as a result, you get a faster function, but you have to spend a bunch of time, sort of a bunch of space, sort of storing all these things. And so you sort of might expect this sort of similar thing to happen with, you know, with your own networks where, you know, if you're able to spend a bunch of space, you can store all these different things, and it sort of doesn't have to spend all this time with your function. Whereas if you sort of force it to sort of compute function every time, then it sort of, you know, then it is able to store all these different things.

It has sort of recomute them. Okay, so yeah, I mean, one thing this makes me think of is like, you know, the paper is sort of talking about these risks from like these learned optimizers and optimizers are these things that are like doing search, you know, like actually doing search like an inference time. And they're not these things that, you know, are behavioral equivalents of things that you search an inference time. And I guess we already got to a bit about like why you might be worried about these nice optimizers and we're going to avoid getting them.

So, you know, it's a mess optimizer, the right category. If it's the case that like, oh, you know, we could instead search for things that are like, like our base optimizer, we couldn't set try to like get these systems that have like, you know, they're really quick inference, you know, and then they're definitely not going to be mess optimizers because they can't optimize. But of course, they definitely could be these, you know, tabular forms of mess optimizers, which is presumably just as big a problem. So like, I guess this makes me wonder is myself like the right category.

Yes, that's a good question. So I think that if you think about the sort of tabular mess optimizer, how likely would you be to actually find the sort of tabular mess optimizer? I think the answer is very like because the sort of tabular mess optimizer sort of has to encode all of this sort of complex options, like results of optimization explicitly. In a way, which I think sort of especially when you get to really complex and diverse environments just takes up so much sort of space and explicit memorization.

But not just memorization, like you have to be able to memorize like what your deployment, like your sort of like our distribution behavior would be according to like performing some complex optimization and like sort of having some way to explicitly code. And that's sort of like, you know, quite difficult. But how could we get presumably like if we're doing this election on short computation time, we do want something that actually succeeds at this really difficult task. So how is that any easier than like it seems like if you think, you know, there are a bunch of these like potential mess optimizers out there and you know their behaviorally equivalent.

Surely the like look up table for like really good behavior is behaviorally equivalent to the look up table for the things that are these like dangerous myself. So if you think about it, there's a reason that we don't train look up tables and instead train look up works. And the reason is that the if we just trained look at tables, you know, if you're doing tabular to learning, for example, tabular to learning doesn't have as good about devices. It is not able to learn like a nice sort of simple and expressive functions.

They were trying to learn to like actually sort of understand what's happening. And it's also really hard to train. If you sort of think about tabular to learning in a sort of really complex setting, but like very sort of diverse environments, you're just never going to be able to get enough data to understand what the results are going to be for all of these different sort of possible sort of tabular sets. And so there's a reason that we don't do tabular to learn.

There's a reason that we use, you know, deep learning instead. And the reason is because deep learning has these nice inductive biases. It has the ability to sort of train on, you know, some sort of particular set of data and have good generalization performance because you selected a simple function to fit that out. Not just any function to fit that out, but a simple function to fit that out.

Especially it has a good performance when you sort of try it on other situations. Whereas if you are just sort of doing training on other situations like, you know, you know, tabular stuff or something, you don't have that sort of nice generalization already because you aren't sort of learning these simple functions to fit that out. And that's sort of, you know, that's what the direction I expect, which you can continue to go, is to sort of develop these sort of nice sort of, you know, in fact, these sort of situations which we have like these model spaces where we sort of have the devices that select for simple models in sort of large expressive model spaces. And if you sort of expect something like that, then you sort of should expect a situation where we're not sort of training these sort of tabular models, we're training these sort of simple functions, but the simple functions that never last like generalized well because they're simple functions are a bit of data.

And it's also what we're doing right, like, why is it listening? Well, I think that like, there's a lot of sort of research that sort of understands, you know, if you actually look at neural network back vises, it does seem like some of the complexities like the right measure, like a lot of these sorts of like the functions which are very likely to exist in initialization, as well as sort of like functions that actually is like the time as a sort of simplicity properties, you know, sort of like in a way which is actually very closely tied to like, you know, computational situation like like a model of complexity. There's like the, I don't know, the sort of three-seater is mentioned recently in front of you are and others was talking about like a lot of boundaries, particularly about model of which is to recognize this model of which is. So there's arguments that like, you know, and there's also a further argument which is, well, you know, if you think about why do things generalize about why would function generalize in the world?

There's sort of answer is something like Ocansr. It's like, well, you know, the reason that things generalize in the world that we live in is because we live in a world where the actual sort of true descriptions of reality are relatively simple. And so if you look for the sort of simplest description of a sort of incident, you're more likely to get something generalized as well because Ocansr is sort of talking to the vast thing which is actually more likely to be true. And so there's sort of these really fun as arguments for why we should expect simplicity to be the source of measure which is sort of likely to be used, and then we're sort of likely to continue to be training these sort of expressive model spaces and then stuck in on simplicity rather than sort of changing something like, I don't know, I don't know.

Okay, so I'm just going to try to summarize it, try for what you're saying. So it sounds like what you're saying is, look, sure, we could think about like training these selecting models to be like really fast inference time. And in that world, like, you know, all sorts of like maybe you get like a tabularized to maximize or something, but like look, the actual world we're going to be in is we're going to like select on sort of description, like simplicity, and we're not going to select very hard on computation run time. So in that world, we're just going to get myself to myzers and you just want to be thinking about that world.

Is that right? I think I just figured that. Okay, cool. All right, so I think I understand now.

So I'm going to switch up a little bit. So what types in terms of like are we going to get these myzers optimizers? Do you have a sense of like what types of learning tasks done in AI, do you think are more or less prone to producing myzers? So like, if we just avoided RL, would that be fine?

If we like only did image classification, would that be fine? Yeah, that's a great question. So we do talk about this a little bit of paper. I think that the RL question is a little bit tricky.

So we definitely mostly focus on RL. I do think that you can get this optimization in sort of more superposition settings. It's definitely a little bit sort of trickier to understand. And I think it's a lot easier to sort of comprehend mesophiles in RL setting.

I think it is more likely to get mesophiles in RL settings. Let's see, make all sort of the other question that was just about the environment. I think that that's sort of a simpler question to understand. I think that basically if you have really complex, really general environments that require sort of general sort of problems on capabilities, then you'll sort of be incentivized of mesophiles.

Because when you think about mesophiles, you think about mesophiles as a way of sort of compressing really complex policies. If you're in a situation where you have to have really complex policies, like, here it depends on the particular environment that you end up in, where you have to look at your environment and figure out what you have to do with a specific situation, and reason about what sorts of things are likely to succeed. You're like requiring these really complex policies that have simple descriptions if they're compressed into search. And so you're sort of directly incentivizing search in those situations.

So I think it's a question of, is it relatively easy to answer in the environment case. In the case of like, reinforcement learning versus supervised learning, it's a little bit more complicated. So certainly in reinforcement learning, you have to sort of set up where, well, we're sort of directly incentivizing the model to sort of act in this environment. And sort of over sort of multiple steps generally.

And you're sort of acting in the environment over multiple steps. You're sort of a natural language sort of optimization community-like situation where it has to potentially also sort of develop a plan and an expu. And especially has some means of storing information over time. But I do think that this seems to happen in supervised learning.

So even if you just think about very simple supervised learning, let's imagine like a language model. The language model has to, it looks at some prompt and has to predict about the future, what the text after that would be by a human that wrote it or whatever. You could imagine a situation in which a language model like this could be just doing a bunch of forensics, as much of understanding of the various different ways in which people write, and the various different sorts of concepts that people might sort of have that would call it as some rules for grammar and stuff. But I think optimization is really helpful here.

If you think about a situation where like, well, you really want to look over many different possibilities. You're going to have lots of different situations where like, well, maybe this is the possible thing which will come next. It's not always the case. There are very few hard and fast rules in language.

You'll look at a situation where like sometimes after this sort of thing, it'll be that sort of thing. But it might be that you're actually using some other standard or you're using some other analysis of it. And so if you build your sort of model, we'll just come up with a bunch of different possibilities that might be, you know, exist for the next. So there's an ex-inclusion and evaluate these possibilities.

And it's still a sort of strategy which is very helpful in this domain. And it still has the same property of compression. It's the same property of simplicity. So you're in a situation where there's a really complex policy to be able to describe exactly what the sort of human depletion of these things would be.

And a simple way to sort of be able to describe that policy is policy which is well, you know, searching for the sort of correct completion. You think about humans actually doing what they're using language. They are in fact doing search for a part of it. You're searching for like the correct sort of way to the best possible way to fill in this situation.

And so if you imagine a model which is trying to be similar to the actual sort of human process of completing language, then you're sort of getting a model which is sort of doing some sort of search. And so I really do think that even in these sorts of supervised learning settings, you can absolutely get this operation. I think it's trickier to think about, certainly. And I think it's probably less likely.

But I certainly don't think it's impossible or even unlikely. Okay. So, yeah, as I guess you kind of mentioned, like, at least a leader too. The supplies I wanted to avoid may as organizers from just being, I just don't want them, you know.

So one thing I could do is I could avoid doing things that require modeling humans. So for instance, like natural language processing or natural language, we're trying to predict what like maybe me Daniel Fallon would say after some prompt and also avoid doing things that require modeling other optimizers. So like really advanced programming or advanced like programs synthesis or analysis would be a face of this where like, you know, some programs are optimizers and you don't use this. And you don't use this to model an optimizer.

You know, you want to avoid that. So a question I have is how much do you think like you could do if you wanted to avoid mess up and you were under these constraints of don't do anything which involves which like for which it would be useful to model other optimizers. Yes, this is the question that I have. There's like there's some writing about that.

So I think there are like, there are some concrete proposals of like how to do a alignment. So I think all of this sort of general category trying not to do sort of not to use case officers. So I have this post overview of a letter proposals building safe. A bunch of possible ways, which you might sort of structure a sort of an as a system.

So one of those proposals, so one of them is sort of STEM AI with basic idea is well, maybe you sort of try to totally confine your AI to sort of just thinking about sort of add the problems scientific problems. Maybe you want to build like, you know, if you think about something like out of a bolt, you want to just do really good protein. You want to do really good like, you know, physics, you want to do sort of like, and you can publish a lot just by sort of having sort of very sort of domain restricted models. So just focusing on the names like chemistry or protein, whatever, they're totally devoid of like humans.

They're all sort of optimized business domain, dark humans is domain sort of model. You're just thinking about these sort of various other domains. There's a lot of privacy in there's a lot of things that you can do just with models like this. And so maybe if you just use models like that, you know, there's sort of you might be able to do sort of powerful sort of advanced technological feats like, you know, things like nanotechnology or whole brain emulation of you really sort of push it.

So that's sort of one or another round that I sort of talked about in the level of proposals post is this idea of Microsoft AI, Microsoft AI is an idea which is due to Chris Ola. Chris sort of has this idea. So maybe another thing we could do is just train us to a predictive model a bunch of data. Oftentimes that data, you know, I think many ways are sort of Chris produces this mind-voluments.

You might still have some human sort of modeling, but the idea would be well, maybe sort of a similar way to sort of a language model being less likely to produce based off the nature. We could just train a sort of supervised setting, a sort of purely predictive model to try to understand a bunch of things. And then rather than trying to use this model to directly influence the world as an agent, we just sort of look inside it with transference tools, extracts this information on all the today has learned that they use that knowledge to guide human decision-making and something. So this is another approach which tries not to produce an agent, which tries not to produce an optimizer or an action in the world, but rather just sort of produce a system which learns some things and then we use that knowledge to sort of do other things.

And so there are approaches like this that try to not build a cell-visors. I think that these are both the semi- You know, the physical problems or country problems, you know, if you, you know, if the main use case for your AI is being a sort of, you know, AI CEO or helping you do more AI research or something. It's gonna be hard to get involved with some of those sorts of things without sort of modeling human's re-example. And so there's certainly limits, but I do think that these are like valuable roles that are sort of worth looking into more.

Okay. Yeah, it seems, it also seems like following these, like if you really wanted to avoid modeling humans, like these would be pretty significant breaks from the standard practice of AI as like my people understand it, right? At least the standard story of like we're gonna develop more and more powerful AI's and then they're just gonna be able to do everything. Like it seems like you're gonna really have to try.

Is that fair to say? Yeah, I mean, I think that there's like a question of exactly how you accept the field of AI to develop. So you could imagine a situation and I talked about this a little bit also in like the post, I have a post-crisolos views on HST which goes into some of the ways that Chris views the sort of strategic landscape. One of the things that Chris thinks about is Microsoft AI and one of the sort of ways when she talks about, which you might actually end up with Microsoft AI being a sort of real possibility would be if you had a sort of major real-minded notion learning community towards understanding models and trying to understand the way that they're not.

And trying to look inside models and figure out what they were doing as opposed to just sort of like trying to beat benchmarks and beat state of the art with that sort of ability to buy boxes. You can sort of think of this as a realignment towards a more scientific view as opposed to more engineering view. Or sort of currently a lot of the way in which people, I think a lot of researchers have approached machine learning is as a sort of engineering problem. It's well, we want to achieve this goal of like getting across this benchmark.

We're gonna try to sort of lose that and achieve that. And you can imagine a more sort of scientific mindset which is well, the goal is not to achieve some particular benchmark to build a system rather to understand these systems but they're doing and how they work. And from a more sort of more scientifically focused machine learning community as opposed to more sort of engineering community, I think you can imagine a sort of realignment that could make Microsoft AI seem relatively sort of standard and moral. Okay, so I guess I have one more question about how we might imagine myself and my users being pretty, like in the paper, you're doing some reasoning about like what types of mess up and I see.

And it seems like one of the ways you're thinking about it is like there's gonna be some parameters that are dedicated to the goals or the mix objectives and other parameters that are dedicated to other things. If you have something like first order gradient descent, it's gonna like change the parameters in like the goal thing, assuming all of the parameters in the belief or the planning or whatever parts are saying constant. And one thing I'm wondering is if I think about these like classically understood parts of an optimizer so something doing search right so like maybe that's perception, beliefs, goals and planning. Like do you think there's necessarily going to be different parameters of your machine learning system for these different parts or do you think that potentially there might be parts like spread across parameters divided to different parts in which you couldn't reason about quite as nicely about how gradient descent is gonna interact with it.

So this is the sort of stuff that gets the most speculative. So I think anything that I say here has to come with a caveat which is like we just have no idea what we're talking about when we try to talk about these sorts of things because we don't have good understanding of exactly the sorts of ways in which neural networks decompose into pieces. There are ways where people are trying to understand that. So you can think about like, you know, personal as a circuit approach and sort of a clear team trying to understand it.

So people as a circuit also I know you have your sort of modularity research you're trying to understand the extent which so you can break neural networks out of so coherence of modules. I think that all of this stuff is great and so it gives us some insight into the sense which these sorts of things are true. Now that being said, I think that like the actual extent of our understanding here is still pretty good all. And so anything I say I think is basically just speculation.

So I think that my guess is that you will have sort of relatively coherent like things like, you know, to be able to do search and I think you will get search. You have to do things like come up with options and then select them. And so you have to have at least some point where you're like, like, coming up, you have to have like heuristics to guide like your ability to talk with options. You have to have like some where generic is options and then somehow prove them and select them.

And so those sorts of things have to exist in summer. Now, certainly we found, you know, if you think about, you know, many of those sorts of analysis that have been done. There exists a sort of polysementicity where you can have sort of neurons, for example, that sort of encode from any different possible concepts. And you can certainly have these sorts of things overlapping because the structure of a neural network is such that, you know, it's not necessarily the case that things have to be totally isolated.

One neuron does one particular thing or something. And so it's certainly the case that you might not have these like really nice, you can point to the network and be like, oh, that's where it does the search. That's where it's objective is coded. But hopefully at least we can have like, okay, we know how the objective is coded.

It's encoded via like this sort of combination of neurons there that do this like particular selection that encode this thing here. I think that I'm optimistic. I know I expect those sorts of things to at least be possible. I'm hopeful that we will be able to guess what we're going as a sort of field in our understanding of how we're understanding of how we're going to turn.

It will actually be able to so concretely point to those things and extract them out. I think it's really good or so ability to be able to address in our environment at a suppensation. But I do expect those sorts of things to exist and to be extractable, even if they require sort of understanding of polysementicity and the way in which the concepts are spread across sort of different than working various surveys. So I'd like to change topics again a little bit and talk about inner alignment.

So in the paper, you correct me if I'm wrong, but I believe you described inner alignment is basically the task of making sure that the base objective is the same as the MESA objective. And you give a taxonomy of ways that inner alignment can fail. So there's proxy alignment where the thing that your specifically that inner alignment can fail, but like you don't know that it's failed. Because if the inner alignment fails and the system just does something totally different, that's like normal machine learning will likely take care of that.

But there are three things you mentioned. Firstly proxy alignment where there's some other thing which is closely related to what the base optimizer wants and your system optimizes for that other thing. There's approximate alignment which is where you have some objective, your base optimizer has some objective function and your system learns an approximation of that objective function, but it's not totally accurate. And there are some optimality alignment.

So your system is just like, it has a different goal, but it's like not yet so good at optimizing for that goal. And when you're kind of bad about optimizing for that goal, it looks like it's optimizing for the thing that the base optimizer wants. So proxy alignment, approximate alignment and sub-optimal value alignment. And pseudo alignment is like the general thing where like it looks like you're in our line, but you're not really.

So one thing I kind of wanted to know is how much of the space of pseudo alignment cases do you think that list of three options covers? Yeah, so it's worth also first, there's a lot of terminology being thrown around. So I'll just try to clear up these things. Right.

So the base of the objective, we sort of mean basically the loss function. So we're trying to understand, we have so much learning system, we have so much learning system, we have so much learning system, we call this a base objective, or the word function, whatever, we call this base objective. We're trying to model according to that base objective. The actual base objective is likely different because there might be many different base objectives performed very well in the train distribution, but which are sort of different properties for the future, like going to the green arrow versus the end of the base.

And we're trying to catalog the sort of ways in which there is sort of model to be pseudo aligned, which is so it looks like it's aligned on the distribution in the same way that you know, getting the end of the base and green arrow both sort of look like they're aligned on the train distribution, but even the profit distribution they don't like. because one thing is one of them is the other thing. And so this is the general category of pseudo-linement. And so yes, we try to categorize them.

And we have some sort of possible categories of the sort of proxy-linement. Proxy-line it being the sort of main one. I think about the most and the paper talks about the most. A proxy-linement and soft-value-line also being sort of other possible examples.

In terms of the extent to which I think this sort of covers the full space. I don't know. I think a lot of this is tricky because it depends on exactly how you're sort of finding these things and understanding what counts as a mesophomizer, what doesn't count as a mesophomizer. Because you can certainly imagine situations which I think is like does something bad after the authors' fusion, but not necessarily because it learns some incorrect objective or something.

And soft-optimality-linement, even is sort of already sort of touching on that. And soft-value-line is thinking about a situation in which the model has a sort of defect in its optimization process that causes it to look a lot. Such that that defect works sort of to be fixed with a whole little bit of a line. And so you're sort of already sort of starting to touch on some of the situations which well, maybe the reason that it looks a line is because of some weird interaction between the sort of objective and the opposition process, similar to sort of your example of the way which it maximizes actually, no no no no no.

I don't know. I think there are other possibilities out there than just those sort of those three. I think that we don't sort of claim to have an exhaustive characterization. And one of the things also that I should note is I do like how I've done some, there's one of those that I made later on after sort of writing a paper that clarifies and sort of expands upon this sort of analysis a little bit, which brings up the case of, for example, soft-optimality-specific-linement, which can be a situation which the sort of model should act adaptively, but hasn't sort of realized that it should act acceptively yet, because it's not very good at thinking about what it's supposed to be doing.

But then later it realizes that should act acceptively and starts acting acceptively, which is sort of an interesting case where it's sort of soft-optimality-line and sort of deceptical-linement. So that's where you take another piece. So if it's not optimal and it's not acting acceptively, does that mean that you notice that it's optimizing for a thing that shouldn't be optimizing for, and just because it does everything that it's basically doing in breaks? Yes, it's not optimally aligned with a tricky.

The point that we're trying to make is it's not it's like self-optimal optimizing that thing it's supposed to be optimizing for? It's actually it looks like it's doing the right thing. So it's very interesting. And simultaneously, self-optimal, being deceptive in your self-optimality, the set of alignment.

Well, if it, yeah, I mean, so it's self-optimal in the sense that it hasn't realized that it should be deceptive. And if it had realized it would be deceptive, then it might start doing bad things and looking because it tries to detect against something or something, it would be no longer actually aligned. Whereas it never figures out that it's supposed to be deceptive and maybe it just looks totally aligned basically for most of the time. OK.

So there are so many other sort of possible pieces that just those. I think that it's not sort of exhaustive. Like I mentioned, I don't know. I don't know exactly about the title, but I think it's more variations on student alignment is the one line referencing.

It talks about some of these other sort of possibilities. There are other sort of things that are exhausted, but it is sort of a kind of some of the sort of possible ways to sit there. Do you think it covers as much as off of the space? There's many three.

Yeah, I do. Actually, what I would say is I think proxy line it all covers half the space. Because I basically expect, I think proxy line that really is like a central example in a meaningful way. And I think it captures most of the space.

And the others are looking mostly at some edge cases. OK. So a question I have about proxy line and really about all of these inner alignment failures. It seems like your story for why they happen is essentially some kind of unidentifiedity, right?

You're proxy misaligned because you have a vacuum. The base objective is to have that vacuum clean your floors. And somehow the base objective is instead to get a lot of dust inside the vacuum and that you normally get from floors, but you don't have to mention other ways of getting it. And the reason you get this alignment failure is that, well, on distribution, you just like, there was only one.

Whenever you had dust inside you, it was because you cleaned up the floor. So what I'm wondering is, as we train better eyes, as more capacity, there's smarter in some sense, on more rich data sets. So whenever they're in perfect sort of probabilistic causal relationships, we find these edge cases and we can select on them. And these data sets have more features and they just have more information in it.

Should that make us less worried about inner alignment failures? Because we're reducing the degree to which there's these unidentifiability problems. So this is a great question. So I think a lot of this comes back to what we're talking about previously also about adversarial training.

So I think that there's a couple of things going on here. So first, I think that my ability is really not the only way we're just going to have. So adding that to my ability has to be required as your trainer. If we're in an situations where we have zero training area, perfectly with the training data, then yeah, in some sense, it has to be misaligned.