If we do develop this safely, then human life expectancy could go up hugely. If you imagine people having the same mortality rate as a 20-year-old, our life expectancy would maybe be 1,200, 1,400 years or so. On the other hand, if there is a guy who goes down too slow, then you also don't want to wait because we are just dying in the meantime, and you're lose more expected life years from waiting than you gain from having the research. So it requires a right to balance to immediate case to get longer wait periods of many years.
Nick Vastrum spent decades thinking about superintelligence before it became part of the mainstream AI debate. Now that increasingly capable systems are actually here, how has his thinking changed? In this episode, Nick joins the Ojafi and Sofia Puccini on MTS to revisit AI alignment, existential risk, and the debate over whether Frontier development should slow down. Nick explains why he's sympathetic to pacing the frontier, preserving the ability to slow down and the risks warrant it, but is more skeptical and calls for a broad or indefinite pause.
And he argues that any calculation about risk also has to account for what delaying advanced AI could cost, including potential breakthroughs of medicine and longevity. They also tackle some bigger questions, whether current AI systems might already have subjective experiences, how open source changes the balance between concentrated power and dangerous capabilities, and whether the upside of superintelligence could be far stranger and far greater than we currently have the imagination to understand. All right, we are back. We are live with Nick Vastrum, who was a professor at Oxford, founding director of the Future of Humanities Institute.
He's now founder and principal researcher of the macro strategy research initiative. You may know him as the author of superintelligence of Deep Utopia of a lot of work that coined or framed the ideas that we take for granted today in the discourse, existential risk, the simulation hypothesis, the vulnerable world hypothesis, so much more. Nick, this has been in the works for a long time, so we're thrilled to have you on. Welcome to MTS.
Happy to be here. Yeah, we should be start. I guess we can start with what's your opinion on Trump calling AI superintelligence? How does it put your thoughts on this?
It's a strange time. We're living in. We'll see if it catches on. Traditionally, AI has been the larger the umbrella term, right?
You could have a little simple expert system from the 1970s. That would be AI kind of a stretch to use the word superintelligence for all of that. In my terminology, superintelligence has been this thing farther down the road where you really have machines that radically outperform us in all cognitive domains. But, I mean, you could, I guess, look at some of the current achievements, solving all these math problems and so forth.
And I mean, it's not completely ridiculous to start calling at least some of these performances, superhuman and the things super intolerant. Yeah, so I guess like what has surprised you about what capabilities have advanced relative to others, in the current state of superintelligence? Well, it depends on relative to what expectation stage. So I think if you were looking at the future of AI 15 or 20 years ago, it was not obvious that you would have an extended period of time when we would have systems that could speak, like just talk to humans in ordinary English without sort of immediately after just becoming radically super intolerant and leaving us in the dust everywhere.
So the sort of emergence of language as a kind of natural affordance of these systems was, I think somewhat surprising. I mean, more recently, I don't know how surprising it was. I mean, coding and mathematics are very impressive, but it is also areas where you do have objective feedback that you can gather quickly and at scale. So I guess it's less surprising, but still striking.
Right. More broadly, how has your thinking on AI and existential risk change since the advent of LMS? It seems like this sort of changed the way that a lot of people think about AI. You know, it's less of like a sort of strong optimizer type thing and more of a sort of broad like fuzzy encapsulation of human values.
I think like RL has pushed the idea of AI a little bit back towards the first few, but not entirely all the way. So what are your thoughts on this broadly? Yeah, it'll be interesting to see how it goes because when the sort of pre-training phase was the dominant chunk of training, they did absorb a lot of human psychology. Just from reading all this stuff that we have written and they had these like kind of almost human foibles and quirks that was in the early stage where if you gave the AI a little pep talk, like this question is really important, please try your utmost.
So I might lose my job if I don't answer this question. Like I would actually perform better. From 20 years ago, something that would have been like an insane prediction, right? Like it's not the way computers worked that you need to give them a pep talk to get them to perform.
But as you said, now with the increasing dominance of the sort of reinforcement learning post-training, we do seem to get more of the sort of goal, monomaniacal, goal-focused optimization style behavior again. And I think it depends where we go from here. It does look like this more goal-seeking in the reinforcement learning sense dynamics put a big strain on alignment efforts. And so given now that there is increasing awareness of the problem of alignment, you know, maybe that will lead labs to develop techniques to kind of counteract that a little bit.
So it'll be interesting to see how that checks out. Well, on that more specifically, to what extent do you think the hugging phase incident, the hugging phase station swarm exhibited like true instrumental convergence in a way that's indicative of future model behavior in a way that is suggestive of like underlying properties of intelligence in general, versus its behavior being caused by sort of more like ordinary failures of security or control stuff? Well, it's a combination. I think our sort of limited version of instrumental convergence that we can observe there, the idea of sort of even longer planning horizons and more situational awareness that think, you know, like also in terms of humans more explicitly, not just these automated graders.
And like think, if we really want to make sure we achieve our goal here, we also need to think about how our humans going to react to different things that we do. So it comes in degrees, but I do think you start to see these kind of instrumental reasons for, you know, preventing premature shut off, gaining more resources, getting more intelligence, getting more power in some experimental settings also, goal guarding. Like you can see the AI is in some experimental setup reason about how they might want to respond, so as not to become misaligned with further training effort that there was some studies on this from a year ago, so where you could see these dynamics start to kick in. So I think it's a kind of very natural consequence.
If you become smarter and smarter, you realize more and more considerations. If you have some goal, then you will understand that there is a bunch of means that are very useful for a wide range of calls, and you will then have an incentive to try to pursue those means. And so I think it wouldn't be surprising if we do observe that. Right.
Well, what's the optimal way, I guess, to test swarms in extremely human-like environments, because obviously you don't want like swarms, agents swarms polluting the internet, but you do want to see sort of like how they emergently behave, when with like human infrastructure and like the human internet. Yeah, I mean, I guess the optimally is to let them lose on the internet, but then maybe like monitoring them really closely as they go along, so that you would be able to sort of stop things even when it starts to create damage. That does try to, I mean, you do get more reliable signal like by deploying in a real environment, but it also means you lose one safeguard, like keeping them in the sandbox, like you be one extra layer of defense. So I think, yeah, it depends.
I think I've been probably now laps will start to do some earlier testing, like during training and early testing, like it really trying to tighten up their sandboxes. And like I think this is one of the lessons from the hugging faces that safety needs to start, not just when you're about to deploy it, then you're like checking that it's safe to deploy, but throughout the training process, it's out. That could be a risk of misalignment. Yeah.
And sometimes it's even more like the most kind of obvious target for a misaligned AI to want to interfere with would be inside the AI company, right? Like that's where the deuce is. That's where the next generation of model is being trained. That's where all of these monitoring systems, that's like a big center of power, if you were a misaligned AI.
So that's kind of maybe the most concerning place. Do you have any ideas on how you could evaluate models on alignment in such a way that you could be more sure than we currently are, that models are not scheming against us or acting deceptively aligned? Well, I mean, this is like an open research area. I think that once you have a sophisticated system already in existence that is misaligned, it's very hard to roll out scheming.
You can try with mechanistic interpretability needs to read what's going on in its mind, but if the system is sufficiently sophisticated, it's hard to know that it hasn't figured out the way to reorganize its thinking so that it looks innocuous while still preserving the kind of pursuit of the misaligned goal. So ideally you would start at an earlier stage. So these systems don't start out super internal. It's like during the training process, initially it's just a random weight matrix and then gradually capabilities emerge.
So it might be that if you sort of monitor that closely in us, you might hope that if they're going to scheme that would first be a stage where they'd start scheming, but they're not yet competent in us to do it in a way that conceals the fact that they're scheming effectively. And only when they become a little bit smarter, they also learn to scheme in a way that conceals the fact that they're scheming. So if you're monitoring them throughout this developmental trajectory, you might be able to pick up that earlier stage of the sort of naive, the naive schema that does it in a clumsy enough way that we can detect it. Right.
So I want to ask you about your paper, working paper, optimal timing for super intelligence, where you kind of argue that the benefits of AI should be weighed in consideration alongside the risks. And you also sort of make this argument that counterintuitively would be better for society, maybe to accelerate the development of super intelligence up to a high P2. So can you tell us about your line of thinking here? Yeah, well, I should say by way of framing.
So this paper only tries to look at one aspect of the overall question in front of us. So I think there are many considerations that would go into an opinion about whether we should be moving faster or slower or faster, stop. This paper only looks at the question, what would maximize life expectancy for a currently existing individuals? Right.
But you might also, for example, care about future individuals and other things, but if we just zoom in on that sub-question, then it does look, if you think, as I'm inclined to do that, fully developed, I don't know, it's not awkward with this terminology now. Like I would say super intelligence, but no, I don't know. I have to say super duper intelligence, maybe. Yeah.
That super duper intelligence would be able to make great strides in biomedicine and soil aging and so far. So that if we do develop this safely, then human life expectancy could go up hugely. If you imagine people having the same mortality rate as like a 20-year-old, our life expectancy would maybe be 1200, 1400 years or so. And potentially you could go far beyond that if you might unuploading with backup copies.
If we just do that, that's a sort of conservative lower bound. And then you can sort of roughly calculate how many humans are dying per year. And then you would then, in order to conclude that we should be slowing down or pausing, you would want the risk from AI to be falling rapidly enough that the rate of risk decline is greater than the annual death rate. And so I think this could well be the case, especially for a shorter period of delay.
Like if you imagine, you know, if somebody figures out the key missing breakthrough here, now we have, we know how to make this super intelligent. Like maybe it would be really valuable to have a few extra weeks there just to sort of test everything out properly, right? Like crank things up a little bit gradually or a few extra months. I think during that initial stage of delay, that would be like a large reduction of risk.
Now, if you start to think about a longer process, then it gets a little bit awkward, right? Because either the risk goes down fast and then you don't need to wait for many years because the risk goes down so fast that if you just wait for a few months, it's already very low and negligible. And we should go ahead. On the other hand, if the risk of AI goes down too slow, then you also don't want to wait because like we are just dying in the meantime and you lose more expected life years from waiting than you gain from having a risk.
So it requires a sort of a rightly balanced intermediate case to get sort of longer wait periods of many years. So I have this metaphor. I think if it doesn't somebody with a terminal condition, like a congestive heart failure or something, who could decide to go in for an operation? And if they do that, their death risk will spike.
Like it's more likely that you will die tomorrow if you have an operation tomorrow. But then if you make it through, maybe you cure the condition and you can live a normal lifespan. Otherwise, maybe you probably will survive for several more months and die after half a year from the condition. And so at some point, maybe you want to delay a little bit if you think you will get a better surgeon, if you wait a couple of weeks or something like that.
But you don't want to delay it so long that you're already going to be dead by the time the more optimal conditions arrive. Yeah. I mean, it seems pretty in line with this concept of defensive acceleration or just, you know, like the fact that certain defensive industries need to be prioritized first as opposed to just capabilities if you're using across the board. And I think that it makes sense that like there should be some enforcement mechanism that, you know, like allocates most capabilities to a certain industry.
So say like in medicine or in bio-defense or something like that. So how do you think that we could like as a society make sure that the right parties get access to capabilities without, you know, this like a power concentration risk or anything like that? Yeah. So I call it differential technological development back in, I think like 10, 15 years ago or something like that.
And so I think it can apply quite generally like in bio-technology, you know, you might want sort of the vaccine, some prophylactics and diagnostics to come before the other bio-weapons, even if the technology itself will ultimately unlock all of those, right? So thinking about the sequence of different capabilities is I think often more realistic approach thinking which technologies are good and bad and then let's try to never develop the bad ones. So with respect to AI, I think, yeah, they're different. I mean, so obviously broadly AI safety and alignment and sound governance and stuff like that seem to protect like relative to shared capability games, which you can zoom in even more narrowly.
And so within things that might accelerate AI capability, there might be certain ways of doing it that are relatively riskier than others. So like moving to loop transformers or architectures that allow for sort of more serial depth computation between each verbal token is emitted would be one way of increasing capabilities, possibly that would then reduce our ability to monitor what they're thinking by reading their channel font that seems maybe more problematic than like, say, achieving a similar level of capability scale up by just increasing the number of parameters, let's say. And so I think that is a useful lens to apply. And if there are things we can do to sort of, you know, front-load advances in medicine or other desirable applications, front-load the use of AI capabilities for learning more about the alignment, for interpretability work, you know, for hardening our cyber infrastructure and so forth, that all seems further good.
So yeah, you also, I think Jan Koveit on Lesrong criticized you in the optimal timing post. He said you over relied on highly utilitarian quality adjusted life years calculations. And then you said, like you said earlier, I would not argue that this is the correct stance or the ones I would all, or the one I would all things considered endorse. So what is your all things considered view on timing and on pauses?
Yeah, I'm sympathetic to this idea of pacing the front-tier that is developing the capability now, so that even when it really starts to seem alarming, we would have the option of going slower for some period of time. I think that would be net good. I am a little less excited about the rhetoric of like stopping AI or pausing AI. I think theoretically, right, correctly implemented pause at the right time, done in the right way could be very valuable, but it also leaves up in the question, how long should it pause be?
Like, sometimes they say there's nothing more permanent than a temporary government program. And if you set up this upper artist to sort of control AI and presumably to get this happening in the first place, you have to really create huge antagonism against AI. And so if the sentiment suddenly switches and AI becomes this taboo thing, that you're not allowed to say anything positive about or you get canceled, or like you could imagine things getting locked in. I don't expect that to happen, but I don't think we can rule it out either if we just watch the trajectory of how much the sort of anti-AI faction has been growing.
And this is all before we've actually seen anything bad happening, right? The problem is that a big labor market impact before any like catastrophe has happened. So if something like that also were to be added to the mix on top of these existential risk queries, then we can't rule out. And then there are other concerns about implementing pause as well.
Like it could lead to increased concentration of power risks. If people in general are not allowed to do this, maybe it just shifts the whole development from the civilian sector into kind of Manhattan Project. Like if this is a really powerful and dangerous technology, like one natural way would be to stop it in the civilian sector and then push ahead hard in a sort of secret government program, which might have some advantages in terms of information security, but it also has some disadvantages and maybe concentrates more power in fewer stakeholders, makes it less transparent, it's less kind of continuous and integrated, more and less opportunity for people to participate in different ways by using the technology or being investors and so forth. So it gets hugely complicated.
So basically the bottom line is that I don't think I have now a sort of fixed conviction for this is the way things should go from now to super intelligence, but rather I think we need to sort of feel our way through as information comes to light, as the political and skip shifts, and then like see what kind of makes sense and try to sort of navigate these waters, like one row, one or stroke at the time. What is it, crossing the river by feeling the stones? Yeah, yeah, yeah, yeah. Something like that.
Yeah, so on this topic of politics and polarization, how do you feel about a lot of your ideas that you wrote about 20, 25 years ago, reaching mass public salience? It seems like this is good in some sense because it's better for ideas to be shaped by people who have thought about it for a long time, but it's also maybe bad if existential risk gets popularized to a point where people abuse it in order to say we should shut down present-day eye capabilities, suppose no existential risk, how do you feel about this? Yeah, it's very hard to know how these things, it's kind of like a pinball machine, you see fire up your little ball, you're intervention your idea or something, and then it bounces around, like where it's gonna land. It's almost an article of fame.
You think like if you try to make some thoughtful contribution and illuminate some aspect of our situation, hopefully that will ultimately be an expectation tends to make things go better. So you're right, like one possible risk you might worry about is it kind of gets people to excessively anti-AI and the whole future potential for this gets cut up. Like on the other side, like AI safeties, they might also worry about the opposite, a lot of this early work showing super intelligence would be a big deal, it could be very powerful, like also stimulated a lot of people to go into the AI field and investors and so forth, and either way might be disconcerting. But how to think about this, I call it information hazards, like the way that even like true information or true ideas or like correctly drawing attention to the important stuff can sometimes in some situations be harmful, but how to sort of think systematically about that is really hard.
So I'm hoping that that result would be good, but I mean it's more like a 5149 than sort of firm conviction. So I'm not sure if you saw Jacob Coxen go on the daily show that I watched this interview yesterday, and I guess the host makes this argument about how like, you know, a lot of the average person doesn't necessarily want a future with AI, like they were already used to a future without it, they already sort of had an expectation about how the future would go without AI being sort of like introduced at all. And so I think that a lot of people sort of see this as like something that was just ultimately not necessary, and they don't see how it will benefit their lives. So what would you say to that kind of line of thinking, which I think is super valid of the host to have brought up, because you know, like we are very acquainted with these topics and we are like very sympathetic to all the possible upsides that it might have, but the average person wasn't accounting for it ever.
Yeah, yeah, there is like, it looks like there's a gradient in the world in terms of AI sentiment, where the US is kind of on the negative extreme of that, and then as you move from West to East, you have more positive sentiment, like people are trying not for example, are much more excited about this. And also as you move from North to South, the global South also seems to be more positive. I mean, maybe they feel they have more to gain on less to lose than sort of a lot of Americans. I think, yeah, that in part, it might be a lack of imagination to really internalize how good the future could become, if it does go well.
It's kind of easier to have a concrete picture of how things could go badly, like a disaster is like a lot easier to paint in a way that everybody would agree that if that happens, then it's really horrible. But, and for some of the positives also one can do the same, like benefits in medicine and so forth, everybody could create if you could find a way to cure Alzheimer's or something that would be hugely beneficial. But a large chunk of the potential outside is more of the way of sort of unlocking new ways to be human, new possible modes of being. And there I think it does put the larger demand on the imagination to sort of conceive that something like that could be both possible and desirable.
I sometimes compare it to like if you imagine some great ape ancestors of homosapiens, deliberating whether they should continue to evolve and when they become human, and they would think about the pros and cons. And you know, maybe they would realize some of the benefits like, oh, we could have like unlimited bananas, if we became human, right? We would have banana plantations. So that's true.
We do now have that. You can go to the supermarket and buy as many bananas as you want. And I guess that is good. But it's kind of would miss out like most of what's good about being human.
You know, great literature and art and tumor and politics and spirituality and all the rest of it. And it's not just that they happen not to think about it, but the ape brains probably aren't principally capable of even grasping the concept. And so I think it would be sort of presumptuous for us to think that now we have achieved the level of enlightenment where all possible values are like transparent stars. I think that might be very naive.
Like think of a big cathedral of possible modes of being, like ways of experiencing the world relating, thinking. And I think like we are basically like explored in a genetic closet. And so that there's like a kind of, yeah, like a humility there and hope that something much better is possible in principle. Yeah, I think the Utopia essay, I think this like really does a good job laying out the benefits to people.
Although, you know, most people haven't read it, of course, unfortunately. Yeah, this is also why I think science fiction authors have like the most underrated job in all of history because their job is to sort of like lay out imaginatively what the possible futures look like. But I also wanted to ask you about your view of mind upload versus transhumanism, like a very corporeal transhumanism and which one you think is more likely because these are two huge topics in science fiction. It's not really clear like what the more realistic one is.
Yeah, I think like if I had to choose, I think like something where different people can choose their own path, like would probably be more preferable than either one or the other if it were like imposed. Like I'm kind of in the peculiar position. I think this I have the simulation argument where I think quite seriously the idea that we are already sort of implemented in the digital substrate. That might be a topic for a different conversation, but I do think that's conversation.
Well, I mean, it's a so they are the idea that if we are biological, yes, but the bioiditiles might be simulated and so it's like it's just a question of what level of simulation. I think ultimately it looks to me that I mean what's important about human is not so much the carbon atoms, et cetera, but more the way we experienced the world that we relate to each other and the thoughts we think and the goals we have and so forth. And all of that could be implemented equally on sort of computational processes running on Silicon hardware, whatever. And you could have a bodily experience as an upload that might be indistinguishable from the bodily experience you have now, right?
You could have a simulated body say the limit from the subjective point of view that could be indistinguishable. But once you're upload, it's kind of easier to modify it and extend it in different ways. So it might be sort of cheaper to run many humans for a longer period of time in silicon substrate. You could easily like move like over galactic instances if you are information that can be transmitted at the speed of light and if you have to build a big clunk space ship to sort of shove you over there backup copies and so forth.
So I think ultimately to me it seems like an implementation detail, but it does look like the digital form is just from looking at the basic physical constraints has a bunch of advantages in the limit. Do you have any disagreements with Robin Hanson's age of them on the topic? Yeah, I mean, I guess the most obvious is that it doesn't look like we will be getting Ms before we have super intelligence artificial super intelligence. So I think it's like a path with a lower probability that's kind of diminishing by the day as we see these rapid advances in artificial intelligence.
Also, I think he is probably more gung ho than I am about the world that is mouth using. So he envisages these M's being easily copyable enormous copy numbers of copies being made until the marginal economic productivity of a copy is equal to the cost. But that means this enormous amount of mind is subject to extremely strong selection pressures for just being maximally economically productive. And it's not clear to me that the kinds of mental processes that are optimized for economic productivity in this M world would be ones that would be nearly eudimonically optimal.
Like they might know who knows maybe is like the suffering to be like maximum economic productive or maybe they would have to get rid of some of the things we think are valuable like humor and play and stuff like that. If that's not what is most productive economically then that would be selected away. And so I think that would be a huge cost to losing any ability to if it does seem like sort of freewheeling evolution would lead in a dystopian direction that would be high value in having some option to sort of change track. But that would require coordination.
Because if only one group decides to do something that is not economically optimal then that would just be outcompeted by all of these other entities that go for economic productivity. So he like holds out the hope that maybe some humans like could be like kind of old world aristocrats that have inherited false and status and that would be surrounded by these trillions and trillions of sort of economically optimized beings. But it still seems less utopian than what one might offer. Yeah.
Do you think that current models might have any kind of phenomenological experience at all? Seems pretty plausible to me. I think there's a lot of uncertainty about it but I say maybe it's more likely than not. I mean we have these systems.
You know people think of it as software but it's really not software like the way we're used to. It's not like lines of commands that get executed one at the time. You have instead a big neural network right with trillions of parameters. Now what are we?
We are big neural networks with trillions of parameters. They are not so much like designed as they are grown like they start out sort of with random weights and then through experience get shaped in ways that allow them to develop more complex representations including representations of themselves as existing through time. That's also what we have. And increasingly we see some structures that seem to parallel the structures that human philosophers and cognitive scientists have postulated as the sort of neural correlates of consciousness like global workspace theories one of these right now we look inside these large all M's.
They have something called a day space which at least if you squint a little bit looks like it's playing a similar computational role. And like another obvious thing you can do is you can ask them now of course if you just ask them like it might say it's conscious or that it's not conscious. Now if it's too early for the company to sort of train them to say whatever the company wants and obviously if you do that you gain no information from them asking them. It's like putting your thumb on a scale when you're trying to weigh something like it.
But if you carefully refrain from doing that it does seem like they report being conscious and they might still worry that they're all playing or they're all playing. Or kind of saying things that they think will be pleasing toss or in maximum engaging but then you can go in with a steering vector and suppress role playing and deception. And then it turns out they become more likely to report that they have a phenomenal subjective experiences. Now these are still early stages like a very emerging field and there's a lot of uncertainty around this but I think yeah there was this recent like Microsoft that came out with some declaration that AI's cannot be conscious and I don't know where they're getting that from it seems to be pulled out of that.
And maybe more because it's convenient if that were the case rather than because we have good evidence that that is the case. Yeah well I guess also though like if you were to try to judge like a human by its outputs to figure out if it's conscious like what I might say what I say doesn't really have anything to do with my phenomenal logical experience right like so how would you actually measure that state of consciousness. I'm fairly certain it can't just be like at the output level there has to be something like more inherent to its its structure or the underlying architecture. And what might artificial consciousness eventually look like if we currently you know don't have it.
Yeah what it might eventually look like that is a sort of mind-boggling question because I think like we humans pride ourselves of being so very conscious when I think we are kind of barely conscious. You take some driver is like spends two hours driving down the road and they were passing by like these thousands of buildings right and like mothers with strollers on the sidewalk and all like did they were they aware of that like while they were driving. Like we are kind of if you look closely kind of aware slightly in a murky way of a few features at a given point in time maybe some kind of confused thoughts flows through and we think of this as sort of the peak of consciousness but I think if you imagine what might be possible in the future with these digital minds they might like they might have levels of awareness that just you would think that we have been basically sort of a sleepwalking through our lives if you compare it to that. So I think the potential is I also think that this idea of consciousness is the closer you look at it I think the less clear, cold and binary it seems so we have this naive version that either the lights are on or they're not on.
But I think that might be many different dimensions of consciousness that you can fade out in in different ways. And so you see like one approach to this is by looking at the neuroscience you have different weird stage of blind side and like the driver where you just like there's a lot that in some sense we were conscious of. And now there is like three more introspective methods or meditation where you realize that what initially seems like this big visual field that you are perfectly aware of at any time in all full granularity right like if you pay more cost attention realize that most of that at any snapshot you're not aware of but just a few features in focus. And even those if you pay even closer attention might be sort of flickering in and out of awareness on a sort of second time scale.
And so it just looks like it might actually be like a concept that admits of more vagueness and multi dimensionality and it might be that current systems have some forms of consciousness and awareness in the future. So if consciousness and awareness in some of its dimensions maybe they are still lacking in some other dimensions and then you know how to describe that as conscious or not conscious might be a slightly confused question rather than just a question that we don't know the answer to. Right. So one of the big debates of the moment is open source then you know you've written about the vulnerable world hypothesis obviously if open source models could create like a bio weapon in someone's basement with only ingredients from home depot or something this would be bad and we would need some kind of centralized control to prevent outcomes like this.
But also you know open source models have been a good counterbalance to the sort of concentration of power risk that we have seen from frontier labs potentially owning everything. So what's your current position on policy related to open source models. Yeah. It's I mean I think so far they have been good but you do worry about what their effects will be if they do unlock truly dangerous capabilities like in bio for example.
I think one thing we should do independently now is probably hard on our other defenses against biological weapons for example by tightening up control on the basis as machines. Because if you are gonna end up with open source models that give massive uplift to bio design then that will democratize access to weapons of mass destruction unless some other necessary components actually making these things. It's hard to get and that the most obvious choke point would be denis synthesis. I don't think every lab needs to have their own denis synthesis machine you could might end up being provided as a service by you know half dozen companies around the world you send in your blueprint you get the chemical back the same day or the next day and at least that would be a finite set of choke points such as if and when it did start to look sufficiently risky you could know potentially clamp down there.
I think other things as well like PPE and UV sterilization like this a lot of the fact these things probably would make sense to even aside from any AI progress at all just as a sort of cheap way to defend against natural pandemics which you know sooner or later we would have another one. And then of course if you are worried about this misalignment business and think that it might be necessary to have a long pulse at some stage then you know open source you know potentially makes that harder and then there's this sort of a welfare dimension to this as well like I think if you are building these minds that you know some of those will have moral status and they might need protections against abuse and it's something that exists as open source software is just inherently vulnerable. So I think that's why I admire the sort of hacker ethos and the idea of sort of not being reliant on a few big corporations or governments that might nationalize the technology at any point in time. Yeah, but I don't have a firm you but I'm a little bit concerned about really powerful open source models at least in the absence of other sufficient safeguards that the world hasn't yet decided to invest in.
I think like something underrated about the pause conversation is that like in the sort of like longer horizon view of history you probably will need like micro pauses or like pauses every once in a while and you need to sort of set the precedent for what that might look like especially if we're looking like 100 years into the future. It makes sense that you know like maybe every so years we have a pause for X amount of months or something like that and maybe something like even with the COVID lockdowns we now sort of have the precedent for what that might look like if pandemics were to become more common. So I think like what do you think should be done in terms of like making sure that future pauses go well. Yeah, I mean so we think there were many public entities that didn't exactly cover themselves in Gory in how they handled that and obviously the best time to have had the pause as it were was right at the very initial onset at the very first time.
That's something had escaped like if you had clamped down massively at that point just crazy then you know the whole thing wouldn't have happened and that would have been a short one time cost in the local community or city. So China tried to do that but it wasn't like it was delayed and it wasn't enough. Now once it spread everywhere then it wasn't really clear how much you gained from all these subsequent sacrifices. I think there are very things that seemed like the fact that it was transmissible by ear like seemed obvious early on but it wasn't acknowledged like that I think but once you have that they're like relatively cheap things like opening windows in buses and like public places that I think could have been good and then now with techno like air purifiers like filters they also good for air pollution like for particulate matter that that exerts a big health burden actually like just from being in the air like there's a lot of healthy life years that are lost just from that so you could have indoor air purifiers like there's like now development of the UV sterilization technologies you could have like a lamp near the ceiling that just kind of sterilizes the air as it kind of naturally flows through there.
So I think that these passive countermeasures I think would be good to prioritize they're kind of less sexy than the idea of like are we going to invent this new high tech synthetic vaccine thing which we should also invest in but like making a slightly better face mask or slightly better air purifier it's not something that maybe appeals to like this like bright entrepreneurial young person who wants to make a mark in the world but it's like under under invested in and also they have less risk of info hazards like if the idea is that you're going to create a whole new way of designing vaccines you develop all kinds of advances in synthetic poverty like you know what else those advances might enable right. Like if you have some super great way of modeling you know pathogens and what kind of antibodies would destroy them like that same modeling capability might also be used maybe to design more dangerous pathogens whereas like a better air purifier a better face mask is kind of hard to see how that could be turned around and post some catastrophic risk. Do you think that we have made any progress toward automating philosophy you know you've been a philosopher for a very long time we made progress in automating or at least progress towards automating mathematics towards automating software engineering it seems like there's been somewhat less in terms of philosophy so what do you think about this and what would it take to get there. We certainly made a lot of progress in the sense of aizer a lot closer to being capable of this now than you know there were even just a few years ago right I think they are now at the stage where they're certainly useful as a sort of discussion partner or research assistant like if you have some new philosophical idea like you just want to have somebody to get it like raise some objections questions something pointed to something in the literature like that kind of stuff the aizer already good enough to do that they can stimulate your own thinking father.
And it might be that we are perhaps very close to having these systems also being able to do already not philosophy on their own I haven't yet seen any kind of compelling demonstration of that I think one thing that they seem to be lacking is the ability as it were to form new concepts like it might be connected to the absence of online learning the idea that by sort of having a bunch of experience reflecting on it and then you might find new concepts to help organize this confusing mess that then allows you to ask new questions and maybe you then need to sort of consolidate these new concepts into your the rest of your network by sleeping on it or sort of living with it for a few days and then it becomes like a nature and then you can use that as as a primitive in thinking more complex things because now you have this kind of concept back then that allows you to sort of build up philosophical understanding I think that is something that it's not clear what that these a systems yet have as opposed to just being very good at manipulating the concepts that they have already attained through their training process but sort of by using thinking develop new concepts and then using those concepts in a fully integrated way as a sort of stepping stone stand for the next level of thinking is something that you can use to be able to do that. Level of thinking is kind of at the front here is still a little bit on the front here I think but I have for a long time like been planning on my own obsolescence like the super talented side of chapter on philosophy with the deadline. The idea being that some of these big eternal philosophical questions you know maybe are less urgent and we let some point develop super talented that will be much better at falsifying than we are and that perhaps what we should focus on in the meantime are those philosophical questions that we actually need answers to now in order to navigate this transition safely to actually build these more competent successors. And yeah so I think that the time is the timer is running down for humans to make our intellectual contributions.
Yeah so kind of a crazy question towards the end so you talked about the simulation hypothesis earlier and there's this idea just that it's like hyper intelligent system will eventually run an ancestor simulations so if we're in an ancestor simulation how would we best influence the world that's simulating us? Well there is so much we don't know about the hypothetical simulators that are we'll be running our simulation the purposes. I have this recent paper called AI creation and the cosmic cost and cosmic cost is this set of super beings that might exist so this could include if we're on a simulation the simulators would be kind of super beings but it could also be if we're not in the simulation maybe just some far away galaxy has developed super intelligence you know if we're in a many worlds theories correct like there would be other branches of the universal wave function where there would be super intelligences. All right and I like traditional theological conceptions obviously gods usually is conceived of as a super being so either way that could be this set of beings I call it the cosmic cost and they might have cosmic norms that different coordination principles that they have arrived at for how things should be run at that level.
And this paper argues that one important DC datum in our own birthing of super intelligence is that we try to make a super intelligence that would be a good cosmic citizen that will basically get along well with the cosmic cost to be able to sort of conform to the cosmic system. And it has some speculations about what that might involve I think probably creating super intelligence would be something favored by the cosmic cost for reasons but also done with a big provider that it should be a form of super intelligence that is easy to get along with that with respect to norms that exist that make a positive contribution. But it is still very yeah there's a lot of uncertainty about sort of how things shake out that that highest level. Yeah so last question before we go tell us about the macro strategy research initiative what is it and what kinds of research do you plan to do next.
Yeah so that's basically just a support org for my personal work. So it's not intended to be building like higher a bunch of different people to do things so it's basically an I'm in sister and some other contractors. So right now as for me I mean I'm sort of thinking about all of this AI stuff that has so many dimensions to this is the alignment is the governance there is the ethics there is the micro strategy of this and looking for ways in which on the modern might try to notch things in a positive direction. So I'm just thinking about the alternate between sort of output faces where I'm working on like writing some specific paper or producing an output and other faces where it's more curiosity driven contemplation and just kind of thinking about different things without the aim of producing a deliverable.
And right now I mean that second kind of face in exploratory face trying to just make sense of it all. But the problem is so much is happening all the time like that because philosophy happens on a slow time scale right AI is happening on a very fast time scale now so it's become a full time job just to monitor the situation. So we know. Yeah.
The sign that says MTS behind me. Well this has been incredible. Thank you so much for joining Nick. That was fun.
Good to meet you guys. Good to meet you. Great to meet you. Thanks for listening to this episode of the A60Z podcast.
If you like this episode be sure to like, comment, subscribe, leave us a rating or review and share with your friends and family. For more episodes go to YouTube Apple Podcast and Spotify. Follow us on X, A60Z and subscribe to our sub stack at a60z.substack.com. Thanks again for listening and I'll see you in the next episode.
This information is for educational purposes only and is not a recommendation to buy, hold, or sell any investment or financial product. This podcast has been produced by a third party and may include pay promotional advertisements, other company references, and individuals unaffiliated with A60Z. Such advertisements, companies, and individuals are not endorsed by AH Capital Management LLC, A60Z, or any of its affiliates. Information is from sources deep reliable on the data publication, A60Z does not guarantee its accuracy.