Welcome to Invisible Machines, a podcast that explores the expanding world of AI agents and conversational technologies. I'm Josh Tyson, co-author of Age of Invisible Machines, a best-selling book all about the agentic approach to automation and the pursuit of organizational AGI. I'm joined each week by Rob Wilson, CEO and co-founder of OneReach.ai. With their conversational nature, AI agents are changing the way we're operating and designing software. Rob and I partnered with UX Magazine for the Invisible Machines podcast to accelerate learning about the cornerstone technologies and strategies that are changing the way we experience technology.
A quick note, the audio version of this podcast is available on all podcast platforms, and you can watch Invisible Machines on our YouTube channel. I also encourage you to head to uxmag.com. The podcast page there has all the relevant links and information you might be looking for. So as we discussed last week, artificial general intelligence refers to a system that can meet or exceed human capabilities across a wide range of cognitive tasks. It's this kind of general intelligence that allows humans to adapt and thrive in all sorts of harsh environments here on planet Earth, and replicating it is no small task.
On this episode, we're going to talk about perhaps the most critical aspect of unlocking AGI or singularity, as it's also called, and that's AI that can write code. So this is much bigger than LLMs spitting out Python or building HTML websites. We're talking about single-use software that's written on the fly by machines to meet individual needs in the moment. So we often call them ephemeral applications, and they will have a massive impact on the emergence of artificial general intelligence. A few months ago, Rob and I had the good fortune to share these ideas with returning guest Ben Gertzel, the founder and CEO of SingularityNet.
and someone who's helped popularize the term singularity. Ben has been working for years to democratize artificial intelligence, and this led to a fascinating discussion where, at one point, Ben points out that AI writing code is essentially the key to singularity. As you'll hear Rob share in a moment, we've been referring to this pursuit of ephemeral applications as Project Iceman. In the Marvel Universe, Iceman, aka Bobby Drake, sometimes rides around on self-generated ice slides. Like ephemeral applications, they get them where he wants to go, and then they melt away.
So the character Frozone does something quite similar in the Incredibles movies. And overall, it feels like an apt metaphor for the way that software of the future might behave. You ask a machine to complete a task for you, and it writes a single-use program that gets the job done. This future might actually come sooner than we think, too. You'll hear Ben point out that the technology that we need to create AI systems that can write code exists right now. So this kind of touches back on the idea of organizational AGI that we dug into last week, because there are definitely opportunities for bold organizations to chart their course toward Oagi and drastically elevate experiences.
for customers and employees. So this episode has all sorts of food for thought, as well as some exciting and actionable ideas that can have an immediate impact. So let's jump right into this great conversation with Ben Gertzel. Invisible Machines is produced for UX Magazine in partnership with OneReach.ai. Named a leader by every major analyst group, OneReach.ai's generative Studio X platform was built specifically to facilitate the no-code orchestration of AI agents. GSX is trusted by global giants like PepsiCo, PwC, Deloitte, and DHL, but anyone can experience the level of agent orchestration we talk about on this podcast through interactive demos at OneReach.ai. Now, on with the show. Well, Ben, welcome back to the podcast. It's great to have you on again.
Good to be back. Rob, a pleasure as always seeing you here. Yeah, thanks. You too, man. Yeah, so Rob and I were actually, we were both kind of digging into the recent episode of Singularity Net's Technical Tuesday. There was an episode on LLMs and AGI, and you were talking about this idea of the possibility of kind of a hybrid LLM. And even from across the country, I could hear Rob's gears start turning. I knew he was getting excited. So I thought to kick us off, Rob, maybe you could fill Ben in on the conversation we were having shortly before this call, because I think that'd be a really interesting place to start us. Yeah.
So we've been, you know, very thoughtful about what we call like Project Iceman. Iceman is the superhero that, you know, creates ice in front of him and then slides on it and then it melts behind him. And the whole idea behind this is that, you know, this concept that once you can create applications or functions and micro applications, ephemerally, meaning in real time. for single use and contextualized based on what just happened and data. Then you have this world where software is written on the fly and software is logic and logic is reasoning and therefore drawing a connection between AGI and machines writing code on the fly. And so when we think about ephemeral functions and ephemeral...
like micro applications, you know, we sort of think about the effectiveness of them. Obviously we want our software to work and buggy software, nobody wants to use. So now it kind of comes down to getting these LLMs to generate code reliably functions. I say code, but it's really functions because there has to be an input and an output. Otherwise it's incomplete. So functions or micro UIs reliably, and we can see, and this is kind of where your episode really got me going is on graphs. We can see that some coding languages
it's better at like Cypher versus like SQL. And so we can also imagine that maybe there's a language that we can create that it would be much better at. And LLMs are very good at translating from one language to another. So we can think of LLMs merely translating from English to code, right? And then the last thing I'll say is...
If those applications are ephemeral and we have this concept of alignment when it comes to producing text, there's also a concept of alignment when it comes to producing code or functions and micro applications. And that alignment might be just the boundaries we put them in. So now we can see the relationship between neural and symbolic being more about the symbolic part being the prompting engine. And that that prompt could just be a template. It could just be if blank, then blank.
let the LLM fill in the pieces, right? And so now what we're seeing is taking natural language, converting it to code in a reliable way, creating these ephemeral functions, and then determining the next function to create based on the output of the last function or the last series of functions. And maybe this is the path to AGI. You mentioned graph. I saw cipher as being like one of these languages you talked about in that podcast, metagraphs. And I was kind of interested in why metagraphs. Yeah. So we're discussing some very
general concepts here, and then we're getting into the weeds about graph-based tools. So I think to address the general concepts first, I mean, yeah, I think that AI that can write code, this is the key to the singularity, right? I mean, once you have an AI that can write AI code and translate general ideas about how to improve AI systems into concrete. architectures or designs for AI algorithms or patches to improve current AI systems, like once you're there, then you're full on into what I.J. Goode in 1965 called the intelligence explosion, right? Because then you've got your AI system, you can do some coding and make itself a little smarter. That smarter version can make itself even a little smarter and you like presto singularity, right? So that clearly is a uniquely important application, right? Like using AI to design new drugs, using AI to write essays, make art and music, like these are also important, but...
using AI to write AI code that has unique power to trigger recursive self self self learning. Recursive self improvement. Right. Right. So yeah, that's a, it's a very important application. And it's also interesting that already LLMs can do some aspects of code generation way better than any, any previous software, right? Like you can, you can describe some simple manipulation of lists or tree data structures or something. And then, I mean, the LLM will
without Python code, right? And often it will work. And I find that quite handy for writing code that, especially in the language I haven't coded in for a while, right? Like if I need to write an R script or something, like I haven't done R for a long time, I want to remember what all the libraries are, right? So then you can just ask an LLM to write some R, doing something kind of like what you want, but a little simpler. You'll put all the libraries in place and you can go in and shake the one part of the code that's actually doing something interesting, right?
incredibly useful. I mean, it also helps non-programmers put webpages with scripts up and so forth without needing to write any code. I mean, no code product development becomes a lot closer to reality. But in spite of all those strengths though, I would say LLMs are pretty far from being able to do AI development or really write advanced code embodying new AI ideas. And the devil sort of gets into the details. So if we look at graph transformations,
So this is also an interesting and important area, right? So a graph, in the sense that we're talking about here, it's not like a line graph of a chart going up and down, obviously. It is a graph data structure with nodes and links going between each other. And the nodes could have weights on them or labels or types or something. And graph data structure is very flexible. I mean, it's used across many types of software applications. And we've seen graph databases like Neo4j, which corresponds to the Cypher.
graph traversal and pattern matching language that you mentioned, we've seen graph databases become a big thing in the last 10, 15 years. And I think in the next year or two, we're going to see using knowledge graphs to guide prompting of LLMs become a big thing. Because already people are into retrieval augmented generation where you're using vector databases. Vector graphs, yeah. Yeah, you're using vector databases to guide what an LLM does to provide with more memory and more grounding in knowledge. But using a knowledge graph instead of a vector database or along with a vector database, yeah, it can give you more flexibility.
how do you supply that external knowledge to yeah to an llm so in one sense what i'm doing with the open cog agi platform is aligned with that. So OpenCog is an approach to AGI. We have this large weighted labeled typed metagraph, like a kind of graph. I'll explain exactly what I mean by metagraph in a few minutes. You have this metagraph and your various AI programs are represented as parts of that metagraph. And the programs represented as parts of the metagraph, what they do is take metagraph as input and give metagraph as output, right? So it's a metagraph of which parts of the metagraph encode programs for affecting metagraph transformations. So this is the sort of
plumbing it's like the infrastructure of the open cog approach to AGI and then the the real AGI design is in like how are you representing knowledge in this metagraph which algorithms which logic or learning algorithms are you are you representing in there right so that that that's what I'd call a fairly sophisticated use of graph-based AI whereas say creating a knowledge graph embodying some sort of relational databases information and using that to guide generation of an LLM is a much simpler use of of graphs for AI. Like LLMs now are up to the task of learning queries in say Neo4j's graph traversal language Cypher or say GQL as another graph query language, another more generic example. I mean, LLMs now can learn queries in these sorts of languages and you can go from a text prompt to a query in Cypher or GQL that traverses some knowledge graph and gets you answers. And this is highly, highly valuable.
the sorts of things we're doing with sort of dynamic self-modifying knowledge metagraphs in OpenCOG as a would-be AGI architecture. Like this is stuff that LLMs are not yet up to playing a central role in, but we still use LLMs together with that sort of graph system in certain ways. Yeah. Yeah. I mean, the way we look at it is it's, you know, relationships between data and then data.
LLMs handle both the data, the words, the tokens, and then the relationship of those tokens to the other tokens. And then we look at data in general, like a graph-based system, and you have, well, semantics is the methodology for relationships in LLMs. To some degree, it's tokens, right? But more so in vector DBs, where we start to see semantics being a key component of how to build those relationships. But then graphs allow you to create other relationship types and custom relationship types. Yeah. I mean, I think the paradigm of...
predicting the next token, while theoretically could be good enough for AGI, in actual practice, that will not be good enough for AGI. I mean, you could do math theory showing that if you can predict the next token with ultimate generality, then you can do anything. Any problem could be ultimately cast into that. But I don't think that's actually going to be a productive way to look at the AGI problem, although clearly it's a productive way to look at... at some parts of the AGI problem. I mean, another ingredient here is, I mean, one of the early ideas in the use of the Lisp language for AI was there's no distinction between data and program, right? Programs are data, data is programs. And our use of a knowledge metagraph in OpenCOG is very much along those lines, like programs for transforming a graph, those programs can be represented as graph content, right? And that requires you to use graphs.
in a little more sophisticated way than this is typical with Neo4D graph databases. One of the things we found is using graphs heavily, but abstracting it with GraphQL, generating GraphQL versus Cypher directly, and then creating these resolvers in GraphQL. So that concept of symbolic and neural, right, is happening. Yeah, I think putting neural and symbolic together is going to be key to the next stage of evolution of AI. And if you look at...
If you look at what LLMs currently are not so good at, I mean, they're not good at a lot of things, but one thing is complex, multi-stage symbolic reasoning, like you need to write an original science paper, which isn't bullshit, but actually breaks new ground and makes sense. And another thing is doing original groundbreaking art, like inventing a new artistic genre, right? Doing something that's not just a derivative recombination of previous things. And this is also... about symbol systems, though, in a different way. I mean, human arts are all about inventing new symbolic representations and combining them in novel ways. So I think it's clear that combining the sort of pattern recognition and prediction power of today's neural nets with some sort of facility for symbol invention and symbol manipulation, that's going to be key for getting beyond the shortcomings of today's
commercial AI technologies, which are amazing for what they can do, but that's some pretty clear limitations. No, you could potentially... make the leap to symbolic facility by just creating the correctly architected neural network, right? Like theoretically, we know some kinds of recurring neural nets could do arbitrarily complex symbol manipulation. And in practice, that's not what any current neural net is good at. So it's very tempting to take current systems that actually are good at some kinds of symbol manipulation, which are logical reasoning systems, knowledge graphs, and whatnot. It's very tempting to take these systems and...
connect them with neural nets and try to get neural symbolic working this way. And of course, there's been a field of neural symbolic AI going back a few decades now. So people have been experimenting with this sort of thing long before neural systems became as practically functional as they are now. Yeah, there's two points I want to make there. One is this concept of purely neural being the answer. Even if, let's just say...
through that mechanism in architecture, you could create flawless code. Like 100% of the time, it's going to generate the code accurately and predict what you want it to do in a reasoning sort of way, in a logic tree, which, you know, it's a long way away, if ever. There's other reasons for symbolic code, like just the climate. Just functionally, it's less. It's less energy, right? It's, you know, why run every decision through an LLM if it's a yes, no? That's to say, we don't know that. And I mean, I think that, so current transformer neural nets have very bloated representations. But I mean, if you train a recurrent neural net with a minimum description length prior or something, maybe you get a quite concise neural network doing things which doesn't use a lot of energy. And then for symbolic AI, we don't yet know how much energy will be used when you're trying to do
control of very complex reasoning. I mean, you may be using statistical methods for inference control, which end up recording large samples from the space of inference histories, and it comes out being expensive, right? So, I mean, in a very short-term sense, it's true, like pretty much anything you can do that's not a transformer neural net is going to waste less resources because these things are just so bloated. But I think there's a lot of degrees of freedom in both the neural and symbolic side to make things that are more or less smart in different ways and more or less energy efficient. Yeah. Does symbolic allow you to, does it make it any easier to put in reliable guardrails on neural systems if you're using symbolic methods as well? So that there are sort of like hard, more, more like rules that might govern it. Probably. I mean, in a practical sense right now, absolutely. Yeah. I mean, right now, if you're implementing a logical reasoning system, I mean, you can, you can implement some axiomatic knowledge and say, like, take this as, as, as true. And you can pretty tractably see.
what implications that will have on the derivations from the system. And right now with a neural net, it's a lot of hacking, right? And we see the consequences of that. Google tried to make their Gemini neural net not racist, but they couldn't just tell it in a clear, simple form what it means to them to not be racist. They had to train it on a bunch of examples and then generalize from those in a silly way, like making the founding fathers of the U.S. Chinese or something. I mean, yeah, given the technologies right now, that is...
Totally true. If you really dig deep, you could say like, in principle, once you're making a symbolic system that can rewrite its own axioms and its own code, like your guardrails are not a guarantee. And I couldn't say it's impossible to make a neural system that is less stupid in interpreting the guardrails that you try to provide it with either, right? Because there's a lot unknown all around. But like, yeah, tomorrow, if I had to implement a system that would obey sort of well-defined ethical guardrails.
Absolutely, make it neural symbolic and put the guardrails on the symbolic side and make the symbolic side the gatekeeper of input-output. And yeah, you've got a much better handle on what's going on. That makes sense. I want to sort of circle back to this, because a lot of people, when they think about LLM's writing code, they imagine it in our current state, how humans write code. Like, they anthropomorphize it, so it's...
write the code, go through the same CIC process, testing, deployment, blah, blah, blah. They don't think, no, no, we're talking about functions being written, one-time use on the fly, disappearing behind you, that, that this logic is happening in real time and it's, and it's ephemeral in nature and, and, and that just changes everything, right? That changes everything we know about writing software. It's not. If you can write software that quickly and effectively, then we wouldn't write it like humans do. Every person would open up their mobile app and mobile apps wouldn't exist. But the application that they see and the micro UI that they see was created and generated right there on the spot. And the function that it calls was generated just for them in that moment. This is feasible right now, right? I mean, it's not what our software infrastructure works. I mean, there's no fundamental obstacle to doing that.
with technology that we have. Right. It just requires like composability in your architecture. Yeah, it requires composability in your architecture. And I think from an AGI standpoint, there's some unknowns and some research to be done of how do you enable composability to be maintained in a system that's doing sort of self-learning and arbitrarily complex reasoning. On the other hand, if you're building practical software systems in various domains.
We know how to make composable architectures, right? So if I look like in sort of technical domains I work in, like music processing software, bioinformatics, say financial analytics applications, I mean, you're... You're always chaining together different tools in different pipelines already, right? I mean, using composable modules is already standard practice for doing a lot of these tasks. The thing is, it's people composing the modules in a certain way and then using that sort of rig that you've created. Whereas if you can compose existing modules in different ways on the fly for each task and try a hundred different ways of composing them and see which one seems like going to work best for that task.
very naturally build on the sort of modular architectures that people are already using in many of these application domains. And I think that's got to be the direction that things go. And then what I've been working on for a while in SingularityNet and then other associated projects like HyperCycle. is being able to do this sort of dynamic assembly of software modules into software programs, carrying out useful functions, even in the case where the different modules being combined are owned and controlled by different people, right? So then you have like a decentralized blockchain-based little modules, and then those can be pulled together dynamically, even if some of those modules were just created by someone and put online 10 seconds ago or something, right? And so there's whole new...
programming paradigms here. And an interesting thing is human programmers may never need to wrap their brains around these new programming paradigms because it's AI's are going to be doing the programming in this new paradigm anyway. Yeah. That's an interesting idea too because we talk a lot about human in the loop and especially as we're evolving these systems, right? It's critical that people are guiding it and making decisions and then in turn, I suppose, helping the system make better decisions. But maybe part of singularity is that handoff point because like you're saying at some point, maybe we're done checking the code, right? When I was...
helped run a hedge fund in Hong Kong between 2011 and 2015. And one thing we found then, just trying to sell the idea of an AI-driven hedge fund to various Asian family offices, everyone was like, well, you must have a new one in the loop in the trading decisions, right? AI was just giving him input. And I was like, well, no. If I have a human in the loop making trading decisions, I cannot back test the system. I cannot historically test what that human would have done in prior situations. Whereas if all the decisions are made by the AI, then you can historically test what would have happened. You can actually have more confidence of what's going to happen with the system going forward. And then they're like, well, there's a human somewhere. Well, yeah, I mean.
if you deploy a system and it's losing a lot of money for a year as a human you may decide well there's some premise flawed here and you you kill the whole thing or if it's doing super well as a human you may decide okay let's you create more money to put into it yeah of course like if aliens land you may decide okay i'm overruling my ai training system here because nothing in the history in which it's trained possibly prepared it for the consequence of aliens landing on earth right and That may be a difference between a full-on AGI trading system and a narrow AI trading system. The AGI trading system might be able to extrapolate the financial consequences of aliens landing on Earth better than the human can. Certainly, the systems that we were training then could not figure that out. But I think now, on the other hand, quite a high percentage of global financial markets are traded by AI systems that do not have a human in the loop of any trading or money management decision. Rather, are humans as CEO of the company making very high-level strategic choices. But the sense in which everyone wanted a human in the loop,
in a trading system in 2012-13. Everyone knows you don't need a human in the loop in the trading system now, right? It's mostly algo trading in more of a pure sense. So I think the same will be true. across the board. Like people feel like they want a human in the loop. Then after a certain period of time, the AI matures, people realize the human in the loop makes the systems work worse rather than better. And then they don't think they need a human in the loop anymore, right? It's sort of an inevitable progression. It's not to say you don't want some high-level human values infused, like in the design and guidance of the system, but then the human is like, and they're in the loop only at the very highest level rather than inside the operation of the system.
Yeah. It's like, if you're running a simulation and you have a human as part of the simulation, it's just going to slow down the iteration of the simulation. And that's, if you're doing AlphaGo, for example, and you were trying to train AlphaGo, you couldn't, with a human in the loop, you couldn't iterate fast enough. Yeah. You couldn't train on those billions of games. Right. Right. I mean, of course it was trained on human games initially before it got to all the, all the simulations that it was trained on. And I mean, humans wrote the code and they made some decisions in how to set up the code, but I mean, in the end, yeah, to make it work.
You can't always have a human in the loop. I mean, what you really need is a system that's smart enough to get feedback from humans on values and goals and so forth. But in that sense, it's not that different than managing an expert employee or contractor or something, right? I mean, if you have to be in the loop of everything all your staff members are doing as a CEO, I mean, you're screwed, right? What you need is you need to work with those who...
can do their own stuff and who will come to you for guidance when it's important and otherwise do their own thing, including things they know how to do way better than you would know how to do. And I mean, that's really what we want from the AI systems. Yeah, I think it's simple to me, like LLMs, they have an input, they have an output, and the output is for a human. That's the purpose. The human doesn't have to be part of constructing the output, but it's the point of the output is to benefit the human. I know it's oversimplification, but... Yeah, it is interesting. It is interesting to know how the AI came to the conclusions it did.
after the fact, even if it's too inefficient for you to be involved in each atomic thought process. And then you hit another difficulty, which is, it's not always possible to give a simple explanation, right? I mean, I remember when I was going through, I was a math major, I got the PhD in math eventually, but I was going through math classes. I mean, often the professor would pose a problem in a class. I would see what the answer was. I would write down the answer and then
If you asked me how I came up with that answer, it just felt right. If I had to go back and reconstruct an audible path to that answer, I could do it. It took some time. It often was not actually how my mind came up with the answer in the first place, but it was a good coherent rational explanation, right? And being able to come up with an explanation...
It's good. It's an important thing. It's one of our cognitive faculties, but the real way that an AI comes out with something, it could be some combination of a million different little things that you just couldn't explain in a, in a, in a, in a concise way. And this goes back. This goes back to a lot of data on human free will, right? Like, you know, Michael Gazzaliga's split brain experiments where they took people who had the two halves of their brain separated to stop them from getting seemingly severe epileptic seizures. The result is what you say into this ear, this side of the brain hears. What you say into that ear, that side of the brain hears. But then the two halves of the brain are acting as basically separate minds. So then you can whisper in this ear, like, go out in the hallway and get a Coke from the Coke machine. Then the person will do it. After they come back, you whisper in this ear, why did you just go out in the hallway? And the person will say, well, I was really, really thirsty, right? They believed it.
They're making up a story for why they want to... That's a fascinating idea. That's a fascinating area. I wanted to... Yeah, yeah. Nietzsche had the saying, free will is like the army general who takes responsibility for the actions of his troops after the fact. I love this. But it's a valuable function. These are pathological. That's a pathological case with a split brain patient. But on the whole, making up a reasonable story for why we did what we did is a very useful function that our minds have. And we want AIs to be able to do that, right? We don't want them to just take input and output. We want them to be able to...
explain in some way why they did what they did we just have to acknowledge it can't always be a fully accurate explanation because if you try to concern the ai to only do things that can be very very simply and compactly explained that will make it very stupid and the same holds for us like if i had to think about a math problem And I knew I had to be able to explain everything I was thinking in extreme detail to others afterwards. I mean, it's very hard for humans or AIs to think that way, right? So yeah, for that reason, what we're going to get are AGI's with...
meaningful but limited explainability to what they're doing. I want to pause you there because you said AGI is plural. And this whole thing, this is an area in your podcast that I heard that I'm very connected to, very interested in. Back a long time ago, I'll date myself. If you're an audiophile, you'll understand what a Fairlight is. I worked with Fairlight.
Frank Zappa was a famous master of the Fairlight Act. He did Des from Hell and all these classic albums. So we did it with feature films and music, and the primary objective was how to get a certain number of tracks to play in sync. We were limited to eight with a processor, and we had to get 48 going. And so what we did is using timecode and multiple processors, and everybody else was trying to figure out how to get more tracks on a single processor. And the idea was, well, let's go with multiple processors, but sync them up.
through time code in some way so that they behaved like one, but they were really separate processors. And I think of that same paradigm with LLMs going, because you said it, like AGI is plural. When we think of consciousness, we tend to anthropomorphize into like a single thread of thought, but these things, we can have multiple threads. We can have antagonistic. They may be neither singular nor plural, right? I mean, it doesn't have to be.
follow the categories that work for humans. Each of us is a separate mind in a separate physical body because we evolve that way. But you could have AIs networked into what I've referred to as a mindplex, right? Where the different AIs, they're distilling, they may have their own goals and some private knowledge, but they can also swap back and forth mind stuff and they can have, you know, Wi-Fi telepathy, right? So you could have collections of AI minds that are sort of...
more tightly bound together than individuals in human society, but maybe less tightly bound together than the lobes of a given person's brain. Right. That you can have a lot of middle grounds between what we think of as individual and as singular versus plural. And yet we shouldn't anthropomorphize what digital minds and quantum minds after that are going to become. Of course, it will be important for these minds to be able to interact with us in a human-like, human-friendly manner. But behind the scenes, there's going to be a lot of a...
a lot of other non-human-like stuff going on, which is kind of how it has to be. Like the human brain is not the be all and end all of cognition. And many of the problems that we have, many of the things we need done to improve our lives, these are problems that can be better solved by cognitive systems that are not architected like we are. Yeah. When we think about like this idea of multi-threaded...
decision-making, right? This idea of two LLMs arguing or iterating together to a result. So the output, you have an input, and then you have different algorithmic, let's say approaches. within that nested, and then that amalgamates to an output of some kind versus like an input to a single LLM and an output. We have this idea of LLMs arguing or iterating. Is that a component of AGI, this idea that, like, let's just look at a prompt, right? Like we have a model where you put in a prompt and then it iterates on itself. So because it's able to improve its own prompt, as long as you have a feedback loop for a positive or negative outcome, it will iterate on that prompt and continue to A-B test.
trying to get an ideal prompt. I think the kinds of feedback and iteration you can have in systems that are able to introspect on and understand their own state will be quite different than what we're doing now with LLMs sort of inspecting their own history and conversing and collaborating. Because if you take a system like an open cog system, as it's intended to be anyway, I mean, its own mind state is a state of this knowledge metagraph that it has. it can then inspect that state and construct descriptions of that state. Whereas transformer neural nets currently are not really doing that. Like they're not looking inside their own mind at their activation state and using the dynamics in their own activation state to help them understand what they're doing. So I guess at some abstract level.
the same sort of dynamics can hold. You have two different agents interacting with each other, debating stuff and refining each other's output. And yeah, you could have a system reflecting on the mistakes you made and trying to improve on it. But I think systems with a higher level of self-understanding will do these things in a variety of different ways though. And LLMs now...
They don't know who and what they are. They're not able to track their own thought processes. And so you have to hack a lot of things to work around these limitations. Yeah. It's like that term, like, you know, variety solves complexity and the idea that you could have different LLMs as in LLMs with different training data and different alignment.
competing against each other for an output or iterating with each other is the same thing as a diversity of human beings around a table trying to solve a problem and seeing it from different angles. I guess at some point, though, there's an output. It's a little bit like that, but it's missing many aspects of that, right? I mean, in the end, it may be more like just subsampling in statistics or something where you're subsampling the training data. And then, I mean, in statistics, if you try to train a model on some data and want to generalize beyond that data, one trick you can use is train an ensemble of models each on different subsamples of the data.
And then, and then you use some sort of voter weighted average of those models. Right. And I mean, having given all of them straight on different data sets, it's, it's much like this sort of statistical technique. And I feel when you have a diverse group of people around the table, trying to come to some conclusion, I mean, they're. They are modeling each other's minds, right? And then they are learning to improve their models of their own minds based on how they see that others are modeling their minds. Like I can learn about myself from seeing how you're seeing me and I can totally change my theory of myself based on how I see others that others are seeing me. And this gives groups of people the ability to make radical leaps of collective insight in a way that I don't think you get just by voting or averaging together or debating stuff. So again, we hit the fact that LLMs don't.
self-understand or understand others. And I mean, the word understand is very fudgy, but you could say the model that an LLM has of itself as a cognitive agent is a very limited model without much explanatory power. And the model that LLMs have of other agents talking to them are also very limited with little explanatory power. And I think that means that the set of collective cognition, the collective intelligence processes LLMs can enter into. They're interesting, and they do go a bit beyond ensemble machine learning, but they're still far short of the complexity of sort of collective intelligence dynamics that groups of people can get into.
There was this interesting idea that came up on this podcast. It's possible that you brought it up, Ben. I really haven't trouble remembering, but we were talking about the AI systems that were trained to play against humans at chess and how the early systems were sort of trained to replicate human, the ways that humans play chess. And the real breakthrough, I think, was maybe like when they started playing in their own way almost, like they weren't necessarily copying past human, like they studied every move that every human had ever used to win at chess. It kind of became more about like just being able to... I wouldn't have been me who said that. Okay.
I mean, the first chess system just used like alpha beta pruning, which is not imitating what humans did, but yeah, then in the, in the evaluation function at the leaves of the tree, you had to use heuristics that were, that were, that were based on, on human evaluation heuristics. Right. I think the main takeaway for me though, was that, uh, once they were able to just beat any human at chess, like those chess masters, I think would say it was like playing an alien because they didn't even understand their opening moves. The thing is, while the basic algorithm is alpha beta pruning or other things like that, that are nothing like how humans play chess, there's some evaluation function you have to train on some data set. Right.
For a long time, that was trained on a basis of human chess games. Then with that AlphaGo, they amped up the idea that existed before, but they did a huge scale of this training based on simulated games of AI basis AI, right? And so once you're training the evaluation function on AI versus AI games at huge scale, I mean, yeah, you're coming up with things that are not going to be seen in any human game. And then there's whole new avenues of strategy that we're just never... never envisioned before. I mean, my oldest son was a decent Go player, so he knew well enough to study the Go games with AlphaGo versus the Go games between Go masters. And you can see AlphaGo is doing things that seem senseless by all the Go theory, right? But then after some number of moves, you're like, whoa, shit, this senseless thing somehow crystallized together into an amazing...
amazing winning strategy, right? And it just kind of puts an interesting perspective on intelligence, right? Like it's not, I think we think of intelligence as like sometimes being the extent of what's in our head, but like they're able to exploit these games in ways that for whatever reason, humans weren't able to figure out or theorize either. Well, yeah, well, that's an interesting thing. Some of these things that AI Go players discovered, human Go players will then learn from that and make their own theory of it and be able to do it. But there could be other things that are just not...
the way the human brain works, right? And you have the same thing in automating mathematical theorem proving, right? Like there's some cases where the AI comes up with a proof and people are like, yeah, well, that's cool. Why didn't I think of that, right? And then you learn from it. You can do other things that are similar. There are other cases where the AI proves the theorem and the AI is just doing it by some insanely huge, messy, cockamamie strategy. Like I have decomposed this into 3,705 separate cases. And for each of these cases, I did this and this. And like, if you have a big enough memory, then why not do it that way, right? But there's...
there's not that much a human can learn from that except yeah let's let the ai prove these these these sorts of theorems and you know it may do something brute force like dividing the 3700 cases but then after that it may do something very abstract and then an elegant following on right like it's not it's not like it's necessarily dumb or brute force and everything it does it it just has the ability to pursue some directions that humans don't have the ability to pursue because of like seven plus or minus two working memory size limitation and various other limitations of the human brain, right? I mean, humans are just bad at math and science and we're bad at Go and chess. Our brains do not evolve to do these things, so it's hardly surprising that some differently architected system can figure out way better ways to do these things. Yeah, it's only recently you can actually survive by playing Tress.
I want to wrap this up with a kind of out there question. It's kind of circling back to AGI and what you said. Once systems can write themselves and iterate on that, you know, that's AI code, right? And so to take this down the logic tree, we start with ephemeral symbolic apps. It's creating these symbolic ephemeral apps. And that's just kind of translating to logic, translating from natural language to logic or whatever.
language to language, code to code. And then we think about it generating, like you said, AI code, which that takes me to like talking about synthetic data, the idea that LLMs or whatever AI system architecture is used can reliably generate synthetic data to train itself or other LLMs. And that's ephemeral in nature, meaning it's fast enough to train and conjure that up in real time and then execute on it. What does that look like? Is that...
I mean, is that bizarro land? Like, what is happening when it's no longer static, but ephemeral? Yeah, I mean, I think that's a lot of what we've designed the OpenCog Hyperon framework for. So, I mean, you have this big knowledge metagraph, where a metagraph is like a graph where you have links that can point to multiple nodes rather than just two, or links pointing to links or subgraphs. And your program code is just graph content, right? And if you know the links have types, the types themselves are just directed to cyclic graphs living in the same metagraph. So, I mean, then...
You can introduce new type systems. You can introduce new programming language interpreters. Like, I mean, you can, everything can revise everything else in real time across multiple processors and across multiple machines. Right. And I mean, that's, that's, I think is a different level of mind than the human mind. Like we, we have some self-modifying ability. I mean, more than the transformer neural net, neural net does in a way. And I mean.
Over our lives, we learn how to learn and even learn how to learn how to learn a bit. And of course, if we lose a chunk of brain in a brain accident, some other parts of the brain can regain many of the functions of that part that was destroyed. Like if we're lucky, not always. But having a system that can in real time always analyze its whole knowledge base and can replace whatever it wants, any part of its own processing with something else. I mean, this is a...
This is a whole other thing, right? Yeah. I'm so fascinated by this. It's a different level of intelligence than humanity has. And using our brains with that level of intelligence will be a whole other fucking human experience to undergo. That's like an acid trip to the power of an acid trip, right? But controllable and rationally decidable and guideable to an extent, right? So it should be a very interesting journey we and our...
engineered artifacts go through during the next decade or two. Yeah. And with the problems out of us with global warming, et cetera, we're going to need all the help we can get. I just want to wrap up on this note. I think a lot of people use LLMs and this idea of like text completion, like finish this sentence, finish this essay, finish this paragraph. But when it comes to code, like write this function, finish this function.
And where it gets interesting as a marriage, this is kind of when I was listening to your podcast, it was resonating with me. This hybrid marriage simply done today is just this idea of fill in the blanks, right? Which is not complete this code, but fill in the blanks of this code. So if blank, then blank, letting the LLM fill in the blanks is an example of symbolic and neural joining together and keeping it on the rails. And so if you have these templates, let's call them these prompt templates in code versus language that are in a graph. Now you have potentially a way to
to kind of ensure a more reliable output of code than just it generating it from scratch. And I just wanted to get your thoughts on that. Yeah, I mean, there's going to be a lot of different routes to get to reliable code generation. I mean, I think for cases, you know, for cases where you have an exact spec for what you want, we'll just be able to use machine learning to machine reasoning to automatically synthesize code doing what we want. But what that means is that all the hard thinking goes into...
creating the spec for what you want the code to do, right? And then, I mean, that's already a lot of what programmers do, of course, is sort of dynamically figuring out what the code should do. I mean, so I think one of the illusions about automating programming is like, if LLMs or other technologies completely automate the writing of code, this eliminates a programmer's job. But of course, probably a quarter or, I don't know, 35% or something of what a programmer does is actually writing the code, right? I mean, it's talking to other stakeholders, figuring out what the requirement is, evaluating if the thing actually made sense.
All that takes up more of your time than typing code. And that stuff requires more general intelligence than writing code. That stuff requires contextual understanding. And I think AIs will learn how to do that stuff by doing it together with people during the next few years. So it's going to be key to have interfaces and methodologies where...
people and AIs work together on creating requirements and on evaluating functionality of systems. By doing these things together with people in different domains, then the AI will learn to do these softer aspects of software development as well, which is really important because for the AI to self-modify itself toward the singularity, I mean, it totally needs to be able to figure out its requirement specifications and qualitatively evaluate its function in a context-appropriate way. That's at least as important as having the... the ability to string together formal symbols in a software program. So if a program itself, if the system's able to execute the program itself in some fashion and get an output, then it's able to iterate on its own, then through that iteration, we could end up with more reliable outputs. Yeah, it depends what you mean by reliable, though. I mean, if you have a formal spec...
I mean, you can just use automated theorem proving to find a program that will reliably execute that spec, assuming your model of the underlying hardware is correct and so forth. And you can then use machine learning to make that formal program synthesis algorithm go faster or something, right? But then I think the subtler question is whether the formal spec is actually what you informally wanted as a human being. And that's more of an AI problem, really, because human life is never concisely formalizable. an ATI problem and self-modifying, self-improving code can help with that also. And that brings us toward the ATI, right? And that's the sort of thing we're aiming to do with the open code system. So will AI be able to read our minds someday? Is that... Well, sure. I mean, I think...
There's a couple of aspects there. One is like, can I tell what my dog is up to usually without looking into its brain? Cat, not so much, right? But the thing is, can an AI, if it's so much smarter than us just by watching what we do, can it just figure out what we're up to? Probably a lot of the time if it's way smarter than us. I mean, and if it's grown up alongside us. So yeah, I mean, of course.
Brain imaging is a whole other thing. And AGIs will probably invent great ways of imaging the human brain through the skull. And we will then be able to train AGIs and human ways of thinking fairly directly from our brains, right? So, I mean, there's a lot of... We'll be the ones entertaining them. Yeah, yeah, yeah. A lot of avenues for video game and VR porn design as well as mind uploading into amazing higher dimensional math realms and stuff that would take a few more podcasts to unravel. But I would say we're not too far from unleashing all this, which is pretty cool. This is awesome. I think it's going to be...
first it'll, it'll, it'll be better at knowing what we need than what we want. And then what we want will be the hard part. Um, but that's, this is awesome. We'll change and what we need will change as our world evolves due to the advent of, um, AGI as, as, as well. True. Well, this is great. Really enjoyed having you on. This was, this is even better than the last conversation in my opinion. It's fun, fun, fun stuff for sure. Yeah. Yeah. Thanks for, uh, thanks for indulging the speculative corners of my brain and, uh, yeah, until the next time then. Yeah. Thanks for having me.
All right. Well, thanks for hanging out with us again here on Invisible Machines. Don't forget to like and subscribe wherever you get your podcasts. Always remember that you can watch this podcast, too, on our Invisible Machines YouTube channel. Head over to uxmag.com. We have a podcast page there where you can explore our back catalog and you can look at episodes centered around popular topics. Many, many thanks to our incredible production team here in the U.S. and in Ukraine, Elias Parker, Michael Lipinov, Allison Harshberger, Ola Slessarenko, Vera Pakhodkov, and Darina Moskovchuk. Till next week, Owaki.