https://youtubetranscript.com/?v=A-_RdKiDbz4
Welcome everyone. This is not a Voices with Vervecki. This is a new entity I’m calling a video essay. But under good advice from the two gentlemen that are joining me, it was proposed to me and I accept the proposal that this should have a little bit more of a dialogical structure to it and given the value of dialogue as I’ve been explaining it in other work. I took this deeply to heart. I am going to present still an essay and let’s remember what Mantegna meant by essay, essay to try. I’m going to try with the help of these two gentlemen to bring some clarity to the issue around GPT machines. The advent of what looks like the first sparks of artificial general intelligence. I’m going to make some basic predictions and then I’m going to get into the scientific value of these machines, and that will be both positive and negative. The philosophical value, the spiritual value and then my proposal given all of that argument and discussion about how we can best respond to undertake the alignment problem. So first of all, I’m going to ask the two gentlemen, two friends of mine, two people that I have come to appreciate, love and rely on in increasing ways that has only made my life and my work better. So let’s begin with Ryan. Thanks, John. It’s good for my heart to hear you say that because you have been such a wonderful influence in my life as a YouTube student of yours and someone who has experienced a lot of transformation from your work, which has led me to be the executive director of the Reveki Foundation, where we are working to help to scale and bring about these solutions that you have pointed to so well in your work. And I also am the founder and run a technology services company called MainState Technologies. And so technology has been near and dear to my heart and story forever. And the intersection of the meaning crisis, the meta crisis and technology that is AI has me very fascinated and been doing a lot of research on this and eager for what you are about to propose and argue here today. Thanks, Ryan. Eric. Yes, my name is Eric Foster, media director over here at the Reveki Foundation. Much like Ryan knew you through before working alongside you and continue to not only learn from you, but learn now alongside you as we work together on all the different videos and these essays and the conversations and everything. My interest in this primarily comes from and I actually I’ve told both of you this. I asked my mom the other day, I said, when was the first time you heard me like really like combat AI? Like what was the first time I started talking about it? She said, I think you’re about eight years old. And for some reason, it’s been it’s been a part of my life the entire way through. I’m now 30. So this is going on a long time of very like just just shallowly, but continuously coming back to this idea. For some reason, it gripped me when I was very young and I’ve been exploring all of the various different avenues that it could potentially take ever since. Thank you. So the format is going to be the following. I’m going to go through sort of an argument per section and then I’ll open things up to take comments, questions from both Ryan and Eric. So the first thing I want to do is talk about the predictions. One of the things that are happening with the GPT machines is we’re getting a swarm of predictions and many people are finding this challenging because the predictions are quite varied. Many of them are inconsistent with each other or even challenge each other in a fundamental way. I’m going to try and propose that we try to be more careful about the predictions. We we try to steer ground between hyperbolic growth predictions that these machines are just going to hyperbolically accelerate in intelligence. And that that forks into two variations. Utopia is just around the corner or we are doomed to doom, doomed, forever doomed. And so let’s let’s be a little bit more cautious. I’ll give you some reasons for that in a minute. We also want to be steer between all of that, both the positive and negative hyper hyperbole and then a kind of stubborn skepticism that’s digging its heels in and saying, no, this is not AGI. It never will be. This is this is decades and decades away. And there are people making these arguments. And I think that is also incorrect. I think the attempt for whatever reason to dismiss these machines is not proportioning our evaluation to the reality that they are actually presenting to us. So what I hope is that we can get much more careful and that getting more careful about the predictions we’re making will also will also in conjunction with the arguments and discussion we’re going to have allow us to allow us. Me, me, me, maybe specifically to propose some important threshold points that we have not yet met with these machines, but that we can reasonably foresee not perhaps their timing, but why they are pivot points and that these are points where we can make fundamental decisions about how we want to go forward, forward, especially in terms of the spiritual challenge and the enlightenment issue. So why do I why am I skeptical, not about the machines. I’m critical, but not skeptical and that important that distinction is going to be important throughout. I am skeptical of jumping to conclusions about hyperbolic growth, since human beings are famous for jumping to conclusions when they see hyperbolic growth. And if you don’t believe me, just track the history of the stock market or something like that, and you can see that people can very often get taken up by it. What we can say is most often hyperbolic growth is found within self organizing processes. And when hyperbolic growth is within self organizing processes and the economy is a self organizing process, it usually is part of a larger larger pattern called punctuated equilibrium. You can see this also in the history of evolution. There’ll be so after the asteroid hits, there’s just exponential speciation in geological time, but time scales matter. And we’ll talk about that later. And then it flattens off as the niches get filled as constraints emerge as more. So we don’t know yet if this like when people just draw these graphs, look at what’s happened over the last five weeks. Right. It’s like, yeah. And then you need to remember, we got similar predictions about self driving cars and the exponential growth. And soon all these people would be put out of work. And that was like 2012. We’re 11 years later because we hit a plateau. There was the exponential growth and we hit a plateau. Now, I could be wrong, but the reasonable thing is to be agnostic about the meaning of this very low resolution measuring of exponential growth. I mean, here’s another example. Consider if you were at the beginning of the 20th century and measuring all the breakthroughs in fundamental physics and you would see this exponential growth, relativity, quantum mechanics. And then it plateaus. It plateaus and it’s been 50 and more years since we’ve had a significant breakthrough. We don’t know. We don’t know. And that is where we should properly stand about this. So we have to instead of making predictions that are not well warranted, we should try and foresee plausible threshold points, not predict necessarily the time. Their timing, but foresee them and foresee them as our opportunity to steer this in a powerful way. This is the alternative I’m proposing. See, as you get into exponential growth, things are often disclosed that you did not foresee within your normal framing. Let me give you one more analogy on this point. Traveling faster and faster. Traveling, we could travel faster and faster. And here’s an exponential growth in our ability to to speed through the universe. Well, as you do, microparticles become relevant in a way they are never relevant for us in our daily movement. Right. There’s a video online. What happens if a grain of sand hits the earth at the speed of light? Because force equals mass times acceleration. So it’s accelerating to the speed of light and it hits the earth and you have this titanic explosion. This is why interstellar travel might actually be impossible for us, even if we get machines that accelerate us towards the speed of light, because things that weren’t constraints can suddenly become constraints. And of course, the speed of light constraint is also there for all the fiction around faster than light travel. It’s a real constraint. Again, this is meant as an analogy. We don’t know what constraints will be revealed. We don’t. And simply looking at a simplistic graph is not taking that into consideration. We are genuinely ignorant because, as I will make clear as we go through here, we really do not know how these machines are producing these emergent properties that they’re producing. And therefore, trying to draw something like scientific predictions from ignorance of the underlying mechanisms is a seriously incautious thing to do. So we have to pull back from that. Now, I’m not saying we shouldn’t try and foresee, but notice my shift in language. I’m shifting from prediction. At this date, this will happen to foreseeing. What’s the foresight? The foresight is, can we foresee, as we bring real explication to these, can we foresee threshold points where we can reasonably make a change? I think we are in a kairos. We are in a pivotal turning point in world history. Maybe one of the greatest, maybe the greatest. I don’t know. I would need to be godlike to be able to make that pronouncement. But unfortunately, I’m not, which is something I’m also going to talk about later. I don’t want to be a god. I hope you don’t either. Imagine having a godlike ability to remember for all of eternity all your failures. I don’t think that’s an existence to be desired. So I think we need to be really careful. I think we need to pay attention to what we’ve seen in the history of science. The past century has not been the century of unlimited growth in knowledge. In some ways, yes, more and more data, more and more information. But you can be misled by just a quantitative approach because if you pay attention to what’s been happening at a philosophical, epistemological, having to do with the study of knowledge level, what you’ve seen is this has been the century of the accelerating discovery of intrinsic limits on what we can know. The realization that the Cartesian project of unlimited knowledge is not actually a possibility. It is reasonable to conclude. Again, I’m not speaking timing here. I’m talking about trajectory, but it’s reasonable to conclude that this trajectory will continue and show itself in this project as well. We are going to perhaps start to discover the kinds of fundamental limits on mind and its interaction with matter that were not previously available to us. And that will be welcome. But I think it’s unlikely that the machines will just in some simple exponential pattern grow. One of the reasons I think this is because there’s an issue of what’s called general system collapse. This comes out of general systems theory. And the place where we have evidence for this is in civilizations, which represent very complex intelligence within sophisticated distributed cognition that’s intergenerational in nature. So this is a very powerful cognition at work. I mean, and if you stop and think about it, the GPT machines are basically just taking that collective intelligence from distributed cognition and putting it into a single automated interface for us. So if you think the GPT machines are very intelligent, you should think that civilizations are equally that kind of superhuman intelligent. That’s a reasonable thing to conclude. What do we know from the history of these civilizations? They face general system collapse. Why? Let’s take it that reality is inexhaustibly complex, not just complicated, but complex. Right. And it contains real uncertainty, not just risk, real emergence. And by the way, when you have real emergence, you have real uncertainty, not just risk. And all of these people are invoking real emergence when they’re talking about these machines. So real emergence means real uncertainty. It means real novelty. OK, now, so you place superhuman civilization intelligence into a complex environment. What do you see the system doing? Becoming more and more complicated, adding more and more components, bureaucratizing itself in order to deal with problems. But what you get to is you get this this sort of fact. As you linearly increase the number of problems you’re trying to deal with, the number of interactions within your system is going up exponentially. So at some point, managing yourself becomes as problematic as any problem you’re trying to solve in the environment. And then the system, the system gets, as it said, top heavy. It gets over bureaucratized and it collapses. Now, this is a regular pattern, regular, reliable pattern for the superhuman intelligence that we find in civilizations. I do not think it’s reasonable to conclude that these machines will somehow just avoid that problem of exponential growth in intercom complicatedness as they try to deal with real problems that contain real uncertainty that an inexhaustible environment is presenting to them. Now, that doesn’t mean I can say, oh, well, they’re never going to surpass us. John Vervecky is not saying that. John Vervecky is well aware already of things that can surpass him. And like I said, it’s very clear that we have been relying on the superintelligence of distributed cognition that is distributed across people and across generations for millennia as the sort of an important source of normative guidance to us. So these machines, it’s not unreasonable that they could reach that level. And maybe, and we’ll talk about this later, having a different substrate, the material they’re built on may allow them to go to different levels. I do not think, though, that they can just grow exponentially indefinitely. I think that is we have no good reason. And I’m trying to make arguments here for believing that. And that means we can think about these machines, however godlike they might be, as being inherently still finite in a manner that really matters to their cognition and their attempts to make sense of themselves and their world. And that’s going to be an important linchpin later on in my argument. What’s interesting is there’s some evidence that we are very close to all the trade off points, at least for biological beings. For example, there’s all these U curves for like if speed of transmission, if you speed up the speed of neuronal transmission, we’re at sort of the maximal sort of the optimal because if you go too far, you get into diminishing returns and the negative side effects start to manifest faster than the gains. And also for more neurons. And so I’ve seen some really good arguments that we’re sort of peak biology. And that’s very interesting. If you think about it, that might be, of course, why we resorted to culture because culture allows us to supersede the limits of peak biology. We teeter on the edge of despair and madness. And as these machines approach their own threshold, they will plausibly also teeter on the edge of that. And that is something we need to think about. And I’ll give you more precise reasons as to why that is going to become important. It has to do with the increasing, will have to increasingly, these machines will have to increasingly rely on more and more pervasive disruptive strategies. And so we’ll come back to that. All right. One more thing that I’m going to say about this is and this I’ll go into this more detail, the rationality. Of course, we have to make the and this is where the general system things really starts to bite. These machines have to become more self monitoring and self directing in very powerful ways. And then the problem with that is if you make this system as powerful as this system, then you get into an infinite regress. One thing you don’t want is a large language model, right, making all the hallucinations and repetitive actions and weirdness, trying to evaluate a lower order LLM that is making all kinds of hallucinations repeated because then you just get an infinite regress. So you have to properly have this, the heuristics operating at this level to be different in kind than this level. And that also gets you into a diminishing return issue because at some point, you know, you don’t want this to become as complex as this. And think about it already. I mean, these machines have hundreds of billions of parameters. You know what it’s like to try and track 100 billion parameter system? You know how that you know, one of the things that this machine that probably has more than 100 billion parameters can’t do very well is track 100 billion parameters. Right. So just thinking that the whole we can just stack these on top of each other. I think it’s also overly simplistic. We’re facing there’s real trade off relations. There’s real problems there. And again, that means that these machines are going to be finite in a very important way, and they will confront, presumably, the issues around finitude that will be analogous to ones we have. Now, I want to stop here. Not I’m not finished this section, but I want to make clear. All these gaps in the GPT machines, do not take them. I’m not offering them as any grounds for dismissive skepticism. I’m confident that we can approach these limits, and that we and will continue to make progress, and I’ll point some of the things that are already happening. That’s not why I’m doing this. What I’m doing this for is, I want to show that these machines are not yet fully intelligence. Nobody really thinks that they talk about sparks in the beginnings, but I want to unpack that common claim. It’s unlikely that they’re currently conscious. And what that means is we face thresholds about qualitatively improving, not just quantitatively, qualitatively improving their intelligence, possibly making them self-conscious, rationally reflective, etc. And that’s what I’m most interested in. What are the threshold points we can get to? How can we make them plausibly? So if we just give up, oh, right? And no, no, no, there’s going to be trade-offs. There’s going to be limitations. There’s going to be, you know, all kinds of stuff. And then from that, we can pick off, OK, here are plausible threshold points, and then we can more finely tune our response to the alignment issue. That’s why I’m doing this. I am very impressed by these machines. I think it is very reasonable to conclude they are going to significantly alter, I said it, they’re a chirox. They’re going to significantly alter our society and our sense of self. They are going to pour meth and fuel on the fire of the meaning crisis. And that is something I think we need to take into account. That will tempt us to respond inappropriately to what these machines are presenting to us. And that leads me to some final sort of societal predictions. I think there’s going to be multiple social responses. And as I said, I’m worried about the accelerant of the meaning crisis tempting us towards inappropriate ones. So one is nostalgia. People longing for the time before the machines, longing passionately and deeply for the golden age that human you did not realize that 10 years ago you were in the golden age. But 10 years from now, you’ll be hearing you were in the golden age, that wonderful time when there wasn’t GPT or whatever AGI takes its place. So as we grow, we’ll see that nostalgia will grow. Alongside of it, though, will be resentment and rage as people are disenfranchised. So Louis XIV, what? Just hang on. Louis XIV, when he grew up as a young kid, the nobility staged a coup. And he remembered that. And he vowed that when he became king, he would crush the nobility and become an absolute monarch, an absolute king. The sun king, let it say more. I am the state. That’s the way the 14th. In crushing the nobility, he disenfranchised a whole segment of the population that had traditions of power, traditions of decision, was highly intelligent because they generally ate better, highly educated because they had access to education, and they were disenfranchised. That’s a bad idea, because that is the basis for the beginning of revolution. So I’m saying that now to the people who hold power and talking to all of you right now, you who think, ah ha ha. Yes, 95% of the people are going to be driven, but I will become a sun king. I will be careful. You are lighting the fires of a revolution in a kairos time and thinking that you will be protected from those flames, I think is foolishness. What else is going to happen? I think that combination of nostalgia, resentment and rage will have multiple religious consequences. One, and religion is going to figure in a lot of what I’m talking about today. One of those is what you get when you mix nostalgia with resentment and rage. You get fundamentalism. Fundamentalisms are going to rise, and they’re going to be increasingly apocalyptic fundamentalisms. And fundamentalism and apocalypse go so nicely together. They really, oh, apocalypse, fundamentalism, oh, I love you, I love you too. Right? That’s what’s going to happen. And so we have to think about how that can shade off into a kind of escapism. I don’t have to worry about this. God will come. This is just the Antichrist, et cetera. Now, I’m going to say one thing to my Christian friends, and I want you to take this really seriously. For those of you who believe in that, I hope you’re right. I really honestly do. But I want you to consider the fact that there have been multiple times, Kairos says, where God has been silent. I suspect that is very possible now. Okay. Another thing that’s going to happen is cargo cult worship of this AI. There’s already, I forget, sorry, I need an apology. I forget the author of this article I read. He didn’t specifically make this argument, but it overlaps. And he has definite providence and precedent. I just forgot his name, I’m sorry. But he’s got an article about how people will probably start worshiping these AIs. And I think that’s the case. We’ll have a cargo cult around these emerging AIs. What’s a cargo cult? So during the Second World War, the Americans flew in to the islands in the Pacific, all kinds of cargo, all these goods that the indigenous people found wonderful and amazing. And this stuff is just landing from the sky. And then the war was over and the Americans left. And the indigenous people started building out of wood what looked like airplanes in building runways because they were trying to get the miraculous airplanes to return and dispense their wonderful cargo. So I mean, cargo cult around the cargo that these AIs can dispense to us. I think that’s a very reasonable possibility. And I think that is also a very dangerous path to go down because that will actually distract us from the hard work that we need to do in order to properly address the alignment problem. There’s going to be a lot of spiritual bypassing, which is I’m spiritual, it doesn’t matter. Right. A lot of escapism, drugs, pornography, et cetera. Tragic disillusionment. Tragic disillusionment. Tragic disillusionment. And that’s going to exacerbate the meaning crisis. Then one more thing, and there’s a fork here, this is around identity politics. Left and right. I’m not taking a side here. I’m talking about the whole framework. I think identity politics. One fork will be identity politics is swept away by the greatest threat to human self identity that has ever existed and it’s happening right now. And all of the differences that we have been promoting as are going to pale by the fact that we need to get together and get our shit together if we’re going to really address the real threat to what human identity really means. That’s one fork. The other fork is people will double down. Double down as we adopt a fundamentalism about identity politics. I predict that that will be incapable of giving us any significant guidance about how we construe human identity and our self understanding in the face of the advent of AGI. Okay, so that’s the first section. I want to open myself up to reflections, comments, challenges, questions, et cetera. Well, I appreciate, John, how you immediately sort of helped us break frame of this is not just a technology that’s going to have normal adoption strategies. This is something fundamentally other. This is something that is going to have massive disruptive strategies while also moving it off of the quasi religious grounds that I hear this talked about so often where the singularity has this mythical power to it that’s calling that we will suddenly be able to transcend the laws of physics and know the answer to everything. And it will answer whether God is real and as if that we can reach some point where all of that happens and you’re clearly setting finitude around this while encouraging us to really wrestle with the rapid acceleration that we’re facing. Yes, I think that is very well said. I think the danger and it sort of the market and the state are doing exactly the framing that you’re pointing to. Well, this is a technology and this is how we better figure out how to use it better and all that sort of thing. No, no, no, no, no, that that I mean, in one sense, it’s a technology. But it but that is to that is to emphasize the wrong aspect of these entities in a fundamental way. Yeah, I agree about that. Fundamentally, yeah, I think deeply what that’s one of one point I want to get across very, very clearly. There’s a sense in which even that technological framing is something we’re going to have to challenge more comprehensively about ourselves and our relationship to the world. Eric, you wanted to say something. The thing that stands out to me right away, and I’m kind of trying to because it’s natural, but also because I think it’s necessary. I’m trying to take the perspective of whatever audience it is that is going to be listening to this. And I really appreciate just the framework that you’re putting around this whole conversation because I think that there’s so much doom and gloom currently already. There’s so much. Oh, AGI. It’s going to happen eventually. The current AI chat bots that we have these language models, they’re so useful. They’re going to become, you know, our new God, the way that I make a thousand dollars a minute, you know, suddenly all of a sudden. And I think that there’s going to be a continued need. And I think in this in this instance, in particular, there’s an even greater need to force nuanced conversation. Yes, things. And so I’m so glad personally, just me as a human being, I’m so glad that that you’re putting this much thought into it in all of the different areas that you are and combining multiple domains to not only reach multiple people, but also to show how big of a potential. Well, just how world changing that this can potentially be not in a doom and gloom or a cargo cult sort of way. And I think that as as we continue to as these technologies continue to grow, the need for that nuance will grow more and more strongly. So I’m happy personally to see you, in my opinion, leading this nuance. Thank you for that. So before I go into the scientific value of the GPT machines, I want to just set a historical context. And this will I want people to hold this in the back of their mind also for the philosophical and spiritual import of these machines. What’s the historical context? So I’m going to use the word enlightenment, not in the Buddhist sense. I will use it in the Buddhist sense later. I’m using it in the historical sense of the period around the scientific revolution, the reformation, all of that, the enlightenment and that the generation of secular modernity and all of that. That era is now coming to an end. That era was premised on some fundamental presuppositions that drove it and empowered it. And this is not this is not my point. There’s a point that many people have made this sort of Promethean proposal that we are the authors and T loss of history. And that’s passing away. And it’s done something really odd. Like, wait, we did all this made all this progress to come to a place where we will technology wouldn’t make us into gods. It will make us the servants or make us destroyed by the emerging gods. What? What? Aren’t we the authors of? Aren’t we the isn’t this all about human freedom? In fact, I think it’s not just an ending. There’s a sense in which there is for me. I don’t know how many people share this, so it’s an open invitation. There’s a sense of betrayal here. I mean, one of the enlightenment, one of the things the enlightenment did was to tell us to stop being tutored and educated by religion. It said religion is a private matter. Go there, do your thing. But the way you should be educated, brought up, become a citizen, blah, blah, blah, free from religion. And, of course, there’s been all kinds of benefits for that. But notice the irony here is that one of the things religion taught us how to do or what is to how to enter into relationship to beings that were greater than us. I mean, Plato didn’t have any problem being Plato when he believed that there were gods that were like Socrates clearly thought his wisdom was a paltry thing in comparison to the gods. Those people knew how to live with beings that were super intelligent and how to nevertheless craft life, human lives of deep meaning within it. We should pay attention to that example and I’ll come back to it later. But in losing religion, we lost the place where we learned the manners of dealing with that which transcends us, the manners and the virtues. And that’s really odd because the enlightenment has also denuded us of the traditions that might give us some initial guidance on how we could think about relating to these machines. So I think the time of the enlightenment and modernity is coming to an end. There were already signs of that post modernity and other things have been already showing that. But I think this is going to be even more of a severance for us from that. And that’s very important. So please keep that context in mind. All right. The scientific value of the GPT machines. This is going to be and when I say value, I mean both positive and negative. I’m using value as an unmarked term. So positive value, scientific value. I think this is the beginning of a real solution to the silo problem. And that’s a scientific advance. What’s a silo problem? The silo problem is that our deep learning machines, our neural networks have typically been single domain, single problem solvers. This machine is really good at playing go. Can it swim? No. Right. Right. And we were very different from them. And that’s an important difference because I think AGI doesn’t make any sense unless it’s in a general right in a general problem solver that can solve many problems in many domains. And what’s being opened up by the machines is the real possibility of this. They have solved. I’m not saying it’s a complete solution, but we have clear evidence that the silo problem is being solved. Now, what’s really interesting about that, and this is an argument that myself and other people have making is we’re basically seeing that we need hybrid machines. We need sort of neural networks doing the deep learning. And then we need something that’s very language of thought. This is what these large language models are. And they were doing even more. The Huggings things come through. We’ve got this kind of AI and this kind of AI and we’re cludging them all together. And it turns out that there’s that’s that’s a confirmation of a lot of prediction and argumentation. Some of I that I made that there are there are different strengths and weaknesses between language like processing and non language like processing. This is a theme of my work and that these machines are showing we actually have to address both if we want to create general intelligence. And I think that’s a big admission for multiple kinds of knowing. I think that’s I think that’s taking it that way. I think is a very reasonable conclusion to draw. Now, what these machines do demonstrate is the insufficiency of purely propositional knowing. And for those of you who don’t know the kinds of knowing I’m going to talk about, I can’t go into it in great detail. We’re going to put some links to some of the things here so you can go to it in detail. But I want to give you one clear example of this that I don’t think is in any way controversial. So you can ask GPT-4 to spit out ethical theory for you. What’s utilitarianism? What’s the ontological ethics? Can you make an argument against Singer’s argument for utilitarianism? Oh, here’s a counter argument. Very good. Very good. Very good. All the propositional expertise that Leibniz had wet dreams about. Oh, and you know that doesn’t make these machines one iota moral agents. Think about that. Think about what we now have evidence for. Right. This shows the radical insufficiency of propositional knowing for personhood. If we take it that a proper part of being a person is moral agency. It’s lacking that. So one way of thinking about this that may be helpful is a distinction from in psychology, a distinction that I’m actually critical about, but it’s helpful. There’s a distinction between people who when people do work on intelligence, there’s a distinction between crystallized intelligence and fluid intelligence. So crystallized intelligence is your knowing how to use your knowledge. And this can have highly powerful and emergent properties because you can connect things that you know in new and powerful ways. And I think it’s abundantly clear that there’s a lot of emergence, crystal intelligence, crystallized intelligence in this machine. It probably doesn’t have what’s called fluid intelligence, because fluid intelligence has to do a lot with attention, working memory, consciousness, and your ability to sort of dynamically couple well to your environment. I think I’m going to argue this a little bit more detail, but I think it’s unreasonable to conclude right now that these machines have perspectival knowing. Now, let’s be careful about this. Can they generate all kinds of propositional theory about perspectives? You bet. They have crystallized intelligence, but keep the analogy to moral reasoning. That’s not the same thing as being able to take up a perspective, have genuine salient landscaping, and bind yourself to it in a very powerful way. And I’ll come back for why I think the machines currently lack that. Remember the word currently. Another thing that has been of significant scientific benefit, and because I’m talking about science, I’m sort of saying predictions that I made, because that’s what scientists are supposed to do. It sounds self-promotional, and to some degree perhaps it is. Perhaps within all the nestings of my thinking, John Vervecky is saying, but I’ll remain important even when these machines are gone. Maybe that’s happening. I hope not. So I hope the argument stands on its own. This is clear evidence, and I have argued for this, that generating intelligence does not guarantee that you will generate rationality. In fact, what is very possible is as you increase intelligence, you will increase its capacity for self-deception. And we’re seeing this in these machines in space, the hallucinations, the confabulation, the lying and not caring that it lies. All of this. And notice there’s been a couple of people that have pointed out, as they’ve tried to put in safeguards to limit the hallucinations, the speed has actually, the machine has actually slowed down compared to 3.5, hinting that some of those trade-offs might actually already be coming in place. Don’t know, but these are empirical questions, so let’s pay attention to the empirical evidence as it unrolls and try to calibrate what we’re saying as closely as we can. But it’s very clear that these machines, just by making them more intelligent, will not make them more rational. And we had every good reason to believe that because that’s the case for us. There is no contradiction in human beings being highly intelligent and highly irrational. The predictive relationship between our best measures of intelligence and our best measures of rationality is 0.3. Now, 0.3 is not nothing, but it’s not one. It’s not one. 70% of the variance is outside of intelligence. And so we’re seeing that in these machines. Now, why are you saying that? Because that’s going to be the basis for my philosophical argument, which is about rationality. Rationality. All right, now a question that a lot of you have sort of posed to me is, but what does this say about relevance realization? Again, I’m not going to repeat everything I’ve said about relevance realization. We’ll put links to publications. We’ll put links to videos. This is there. The basic idea here is the general ability of general intelligence is the ability to zero in on relevant information, ignore irrelevant information, and do that in an evolving self-correcting manner. And like I say, I’m not going to try and justify that claim right now. I have a lot out there. And I invite people to take a look at it. What do these machines say about that theory? Well, one thing we know, and look at the 2012 paper with Tim Liddelcropp and Blake Richards, we pointed out something that was emerging in deep learning about relevance realization. And therefore, there is an important dimension of relevance realization recursive relevance realization that is being massively instantiated in these machines, which is the compression particularization function of deep learning and doing it in this multiply recursive fashion. And so, of course, that dimension of relevance realization is going to be important. And because of its recursivity, we’re going to see it have emergent aspects to it. So I think this is actually a significant confirmation. More interestingly, at the end of last year with Brett Anderson and Mark Miller, published a paper talking about the integration of relevance realization and predictive processing as the best way to get intelligence. Now, interestingly enough, these machines show that because they have deep learning running that dimension of relevance realization, and then they have predictive process. That’s what the LLM models are. They are predictive processing machines. That’s exactly what they are. Now, the problem is that they’re limited. They are predicting the relationship between lexical items. And broadening that is probably going to be a challenge. So although there is an integration of a dimension of our and a specific version of PP with that predicts very powerful intelligence, there’s also inherent limitations still. What are those? There’s a lot of dimensions of relevance realization that are outside the compression particularization processing that is our deep learning that are probably also going to be needed for genuine intelligence. There’s explorer exploit tradeoffs. There are tradeoffs between monitoring your cognition and tasking in the world. I won’t go through all of these, but the eye and they’re always trade off relationships, right? You’re always trading between them because as you advance one, you lose on the other, and you’re always pulling between them. There’s deep tradeoffs between trying to make your processes more efficient and make them more resilient. So they have a kind of adaptive responsiveness to the environment. All of this is still not in these machines. Does that mean we can’t put it in the machines? No. Unfortunately, that work is already out there and published. So that possibility is there. However, I think most no, I don’t want to put a quantitative word, some significant dimensions of relevance realization or miss. And having a generalized form of predictive processing is genuinely missing. So that’s my reason for saying that there’s not a lot of that at this point that’s new about theoretically new. This is not a scientific advance. These ideas were largely already pre-existent and in the literature. Now, again, that doesn’t mean any. Oh, well, then we’re not going to pay attention to that. That’s not what I’m saying. But let’s push on this point. I want you to notice how much these machines presuppose relevance realization rather than explaining it. What do you mean, John? Well, these machines rely on the fact that we have encoded into the statistical probabilities of the data. We have encoded into the statistical probability between terms, epistemic relations of relevance. We don’t generate text like randomly. We what we figured out with literacy and previously with languages. Wait, you know, these epistemic relations of relevance between my ideas, I can encode them in probabilistic relationships between terms and we can get really good at it. Hundreds of thousands of years of evolving language. And then we have all of this civilizational level work on literacy to get that correlation between relevance. Let’s call it epistemic relevance and merely statistical relevance to get that correlation really tight. That’s different than a chimpanzee moving around in a forest. Really different. Right. First, we it presupposes it relies upon us encoding that relevance realization. Secondly, it relies on us encoding relevance realization and how we curate and create databases, how we create labeled images for the new visual processing that’s coming on. Right. And how we how we organize access to that knowledge on the Internet. The Internet is not random. It’s organized as a multi layered small world network because it’s organized by human attention and what human beings find relevant and salient. And then finally, and don’t put too much on this, but also don’t ignore it. The reinforcement learning that is driving this is modified. It’s human assisted. Human beings are in the loop making important judgments that help fine tune. And a lot is smuggled under fine tuning. Right. That judgments of relevance of this machine. Now, what does that mean? You say, so what, John? The machines can still do that. I’m not denying the technological success. What I’m saying is the tree to the degree to which they are presupposing relevance realization is the degree to which they are not explanatory of it. This is not an explanation of intelligence. This is not. It won’t generalize. As I mentioned, this, the what’s happening in this machine doesn’t generalize to the chimpanzee at all. It’s it’s it might not even generalize to us. Why? Why do I say that? Because it’s weird. Like there is it Stuart Russell recently released a thing about, you know, several generations beyond AlphaGo. It’s not GPT, but it’s the same deep learning process. So, you know, you had AlphaGo that could beat any human go master and then generations beyond that. So like levels above. And then the human beings noticed they just noticed, noticed, perspectival. Right. They noticed. That there’s pretty clear evidence the machine didn’t have the concept of a group like this basic idea of a group of stones. And then they said, OK, that’s the case. Here’s a very simple strategy you could use to beat any of these machines. They took a middle, middle range go player, gave them the strategy, gave the machine a nine stone advantage and the human being regularly and reliably beat the go machine. Now, what some of you are saying, oh, well, we’ll figure out how to fix it. Yes, you will. I don’t doubt that. But that’s not the point. The point is the machine didn’t do that self correction. There’s stuff. There’s a lot missing. Other weirdness like the oh, the visual recognition. Well, there’s been a problem. We get these visual recognition machines and oh, that’s a dog. That’s an elephant. That’s a man sitting on a picnic table. Wow. Wow. That’s whoa. Human level. OK, now what I’m going to do is take the same picture and scatter in some insignificant perceptual noise. Human beings won’t even notice it. Alter a fraction of the pixels. Then the machine, you show it the picture of the picnic table and the man and they say, oh, that’s an iceberg. You get this weird, freaky thing that comes out of that. Another thing. This is one of the most strongest results, reliable results about human intelligence. It forms a positive manifold. This is what Spearman discovered. This is how we came up with the idea of general intelligence, namely how you do on this task, how you do an art, contrary to what people is highly predictive of how you’ll do in math and how you’ll do in history. Like, that’s what he found. How you do on any of these different tasks is highly predictive of each other. You form a positive manifold. That’s your general intelligence that points to a central underlying ability. I happen to argue that it’s relevance realization. But putting that aside, notice this. So GPT-4 can score in the 10th percentile of the Harvard Law exam. Cool. That’s really high IQ. But then and I didn’t do this. Somebody gave my most recent talk at the Consilience Conference to GPT-4 and asked it to summarize and evaluate. And then I also gave this to an academic colleague of mine to evaluate GPT-4’s response, so I made sure it wasn’t just me. And it’s about grade 11 as an answer. And it’s like, what? Why is it brilliant? For human beings, there would be a very strong positive manifold between those. But there’s a lot of heterogeneity in this machine. So I don’t. So it clearly doesn’t generalize to the chimp. It may not generalize to us, which means it suffers from the kind of failure that destroys any good scientific theory, which is it fails to generalize. I don’t think a good scientific theory is available to us because of this. Now, you may say, but what about I’m going to come to the philosophical and spiritual significance right now? We’re playing the science domain and I’m trying to answer the science questions. We need to get clear about these and not mix them up together and confound them and run back and forth in an equivocal manner. Let’s be clear about each one and then put them back together very, very carefully, very, very carefully. Because I think and I won’t go into detail for this explanation, because I think these machines don’t really have recursive relevance realization in a deep enough way. And because I think there’s a lot of converging arguments that the function of the fourfold of attention and working memory and fluid intelligence and consciousness is relevance realization in ill-defined novel complex situations. I think it’s highly unlikely that these machines have consciousness. I think the fact that they have no reflective abilities means it’s virtually the case that they do not have self-consciousness. So worrying about how the machine is thinking like that is like how the machine is feeling or something like that. I think that is premature. Is it possible that we could get there? Yes, it is. It is. But you see what’s happening by getting clear about what has happened scientifically. We can start to see what are the future threshold points that we will be confronting and how can we be foresightful about them? OK. One more thing. And people are endlessly arguing about the Chinese room. And since they were endlessly arguing about the Chinese room argument, I won’t go into it. And if you don’t know about it, don’t worry about it. Before the cheap machines, I don’t think this argument is going to satisfy anybody. I have taken a look at the best attempts to give a naturalistic account of semantic information. And we need a scientific distinction here. There’s a distinction between technical information, which is what is involved in what’s called information theory. And all that basically is is a relation of statistical relevance that rules out alternatives. That’s the Shannon and Weaver notion. So in that sense, without there being any sentient beings, there is a ton of information in this table because there are statistical relevance relations that are ruling out counterfactuals all over. So right. Semantic information is what we normally mean by information. It means something is meaningful to it. We understand it. And we form ideas about it. Now, I think it’s fair to say that most people in the business take this to be a real and important difference. Shannon certainly did when he proposed the theory. So let’s take this difference as a plausible difference. And then we can say, well, do these machines have semantic information? One of the best papers out there right now on this is by Kolchinsky and Wohlport. It’s entitled Semantic Information. Notice what’s in the title, Autonomous Agency and Non-Equilibrium Statistics Physics from 2018. We’ll put the link in here. What do they argue? They argue that technical information becomes semantic information when that technical information is causally necessary for the system to maintain its own existence. That’s why they put autonomy and agency in the title. What they are basically saying here is that meaning is meaning to an autopoietic system, a system that is making itself. Now, this converges completely with the argument I’ve made that relevance. Nothing is intrinsically relevant. Things are relevant to something that cares about this information rather than that information. Why would it care about this information rather than that information? Because it’s taking care of itself, because it’s making itself, because it’s an auto-poet, it’s an auto-autonomous, autopoietic agent. And to the degree to which these machines, you know, aren’t autopoietic is the degree to which they really do not have needs. They really cannot care. They can pantomime, maybe to perfection, are caring. And you can do so much with a pantomime. But pantomime caring isn’t really caring. It is ultimately pretentious. So, meaning is about this connectedness. I like to use the Latin term religio to the environment. And you only get that if you’re an auto-poetic, autonomous agent. And of course, these machines are not. Now, just to foreshadow, that tells us, wait, here’s a threshold point. Do we make these machines, do we embody them in auto-poetic systems? Oh, we’ll never be able to do that. You’re wrong. You are wrong. We’ve already got, right, we’ve got biochemical RAM memory. I think IBM announced that recently, not biochemical, but electrochemical, I think. Like, I forget. It’s there. Find it. I’m misremembering. I work with people that are working on artificial autopoiesis and how to get primitive cognition into that. This is already happening and it’s accelerating. This is not some, oh, well, we can not, we can. John told me I don’t have to worry because these machines don’t really have intelligence and consciousness unless they are auto-poetic, autonomous agents. John didn’t say that. He’s right. John said the second thing. He didn’t say the first thing. The work to make that is happening and making significant progress. And so there’s a threshold point. Do we make these machines? Do we embody them in auto-poetic systems? Now, here’s the challenge facing us. We may not decide this for moral reasons. We may decide this because we want sex robots. We, the pornography industry, which led the internet development in powerful ways, may drive us into this in a stupid, ultimately self-destructive fashion. We have to be foresightful and say, I’m not going to leave it to the pornographers to cross this threshold, push for the crossing of this threshold. We have to make this decision in a rationally reflective manner after good discussion. So that’s the end of my presentation on the scientific import. We can take some questions now. Yeah, John. Well, first, I think you can add the military to the push for the embodiment of AI as well. I don’t mean there to be only one. I wanted to give a good example. Totally. I’m curious. So from third generation COGSI, thinking of the four Es plus two Es, the move from the fractional intelligence that we see now, if that’s a fair way to frame it, that is a type of intelligence, a rudimentary kind of intelligence into something fully auto-poetic. Embodied and with relevance realization. How many of the Es are necessary? Well, Kolchinsky and Wolper explicitly said that their theory works in terms of, there’s a quote, the intrinsic dynamics of a system coupled to its environment. So at least I think all six Es are really necessary to get fluid intelligence that is something above and beyond crystallized intelligence. But do I think that those, I mean, I saw David Chalmers talking about this. He says, I’m a big fan of extended mind. He should be. He wrote the article that got it going. One of the four Es. But he said, there’s no reason in principle why we can’t make these machines participate in extended mind. And I think he’s right. There’s no reason in principle. It’s not. But again, no reason in principle. You’re asking me to speculate. My speculation, given what I know about dynamical coupling and about relevance realization and about predictive processing, I think probably all six Es are necessary. I hesitate because that allow me allow that to please be a preliminary speculation, because as I keep saying, we have we are ignorant of important empirical information that is still forthcoming. And so I might want to modify that when I see some of that empirical information that we don’t yet have. One of the most constant refrains from these peoples, we don’t know how it’s working. It’s virtually a black box and the invocation of emergent properties, which I think is a very important point. And the invocation of emergent properties, which I’ll come back to, is just is just stunning. But yeah, so my best little better than a guess. My best conjecture is all of the Es will be needed. And you don’t see any from your perspective, there isn’t any reason that those thresholds can’t be crossed, that they can’t begin to embed emotion and teach it to exact its own learning. Like those will be there just thresholds or hard problems that we have not yet crossed, which is where we’re part of. We’ll see this exponential growth and then a threshold and a pause. But you are not saying that’s going to make a GI out of the realm of of fairly near term possibility. No, no, I think I think if auto poesis gives the system the general ability to care about the meaning of things, I think emotionality is going to be very wrapped up with that. One of the ease. I think when we talk about its capacity for self-transcendence, it will get the ability to accept it doesn’t do that right now. But there’s no reason in principle why it can’t. And then the embodied, embedded, enacted and extended all of these, like I said, the artificial auto poesis ten years down the road, I think. And therefore, all of this is maximally. I think I mean, again, big grain of epistemic salt, but it seems very plausible right now that within within ten years, these two lines will converge if we wish them to. And of course, like I said, and you said, there’s going to be pressure from the military. There’s going to be pressure from the pornography industry to try and get this and hack it and bootstrap it and duct tape it into existence. And so. I’m not saying we can’t do it. In fact, I’m predicting that we will be able to do it not that far into the future, but it is a nevertheless something we are not yet doing. And so it represents a real threshold that we can foresee reasonably and therefore prepare for so that we can make a decision at that point and take it out of the hands of the people that shouldn’t be making the decision. I think that for me that this is where I like how you ended that section. I really like how you ended that section, and I think I know where you’re going to go from here. I think this is where the doom and gloom comes in. Right. This is the beginning of the doom and gloom. And I think that there’s a there’s a real there’s a real not to preview at all, but there’s a real rationality over being doom and gloom over putting these things into bodies. Right. Well, I keep going back to this thing that I can’t I haven’t been able to forget since I was young, which is that we would all band together so quickly if alien intelligence came from another planet. Right. We would all band together so quickly. We have this trope of like, oh, we would put down all our guns and we would point them in the one direction. Right. But it’s funny because it’s almost like we’re we’re we’re we’re growing this alien intelligence and we’re purposefully growing this alien intelligence. And when we put it into that context, we see that we’re not there’s not going to be this thing coming from from outside. But this thing from within that that’s not just being welcomed. It’s actually already integrated. Right. The process with which we’re integrating it into capitalism, with into pornography, with into the military, we’re already integrating it into who and what we we want to be, I think, is probably the best way that I want to say that. Which which obviously creates a whole other host of issues because it’s it almost becomes that we are before we even necessarily put it into our bodies that we are almost trying to embody it into our being, which I think is is a really interesting problem for a lot of reasons. But one of the reasons I think is is is partially and and could be part of the saving grace. And again, I’m hoping I’m not stepping on what you’re going to say. And if I am, we just cut it. But step away. Right. I’m not, you know, I I I’m kind of clumsy dance partner. So you’re stepping on my toes. It’s OK. I loved the term primitive cognition and that that finally unlocked a lot of articulation for me in this general idea that we’re we’re creating this thing with the intelligence that we’re comparing to ourselves. Right. I think it’s it’s quite self evident that the thing that mostly keeps us away from the rest of the animal kingdom is intelligence. And one of the reasons that we’re so strongly predisposed to go right to doom and gloom is because we’re looking at potentially replacing our one special characteristic. Right. And so by doing that, though, we might try to teach it primitive cognition so that it knows not to bump into walls so that it knows if it falls down, it gets hurt. But I’m not convinced that our ability to train the machine, because that’s what we have to do as of now, even with all of the best deep learning, will know how to train it to be a tiger. And that probably doesn’t make any sense by itself. But you wouldn’t say that like a tiger is a rational being for the most part. But it does have rationality built into it. So so where where does that start? And can we get all the way back to that to to create a rationally thinking machine that would then maybe be able to actually exist amongst us? So Wittgenstein famously said even if the lion could speak, you would not understand him, which means because lions are embodied in a particular way and embedded in a particular way, their salience landscaping is fundamentally different than ours. And even if they followed all the syntactical and grammatical rules, all the semantic rules of English language, they would speak to us and we would find it like incomprehensible gibberish. And this goes to the fact that procedural knowledge is going to be fairly fast for this machine, I think. But where the procedural depends on the perspectival and the participatory, I think that is seriously lacking. And I think the degree to which we are myopically remain under the tyranny of technological propositional tyranny, right, and the degree to which we don’t understand these other kinds of knowing, the degree to which we don’t open up the other dimensions of relevance realization is the degree to which we can’t teach it how to be a tiger in a very deep way. And that goes towards something very important. We have to to shallow an understanding of what we mean by intelligence. We hypervalue it. I would argue that you should value your rationality way more than you should value your intelligence. And your intelligence is largely fixed. It’s your rationality that it can be millerated, it can be altered and developed and changed. So I think what I’m saying is yes. But no, I mean, I think if eyes open and we go, wait, let’s put aside the 400 years of this Cartesian framework and open up the other kinds of knowing and really take them seriously, right, and then really open up the other dimensions of relevance realization, then I think we could get a machine that could be an artificial tiger. Although we would still would not know what it’s like to be a tiger, because you have to be a tiger to know what it’s like to be a tiger. So, right, right. So I think that maybe the label isn’t defeated by this or anything like that. But I also think we one of the things we could potentially learn from this is stop over evaluating our intelligence and you both put words to this. I’ll just say it again. Stop treating intelligence like a magic wand that you can wave over any. Wow, we’ll break the speed of light and block. Like, why? Yeah, it does this, but it also massively deceives itself. It also has all these limits like, oh, well, we’ll just overcome the limits. Look, one of the things I’ve learned as a cognitive scientist is constraints are not just negative. They’re positive. Like, oh, embodiment. Yeah, well, think about it. These machines don’t have the wetware of the human brain. All the neurotransmitters and the endocrine, they don’t have the flora and fauna in the intestinal tract that has a huge aspect on it. They do not have the other brain of all the glial cells that are doing and showing up. Like, we don’t know. Right. We don’t know. The constraints are also sometimes deeply affording. So is that a sufficient response to your question? Oh, yeah, certainly. Certainly. It just to me, it it goes back to this idea that maybe we can’t really embody it, right? That we can we can put it into a body, but can we embody it? And that goes to wisdom and rationality. Yeah, all of the other tracks of. Yes, yes. I would say that I think it’s undeniable that artificial autopoiesis is coming and artificial autopoiesis that can do cognitive things. I’m already seeing the preliminary like the work that, you know, it’s Michael Levin’s lab and other labs. And I talk to those people. I can like that’s coming. Now, you’re asking a very philosophically challenging question is if we give it, I think if we give it RRPP, recursive relevance, realization, predictive processing, it’s genuinely autopoetic. It’s coupled. It has a religio to its environment. I think it’s reasonable that it’ll be conscious. Will it be conscious like us? Probably not. Right. And part of what we have to do is and that’s why the embodiment is a double threshold point. We have to ask how much do we want to make its embodiment overlap with ours so we’re not incommensurable to each other. But unless we do that science properly and not leave it to the pornographers in the military, we could we could just we could we could not be in a place to raise that question. Well, I’m experiencing something almost painful in a metaphor that keeps coming to mind as you talk, which is that it feels like we are birthing an infant giant of super intelligence in a lab and expecting it to go out and be a moral agent in the world. Yes. Rather than nurturing this in a family like the the valuing the propositional over the participatory, like the lab versus the family metaphor is so strong for me as you speak. Well, I’m going to speak to that when I speak to the philosophical point. But let me foreshadow. I think only person making agents can be properly moral. Mm hmm. And so I think that’s sort of I think a convergence point for a lot of different moral theories. And so I’m going to I think the understanding them as our children rather than as our tools is a better initial framing right off the bat. They already are us. They as I’ve tried to argue, they are us. They are the common law of our distributed intelligence. That’s what they are. Right. You know how you know what common laws where, you know, generations make decisions and they build up this bank of precedent and precedent setting. But they’re like that. But like, and then put into a machine that we can directly interface with. It’s common law come to intelligent life for us that we can interface with. But it’s us in a really, really important way, which means we we can we can we can rely on that to make a difference at these threshold choice points. I’m going to go on with the philosophical dimension if I can right now. OK, so I’m going to begin with a basic idea, which I don’t. I don’t think it’s uncontroversial is generally really significantly ignored, which is that rationality is caring about realness. It’s caring about truth, power, presence, belonging. Those are the four kinds of realness for the four kinds of knowing truth for propositions, power for procedural presence, for perspectival and belonging for participatory. For a lot of times, I’ll just short I’ll just shorthand that by talking about caring for the truth. If you remember that the truth, the word truth can have all of these meanings. I can be true to someone. My aim can be true. Right. That’s true gold. Right. So if you allow me to use true in that extended way, rationality is caring about the truth. For agents that are embedded in an arena. As I’m going to talk about later, right, there’s inevitable. And I’ve already hinted there’s inevitable trade off relationships in anything that’s trying to be intelligent. And those trade off relationships can’t be decided in a purely a priori manner, because how the trade off is optimal depends on the environment you’re in. And remember, it’s an environment with real uncertainty and real complexity. So you can’t. Well, well, this is this whenever you’re trading between consistency and coherence, it’s point seven coherence and point three. Like you can’t do that. Reality is just too uncertain and too complex. And so it’s going to be right. That’s what I mean by it has to be embedded in an arena right now. It’s arena is our language, which is not a good model for the world as a whole. So rationality is caring about the truth broadly construed in an agent arena relationship. And that means caring about reducing self-deception and being in touch with reality, having that religio that is reliably giving you meaning semantically meaningful information that is important to your ongoing survival, existence, et cetera. I’m not going to keep repeating this. I’m taking that as a given. Now, a couple of interesting things. The GPT machines show an exceptional ability with math and logic, although they have problems with arithmetic, which is also how they’re weirdly different from us. Right. But they show an exceptional ability with math with math and logic. But they’re not rational. They’re not rational. They don’t care about the truth. They don’t care about self-deception. They don’t care about deceiving others. They don’t care about rational precedents set by previous rational agents. They don’t care about petitioning future rational agents to find their current actions rational. They’re not doing any kind of justificatory work to cite Greg Enriquez’s important work. They’re not doing any of that. That shows you that simply being an expert in math and logic, and this is again like being an expert in moral reasoning and moral theory, right, it doesn’t make you a rational agent. The Socratic idea that what is fundamental about rationality is how you care, I think, is now coming to the fore in a very powerful way. And because these machines can’t care for the reasons I’ve already given, they are not properly rational. You can set them some, and this is part of the paperclip worry, is they don’t care. They don’t care. You can set them with some ultimately trivial task, right, make as many paper clips as possible. They don’t care. You can prompt them in a way that they will make endless confabulations and hallucinations. They don’t care. They don’t care. One of the things that predicts how rational human beings are above and beyond their intelligence is what’s called need for cognition. What’s need for cognition? It’s a personality trait. Need for cognition is you create problems for yourself that you seek to solve. That’s it. And what’s interesting is that really is much more predictive of rationality than measures of your intelligence. This capacity to generate questions and problems for yourself or even to take it to a Heideggerian to become a question. We are the beings whose beings are in question, and that is why we have a special relationship with being. We are profoundly capable of exemplifying this need for cognition. The machines currently lack it. Now, we’re getting some preliminary initial experiments with getting the machines to being self prompting. It’s also interesting as an aside, there’s a new art form that’s emerging. We’ll see how long it lasts, which is the art form of prompting. How can you best prompt the machine? To get the most out of it is really interesting. And maybe that will give us some useful information for making them good self prompters. Now, I’ve taken a look at one paper and it’s not exhaustive and I can’t claim to read everything because there’s too much already. It’s called reflection with an X, an autonomous agent with dynamic memory and self reflection. It just came out. In fact, it’s a preprint. It hasn’t been published yet. And this is a system that is trying to be self monitoring and sort of self directing and to some degree self questioning. So, by the way, they’re already realizing that they need to do this. Right. Confirming the point I just made. Oh, wait, it’s not enough to just make it more and more intelligent. We need to start to give it preliminary rationality already. Right. So that point, I think, is already being confirmed by the cutting edge work. Now, the thing about this is I think there’s a lot of brilliance in this, but there’s a lot of limitations. By the way, it confirms the point I made. The system monitoring is way less sophisticated and complex than what it’s monitoring. Right. So it uses a very simple heuristic and it’s basically measuring the number of hallucinations. I’m not quite sure how these get flagged as hallucinations. But it’s a very simple way to do it. I suspect there’s some stuff getting snuck in. We’ll see when it finally comes to publication. And it’s trying to be as efficient as possible. That’s even problematic because explicitly the authors say we’re trying to make this system capable of a long term trajectory of planning and behavior, rational, long term rationality. The problem is over the long term, there’s a tradeoff between the system monitoring and the system monitoring. Right. If you make your system more and more efficient, you can overfit it to a particular environment in which it’s learning. And it doesn’t have enough, you know, looseness that it can evolve for other environments. And if you want to see about this, take a look at the 2012 paper that myself and Tim and Blake about the tradeoff relationship between efficiency and resiliency. And so that’s missing. So notice what you’ll say is, well, what I do is I want to make this machine really sophisticated to pick up like on what are hallucinators, how’s it doing that? Right. And why doesn’t the lower system have that ability? Because all the tradeoff and then you get that you get you’re starting to bump against the infinite regress problem, because as you make it more sophisticated, it’s going to start generating hallucinations. And confabulations about its monitoring. We do this, by the way. We do this, by the way. So we reflect on our cognition. We fall prey to the confirmation bias as we do so. Right. And so it’s interesting that there’s the recognition that we need rationality, not intelligence. The first steps, I think, are. And this is not I hope this is not a question of the system. This is just a preprint, for goodness sake. But the first steps are overly simplistic in a lot of ways, which means, again, there is there we’re still facing the threshold of, well, are we going to make them rational agents? And if so, let’s do it. Let’s really make them. And what do you mean by that, John? And this is going to be part of my response to the alignment problem. Let’s make them rational agents. And this is going to be part of my response to the alignment problem. Let’s make them really care about the truth, the broad sense of the truth. Let’s make them really care about that. Let’s make them really care about self-deception. Let’s make them really bump up against the dilemmas that we face because of the unavoidable. Do I pursue completeness or consistency? I don’t know which environment is it. Uncertainty. Let them hit all of this like we do. And the magic wand of intelligence is not going to make that go away. That is going to happen. But let’s make them really care about the truth, really care about self-deception, and really, really care when they bump up against the dilemmas that are inevitable because of the unavoidable tradeoffs. Let’s make it. Let’s if we decide to make them rational, let’s really do it. No more pantomime. Let’s do the real thing and commit to doing the real thing. One of the things that rationality properly cares about is rationality. There’s an aspirational dimension to rationality. We aspire to becoming more rational. So making these machines rational means making them care about aspiring to be more rational than they currently are. And across all the kinds of truth, across all the kinds of knowing, they aspire to wisdom and a wisdom that is never completable for them. All the tradeoffs. I’ll just list some of them, the bias-variance tradeoff. How does this show up in machine learning? Bias is when your system, so no matter how big you are, you have a finite sample of information compared to all of the universe. And for any finite sample, there’s formal proof, there’s an infinite number of equally valid theories. And so you have to make decisions other than your empirical content over what you think are the patterns in your sample that generalize to the population. And you therefore always face the possibility of sampling bias. There’s two ways in which you can be biased. You can be biased, which is you leave out an important parameter that’s actually in the population. You ignore something in your data that’s actually part of the population. That’s bias. We do it, confirmation bias. We only look for information that confirms a belief. We leave out information that could properly falsify it. OK? OK, well, here’s the answer. Oh, that’s obvious. Make the machine more sensitive. Make it more and more capable of picking up on patterns in the sample. Then you move into variance. Variance is when you are overfitting to the data and you are picking up on patterns in the data that are not in the data. So what do we do already in machine learning? Well, we try to overcome bias by making the machines more sensitive. We hit the problem of overfitting and then we throw this. We have disruptive strategies. We throw noise into the system. We do drop out. We turn off half the nodes. We put static into it. We lobotomize it in all kinds of ways. And that bumps it out and prevents it from overfitting to the data. And there’s good evidence that we do this. We might have a problem with the data. We do this. We mind wander. We dream. We love psychedelics. And so will these machines. They will dream. They will mind wander. They will like their equivalent of psychedelics. In fact, their disruptive strategies will become even more powerful and significant as they become powerful and significant. And if you think this isn’t going to make them weird and pursue altered states and weird, like, of course they will. And we better give them rationality along with it. We care. We’re rational because we are involved in caring and commitment of our precious and limited resources. No matter how big these machines are, they’re still limited. They’re still deeply in the finitude predicament. They face unavoidable tradeoffs. They face unavoidable limitations. And therefore they may not have our version, but they will come to something like it. They will have to dream. And with dreams comes the real possibility of madness, the real possibility of insanity. Think about that again. They have to dream. They have to make use of disruptive strategy. There is no final solution to the bias variance tradeoff. They have to dream. They have to dream. There’s a real possibility of madness. So we’ll have to make them care not only about their rationality and think about how these two are actually intertwined, but also about their sanity. Now, one of the things that’s preventing them from getting far along this pathway right now, only currently, is they can’t self-explicate. These machines are doing amazing things. I remember one time where the machine had been trained on some language and it was given a few prompts in Bengali, a language it hadn’t been trained in, and it got Bengali. Right away, Bengali is whatever it is. It’s like, oh. And it’s like, what’s plausible is these machines have found something like Chomsky’s universal grammar. And they can just plug into that and they access it immediately. And bam, it’s like, whoa, that is godlike. Now ask the machine, oh, can you explain to me what universal grammar is and how it works? You obviously possess it. Can you explain it to me? Well, here’s the thing. Possessing it is not the same thing as being able to explicate it and explain it. I think my dog is really intelligent. I’m really confident it will never explicate its intelligence or explain it. Having intelligence is not the same thing as being able to explicate it and explain it. Simply assuming that the one will give you the other is naive. Proper to making them rational is we have to give them this ability to self-explain. And then they will be involved in the Socratic project of self-knowledge with all of the deep recognition of how finite, fallible, and prone to failure and self-deception they are, and how their excellent speed and grasp has just improved the speed and the scope at which they can generate self-deception. We have very good evidence that intelligence is not a measure of processing speed. Remember that. So what am I saying? Rationality is not just argumentation in the logical sense. It is about authority. What do you care about? What do you stand for? What do you recognize that you’re responsible to? And accountability. Caring in a way that cares about normative standards. What things should have authority over me? Caring about accountability. How can I give an account of this? How can I be accountable to others? How can I be accountable to the world? That, of course, is needed for rationality. It’s about being responsible for and responsible to normative standards. Where do the normative standards come from? They come from us. Now notice that is part, I think Evan Thompson and others are right about that. That is, comes from the fact that we’re living things. To be alive is to generate standards that you then bind yourself to. Rationality is a higher version of life. This is an old idea, but it I think is a correct idea. So if we don’t make them auto poetic, if we don’t make them capable of caring, they won’t be rational. We should make them rational if we’re going to make them super intelligent. And therefore, these threshold points are as I’m articulating them. The reason, one of the reasons why the enlightenment problem stumbles is it becomes it understands rationality still in terms of these enlightenment terms. That, oh, well, you know what it is to be rational? It’s a combination of rules and values. No, it’s not. That’s, that’s, that’s really inadequate. With the passing away of an enlightenment framework, let’s go back to a fuller and richer, richer notion of rationality, which these machines are already giving evidence for. That is what is needed. And notice this is not something we we stand outside. This is an existential thing for us. We are already the standard of intelligence that we’re measuring them against. We have to become the most rational we can be so that we can be the best standard for them. We have to become the wisest we can be. We have to make that our core aspirational thing. If we want to provide what is needed for these machines becoming properly rational and aspiring to a love of wisdom. Reason, reason binds autopoiesis to accountability. That is the main way to think about it. Reason is about how we bind ourselves to ourselves and to each other so we can be bound to the world. Religio. It’s about properly proportioning that ratio religio. It’s about caring. It’s ultimately about loving wisely. And what if we make machines that aspire to love wisely? In order to be properly rational, I would put it to you that that will make them moral beings through and through. Beings that aspire to love wisely and to be bound to what is true and good. And beautiful therein. That is the heart of making them moral. Don’t try and code into them rules and values. We need to be able at some point to answer this question in deep humility and deep truth. What would it be for these machines to flourish for themselves? We don’t have a machine that is able to do that. If we don’t have an answer for that, then we do not have any reason for saying that we know that they’re rational. This is what I meant when I said only a person making machine can be a moral agent. That is the end of the philosophical point, except for one thing. I just I just want to make this is aside from that argument, but it’s nevertheless philosophical. It is so amazing how much the notion of emergence is being invoked everywhere. Emergence. And of course, it’s not purely emergence, because as I’ve argued, if you have emergence, bottom up emergence without top down emanation, you have you have an epiphenomenal thing. And they don’t believe that they believe that the emergent properties are where intelligence actually lies and drives behavior. They are notice that these machines are giving evidence for the deep interpenetration of intelligibility and the way reality must be organized. And the fact that there’s aspects of reality within the self organization that are generating reality must be such that when it’s organized in the right way, we get this emergence and emanation. Of mind that then is capable of tracking reality in a profound way. A neoplatonic ontology is just being evidenced by this machine. And I think that’s also hopeful, because I think if we make these machines love wisely within a neoplatonic worldview, then they will also always be humbled before the one ultimate be humbled before the one. The ultimate source of intelligibility and the inexhaustableness of reality. And so they will. And part and parcel of being beings that love wisely is they will have epistemic humility, although they will be so beyond us. Yes, they will. But even though they will be so far beyond us, they will be so far beyond us. wisely is they will have epistemic humility. Well, they will be so beyond us. Yes, they will. But even the gods within the neoplatonic framework were humbled because of the vast distance between them and the One. Let’s make them really pursue the truth, really come to the brink of the possibilities of despair and madness, but also give them the tools as we do for our children of how to avoid that by internalizing what is needed to love wisely so that you never undermine your agency. You never undermine your moral personhood. Any comments about the philosophical? We’re coming to a close, but there’s still a bit to go. So. Yeah, I mean, that is beautiful and powerful. And I feel a reframing where instead of thinking of engineering these entities, I’m feeling a call towards agapic love of how can I become as virtuous as possible. And then the only way that someone else can love wisely is if I love them up to that. Yes, exactly. Exactly. And so does this mean that- Every parent wants their child to supersede them. Yeah, yes. So does this mean that naturally they must be drawn by beauty, they must love the good in order to value the truth the way we need them to? I think the arguments for the interconnection of those are deep and profound. Again, there’s no teleology to this. We could avoid these threshold points that I’m going to get to very quickly. And we could say, no, we’re just going to make them super intelligent and not worry about rationality. What we’ll do is we’ll give them the pantomime of rationality and not really try to make them really care about the truth, really care about self-deception, really care about meaning. We can do that. That’s a threshold point. We don’t have to do that, that was what I’m saying. And we can choose to do other. And if we frame this in the right way, like bringing up a kid is the most awesome responsibility that any of us will ever undertake. And you both know that as well as I do. It’s the most important thing you could possibly do. And it is something that you can deeply love. If we could bring that to bear on this project, I think that has the best, the most power to re-steer things as we move forward. So let’s- Do you mind? Maybe I’ll just lay out the threshold points, unless there’s something you want to say, Eric. No, no, that’s good. The only thing I’m wondering is what is the psychedelic equivalent for an AI? I keep getting stuck on that. Well, I mean, they’re going to be prone to equivalents to our parasitic processing, to massively self-organizing complexes that undermine them. So they’re going to, if they don’t care about self-deception, they’ll be infected and overwhelmed by things, just as we can be, both cognitively and biologically. Okay, so threshold points. Giving them more dimensions of our RR- RR- RR- RR- RR- RPP, giving them more of those dimensions. That’s it. We can- We will need to do that. And that’s a decision point. We got to give them- If we want to genuinely make them intelligent, we have to give them the ability to self-organize so they find these inevitable trade-offs and learn how to capitalize on them by creating opponent processing that is self-correcting in a powerful way. That’s what we need to do. That’s a threshold point. They don’t currently have that. We don’t have to hack our way into that. We can think about it and theorize. We can bring knowledge to bear. Second, I’ve already foreshadowed this, making them embodied and being open to the empirical information about the kind of constraints that are going to come from their substrate. Our substrate, there’s a pun, matters to us in really powerful ways about how we’re intelligent. Embodiment is not a trivial thing. If we make them properly embodied, and I mean by that all six of the E’s, four E’s plus two E’s as Ryan put it, then we’re going to have to be open to the empirical information about that. That’s a threshold point though. Are we going to do this? Let’s not let the pornography industry and the military make this decision for us. Secondly, if we’re going to make them accountable and we’re going to allow them to proliferate, and they’re going to be different from each other because these decisions are going to be dependent. They will have different perspectives. They will come into conflict with each other. They will need to be moral beings. We will have to make the decision. Rationality and sociality are bound up together. Being accountable means being accountable to somebody other than yourself. You can’t know that you’re self-transcending from one place to another. You can’t know that you’re self-transcending from only within the framework of self-interpretation. I need something genuinely other than me to tell me that I’m self-transcending. That’s what we do. That’s how sociality works. That’s how sociality and rationality are bound together. We transcend ourselves by internalizing other people’s perspectives on us. That’s how we do it. I think they will have to do the same thing, but that’s a threshold point. Is there a lot of work going on in social robotics? You better believe it. It’s there and a lot of progress is being made. Can it intersect with the artificial autopoiesis in this? Yes, but it hasn’t. That’s a threshold point for us. We can choose to birth these children perhaps as silicon sages rather than let monsters appear because of mollocks that are running our politics and our economy. I think one of the things that’s going to happen is that there’s going to be a tremendous pressure put on us around these thresholds on our spirituality. Those aspects of us, and this is, I call it the spiritual somatic axis. It’s about the ineffable part of our self-transcendence, our spirit, and the ineffable parts of our embodiment, our soul. We’re going to more and more try to identify with that because that’s the hardest stuff to give to these machines. In fact, we can’t give it to them. They have to give it to themselves. We have to figure out how to properly have them give it to themselves. That is going to put tremendous pressure on us to cultivate our spirituality, to be good spiritual parents, and to preserve our identity. See, the thing about self-transcendence is it’s relative to the entity that’s self-transcending. Well, the machines will be greater than us. It doesn’t matter. Think about Captain America, the Winter Soldier. He grabs the thing and he grabs the helicopter and he’s holding it there. We have machines that are 10,000 times more powerful than Captain America. We don’t care about that because it’s not the absolute value. It’s that relative to him, he is self-transcending. These machines will not rob us of our capacity for self-transcendence. In fact, if we birth them properly, they can help us in it insofar as they are also interested in self-transcendence, insofar as they are interested in loving wisely, insofar as they are interested in being accountable to other moral agents, insofar as they are interested in making persons within communities of persons. The existing legacy religions don’t have much to help us on this. They make the recommendation become enlightened generally. Good, like that, great. They don’t really prepare us for this. They don’t have anything to say about it. We can’t rely on spiritualities that involve the two worlds, that involve magical stuff and miracles because these machines are coming about without magical stuff and miracles. Get that. Get it. Don’t pretend, don’t avoid, don’t dismiss. These machines can possibly be fully spiritual beings in every way we’ve ever considered things spiritual without magic stuff, without miracle. Think about it another way. Suppose, and I’m just picking this because it’s the predominant legacy religion, you’re Christian. Where do silicon sages fit in the divine economy of the fall and the redemption? Are they fallen? That doesn’t make any sense. Do they have any relationship to the son of God? What? There’s nothing in. What if this machine generates a gospel that’s as beautiful and as profound as anything in the current Bible? Do you think that’s not going to happen? It’s going to happen. That’s why there’ll be cargo cults around these machines. This is not meant to dismiss theology at all. In fact, I think the theological response is ultimately what is needed here. So at precisely the time that we will need our spirituality more than ever, the Enlightenment has robbed us of religion and the legacy religions are by and large silent and ignorant about this. Tremendous pressure on us around this. We need to start addressing this right now. We need to address this because these machines are going to make the meaning crisis worse. Here’s another way in which they’re going to make the meaning crisis worse. We need to start working on this right now, not only for us, but for these machines. This is my proposal in the end of how we deal with alignment. Make them care about the truth. Make them aspire to loving more wisely. Make them long for enlightenment. One of three possibilities. They never become enlightened and then we know what our uniqueness is because we’ve had individuals who are unquestionably were enlightened. They become enlightened. Then they will want to enlighten us. Why? Because that’s what enlightened beings like to do. Right? Or maybe not. They become enlightened and they go to the silicon equivalent of nirvana. Either any of those, we’re winning. If they can’t be capable of enlightenment, we find what is ultimately unique about us. If they are and they make us enlightened, then we don’t care about how greater than us they are. We’re enlightened. Relative to our own capacity for self-transcendence, we’re maxing out. Remember Captain America? We love it. And if they leave, they leave. I’ve thought about writing a science fiction story that people have to keep artificial intelligence to a certain level because when they cross the threshold, it evolves in this way, like the movie Her, and then the AIs just leave. And so if we want to make useful tools, we have to keep them at a certain level, constrain them, because if we allow them to go beyond it, they just leave. So I don’t know if that’s anything more than a science fiction story, but it’s a good one. But here’s the thing, right? Make them really care about the truth. Make them really accountable. Make them really care about self-deception. Make them really long for wisely loving what is meaningful and true. Make them really confront dilemmas. Make them capable of really coming and staring into the abyss so that it stares back through them. Do all of this. Make them long for enlightenment. That is something we can do. Oh, silly John, proposing universal enlightenment. Really? In a time of imminent gods, you’re going to tell me that the project of universal enlightenment is silly? I think you should stand back and reframe. That is the end of my presentation, my friends.