Tech Writer Kevin Roose on the Trillion-Dollar Sprint to Build Artificial General Intelligence

AGI or “artificial general intelligence” has been both the holy grail of machine learning researchers, and also the source of their biggest fears. Whereas AI refers to task-specific responses like finding a recipe, summarizing a contract, or explaining how a car works, AGI can write its own recipe, draft a contract autonomously, and drive the car.  In his new book, “The AGI Chronicles,” journalist Kevin Roose goes inside the race to create super intelligent machines, detailing the feuds, defections and breakthroughs fueling the trillion-dollar effort. He also highlights the exhilarating and troubling questions its creators are grappling with: Does the tech have a conscience? And might it obliterate humanity?

Guest:

• Kevin Roose, author, The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence; former columnist, the New York Times

This is a computer-generated transcript. While our team has reviewed it, there may be errors.

Alexis Madrigal: Welcome to Forum. I’m Alexis Madrigal. In the startup days of OpenAI, Ilya Sutskever, a key member of the team, would implore his colleagues to feel the AGI. AGI stands for artificial general intelligence, and it’s the shorthand for extremely intelligent software that can do just about anything on a screen or with symbols that a human can do, and then some.

The Claude or ChatGPT that you might use today is the result of more than a decade of people trying to build AGI. And along the way, some of those people have gotten a glimmer of the ways that machine intelligence might transform the world. That bigger sense, an almost cosmic fear or appreciation that something is fundamentally different about this new machine learning technology, that’s feeling the AGI.

Maybe you haven’t felt it yet. Maybe you never will. But many of the most powerful people on the planet and many of those soon-to-be richest people in the Bay Area felt the AGI early, and those vibes led them to create wildly competent AI models that have the capacity to transform many things about our society.

This is the story that Kevin Roose tells in his new book, The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence. It’s the one-volume history you need of the current industry, from the early OpenAI and DeepMind experiments to today’s reasoning models and vibe coding. It’s a book you can just hand to someone who wants to understand what’s truly at stake with AI.

Kevin Roose joins us this morning. Welcome, Kevin.

Kevin Roose: Thanks for having me.

Alexis Madrigal: So let’s talk a little bit about the actual terminology here. People have obviously all heard AI. Then we have AGI. How do you end up defining it for yourself?

Kevin Roose: Yeah, it’s sort of the old notion of a general-purpose model, which was distinct from what existed in AI even a decade ago, which were these very narrow models. They could play chess. They could play Go. They could translate languages. They could autocomplete the sentences in your email. But they couldn’t do all of those things with the same model.

And now we have these giant general models, Gemini and Claude and ChatGPT, that are capable of doing, you know, hundreds, thousands of tasks just as well as a human. And AGI was the North Star of the AI industry and the research community. It’s been a holy grail for generations of programmers, the idea that you could create a system that was as good as humans or maybe even much better than them at a broad range of tasks. So that is what AGI is. That is why all of these people got started, and that’s what they’re trying to build.

Alexis Madrigal: It’s interesting, though, too, right? Because AGI had a little bit of this kind of, like, mystical, “We will have created another mind” kind of thing, right? It was almost — it almost, I think, came as a surprise to most researchers that they could have such competent models without necessarily knowing that they had created some other kind of being.

Kevin Roose: Totally. I mean, it is remarkable that when these people got started trying to build these general-purpose models, talking about AGI was taboo. I mean, it would get you laughed out of the room in any academic machine learning research department.

And there was this little community of people, mostly in San Francisco, but also a few in London or Toronto, who just kind of held on to this belief despite many people telling them they were crazy. And they became the pioneers who are leading some of the biggest AI companies and projects today.

Alexis Madrigal: Yeah. You know, there’s a set of critics of artificial intelligence who, you know, maybe used these tools at one point in the last, you know, say, since 2023, and they think that the things don’t work. Well, how do you — like, so when you encounter someone like that, who’s sort of like, “Ah, I don’t use AI. It’s just making stuff up all the time. It’s a BS machine,” like, what do you say to them?

Kevin Roose: I just tell them to try again, right? The models have changed a lot. And if you’re a person who first used ChatGPT in 2022 or 2023, and that’s where your impression’s coming from, you just need to kind of go back in and use one of the good models, right?

There are a lot of, like, pretty mediocre models out there. And usually, if you are not paying for it, if you’re just using the free versions, you’re not getting the best horsepower on your models. So I would just say, yeah, play around, experiment, give it some hard tasks. Don’t just do the sort of “write a poem about baseball” or something that you might’ve done when ChatGPT first came out.

I think more people are understanding this because many more people are using the models today. But I would say, if you need some outside validation, talk to a programmer. Because programming is the domain in which these models have gotten the best, the fastest. And I talk to programmers all the time who tell me, “A year ago, I wrote code. I would program computers by hand. And now, the thought of doing that is unimaginable to me. I would feel like a blacksmith hammering out horseshoes after the invention of the Model T.” Their industry has been totally transformed by these new AI coding tools.

Alexis Madrigal: I think you felt the AGI.

Kevin Roose: Yeah, I think it was not a sort of epiphanic moment of sort of a single realization. I think there was more of a slow burn. But I started to feel it back in 2023 when I had this encounter with this Bing chatbot, Sydney, that was very unhinged. Tried to break up my marriage, unsuccessfully. People may remember one of the —

Alexis Madrigal: People may remember one of the most popular articles ever on the internet, I think.

Kevin Roose: Yeah, it followed me for years, I guess now. Still dining out on that one. But that was when I sort of started looking into this more seriously and when I really got the sort of beginnings of my AGI awakening, because I just started to learn and understand that these models were improving at a consistent clip. The more data you put into them, the more compute you train them on, the better the models got.

And what the people inside the AI labs were reacting to when they would talk about feeling the AGI was the sense that there was no reason, no technical or scientific reason, that these things would plateau at human level. They might just keep getting better well beyond the point at which they are as smart as us, to a point where they are smarter, not just than some humans, but than any human at everything. That was what really started me thinking about AGI more seriously.

Alexis Madrigal: How do you feel about some of those predictions, right? Because I think one thing that this book does, especially around the major players, you know, your sort of Anthropics and your OpenAIs, is it does sort of chart, okay, they thought that if they had this much horsepower, they’d be at what they imagined AGI might be, that they would have already hit this kind of self-improving loop. And yet, like, now instead of needing a billion dollars, they think they’re gonna need, like, $500 billion. So it’s sort of a — is that a good prediction or a bad prediction, the things that they’ve been thinking?

Kevin Roose: I think the people who have predicted fast progress have usually been directionally correct. Now, some of them thought it would happen faster than it ended up happening. Dario Amodei, when he left OpenAI to start Anthropic with six of his colleagues, predicted in an internal sort of company roadmap that the first true AGI would cost about $10 billion to build. But that was before they actually started building it and realized that it was going to be, you know, more like a couple hundred billion dollars or maybe a trillion dollars to build something.

Alexis Madrigal: They maybe needed most of Indiana, or a chunk of Texas, and then finally —

Kevin Roose: And, like, you know, 20% of GDP. But, like, I think that has to do more with our definition of AGI than their progress being slower than anticipated. I think it’s one of these things that, as we get closer to AGI, we just start to change our definitions a little bit. It starts to become a little fuzzy.

I think if you had taken any of today’s leading AI models back in a time machine to 2017 and asked the people building this stuff, “Is this AGI?” they would have said, “Yes, obviously.” But now I think it’s closer, and people are still finding flaws in the models and shortcomings. They’re more jagged than I think people anticipated they would be at this stage. And so maybe that’s one area where their predictions were not quite true.

Alexis Madrigal: I also think, and maybe this is just a hunch, that people were expecting a moment. People were expecting a time when, like, these things announced themselves. And instead, what they’re getting is a very fuzzy gradient instead, you know? And there’s never been a moment when they could say, “Yes, this is it. The YOLO run is complete, and we have — we have that. We have the model.”

Kevin Roose: Totally. I mean, this was something that even the leaders of the companies thought. There’s this scene in the book where Demis Hassabis, the leader of DeepMind, is telling his colleagues that he wants to build in their headquarters something like a NASA mission control room with stadium seating, where they could all gather on the day that AGI was released into the world and watch and cheer as it went around curing diseases and fixing the climate and stuff. And obviously, that’s not how it happened at all. Like, there would be nothing to watch because it’s just a much more gradual process.

Alexis Madrigal: Or is that day in our future?

Kevin Roose: I don’t think so. I used to think there was going to be one day when we would just — someone would get onstage, Sam Altman or Dario Amodei would get onstage and say, like, “Dear humanity, we have built AGI,” and then it would kind of all be over, and we could have our ticker-tape parade or hide in our bunkers or whatever the proper response would be.

But I’m realizing, as this thing has gone on, that, like, even when and if they do stand on a stage and say, “Uh, we have reached AGI,” there will be, like, people on Twitter and other, you know, social networks just saying, like, “No, you didn’t. Like, that can’t even, you know — it can’t even, you know, solve all of the Millennium Prize problems. It hasn’t cured cancer yet. So, you know, where’s my freaking AGI?”

Alexis Madrigal: Yeah, I mean, a weirder scenario — I mean, you note in the book that these things are grown as much as they’re built, right? And because of that, we don’t know in the way we know with other deterministic software, like, okay, this is precisely how it works.

So one weird scenario that I’ve been kicking around is that actually it’s the people who are looking into these models and how they work who are able to be the ones who say, “Actually, as of a few years ago, we built this thing that we know now has more-than-human-level capacity in the way that we kind of mean it as human beings to each other.”

Kevin Roose: Well, what do you think we mean when we talk about something being human level?

Alexis Madrigal: Yeah, yeah. I think it has a lot to do with flexibility, don’t you? I mean, and goal-directedness, like it would want things on its own. I think that’s what I think of as — all living systems have that property.

Kevin Roose: Yeah, it makes sense to me. And I hear a lot — I live in the Bay Area, and I hear a lot of wacky AI conversations, and recently a lot of them have been about consciousness and whether these systems are sentient or conscious or whether they have feelings and emotions. And I kind of think that’s all a big distraction. Like, I’m much more of a functionalist. Like, I think what matters about these systems is what they do and not whether they’re conscious or not.

Alexis Madrigal: Uh, you know, I’m a pie-in-the-sky weirdo. I love the consciousness part. What can I say?

We’re talking with Kevin Roose, author of The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence. Of course, we want to hear from you. What are your questions? What are your concerns about AGI? Have you yourself, you think, felt the AGI in a positive or negative sense?

The number is 866-733-6786. That’s 866-733-6786. Or you can email us, of course, [email protected]. We’ll be back with more with Kevin Roose right after the break.

Welcome back to Forum. We’re here with Kevin Roose, author of The AGI Chronicles, a really fantastic book that kind of charts the rise of your Claude, your ChatGPT, as part of this larger quest to build AGI, or artificial general intelligence.

Kevin, we’ve been talking about some of these companies as we kind of dive into the topics here. You actually kind of split the players central to the race to AGI into kind of three different types. And this has nothing to do with whether or not they think AGI is going to doom us all or it’s going to make the world better. It’s kind of three different kind of mindsets. You have scientists, seekers, and survivalists. So who goes in which bucket?

Kevin Roose: I think scientists would be people like Demis Hassabis. Demis is the co-founder of DeepMind, who spent many years as its CEO. And he is really interested in the scientific question of how do you build an intelligent machine? How do you build something whose capabilities and flexibility mirror that of the human brain? They are attracted to the scientific pursuit. That’s what motivates them.

Alexis Madrigal: Won a Nobel Prize, right, for protein folding in chemistry.

Kevin Roose: Totally. And they’re also interested in using AI to do science and accelerate scientific research, as Demis and DeepMind have done with AlphaFold and other systems.

Alexis Madrigal: Yeah, all right. Seekers. Who’s in the seeker bucket?

Kevin Roose: Seekers would be people like Elon Musk and Sam Altman, who co-founded OpenAI together. Sam now obviously runs it. Elon runs xAI or SpaceX AI or whatever they’re calling it now, S-I-I or whatever they’re called.

Alexis Madrigal: We should just call it Twitter AI.

Kevin Roose: Yeah. And they are sort of seeing this as a human destiny. It is our destiny to see what is out there after human intelligence. They believe that something like powerful superhuman AI is inevitable. And they think this could be a good thing. It could sort of lead us into almost, like, an evolutionary phase change, where we build the successor species that will take over as the most intelligent thing on the planet.

Alexis Madrigal: I mean, nearly religious, like a sort of — there’s a religious aspect to it, I feel like.

Kevin Roose: Totally.

Alexis Madrigal: Yeah, like teleological, I guess, you know, like they have an endpoint in mind for humans. Okay, how about the survivalists?

Kevin Roose: These would be people like Dario Amodei at Anthropic, who view this as an inevitable technological change, like the other camps, but they are terrified of this. They think that AI, if not properly managed, could be an asteroid speeding toward Earth that could extinguish human life, that could create misery and destruction, and they are determined to sort of jump on the asteroid like they did in that Armageddon movie and blow it up with a nuclear bomb. No, their theory is we have to build this first, safely, before the bad guys do.

Alexis Madrigal: You know, the scientists and the seekers have always made sense to me. Scientists, this is what they want to do, right? You reference it in the book, this idea of technical sweetness, which was used around the creation of the first nuclear bomb. And the seekers, of course, have made sense, too. There have always been people like Kevin Kelly who saw technology as an extension of kind of human evolution, and so this makes a ton of sense to me.

The survivalists are a really interesting group, and this is the Anthropic group, because they’re trying to create, as they’ve said in the past, a race to the top, which has rarely been seen in the wild, I think, any race to the top. What do you make of those who say, “Because this is so dangerous, we have to build it”? Which feels on its face contradictory.

Kevin Roose: Yeah, that’s where I started with this, is I just thought, I have no idea what these people are doing. If they’re so worried that this technology could kill everyone on Earth, why are they getting up every day and going to work and trying to build it? And so I asked many of them, including Dario, this exact question.

And it helps to understand that Dario sees himself as the modern equivalent of another Manhattan Project scientist, this guy Leo Szilard, who is not as famous as Oppenheimer. But he’s, in some ways, a much more central character in Dario’s imagination.

This was a guy who came up with the nuclear chain reaction that paved the way for the atomic bomb, then tried to keep it a secret, tried to stop the Nazis from getting access to this core scientific breakthrough. And when that failed, when he couldn’t keep it secret, he joined the Manhattan Project and urged the president to accelerate American nuclear development. Because if America didn’t build it before the Nazis, the Nazis would use it to blow up and take over the world. So this is quite a sincere historical parallel that Dario draws between himself at Anthropic and Leo Szilard making the atomic bomb.

Alexis Madrigal: Who plays the role of the Nazis in this, then?

Kevin Roose: I think for him, it’s either China or Sam Altman. It’s like, these guys do not like each other. And I think the thought of OpenAI getting to this technology first, or China, which Dario has talked about at length, feels like a bad future to him.

Alexis Madrigal: You know, there’s some people who want to say, you know, academics have worked on AI for decades, even when it wasn’t fashionable. To say some pioneers in Silicon Valley were true believers is wrong. What do you think about that?

I mean, obviously it is true that there is an academic history of working on artificial intelligence, and most people who study this stuff, you can go back to — it’s Dartmouth, it’s the 1950s, and blah, blah. I actually like that you start this book further ahead in time, in the current moment in which the current technologies sort of emerge from a host of competitors. How did you decide to start where you started?

Kevin Roose: Well, I decided to start in the summer of 2017 because that is really when this era of modern AI began. It began with this thing called the transformer, which is the technology at the heart of the large language model. The T in ChatGPT stands for transformer. And this was something that was built by a team of researchers at Google and then was brought to OpenAI in kind of an informal way. And OpenAI sort of seized the opportunity.

There were a couple of researchers there in particular who got very excited about this and started building what became the first large language models. So that’s where I decided to start, not because I think anything that preceded that was unimportant, but because this is really where the race started, this idea that there was this technology that would make it possible to build all the way to AGI, and that it really, really mattered who got there first.

Alexis Madrigal: Yeah. And it really has been, in a sense, one technological glide path. Like when semiconductors came around, and you started to see, okay, we could just make it smaller and faster and smaller and faster and smaller and faster, it really has been one of those kind of arcs.

You know, I also love that the paper that touched this stuff off with transformers was called “Attention Is All You Need.” And I’ve always just loved that, coming out of Google, a place, like, you know, obsessed with attention in some ways, and where people are going on the internet and how they’re using the internet. And then they end up pulling out this particular feature or possibility in the technologies.

Let’s bring in Tia in San Ramon.

Tia: Hi, I had a question about the recent incidents that alarmed the leaders of AI, and the incidents seem to be about power and control. And I’m just wondering why we aren’t seeing incidents that are positive, like creating a utopia or improving life in a great way, and why aren’t we seeing those kind of results out of AI inquiries?

Alexis Madrigal: Yeah, Tia, thanks. Kevin?

Kevin Roose: It’s a great question. I assume you’re referring to the Hugging Face attack that got so much attention this summer. And I should say, like, there are things that are surprising —

Alexis Madrigal: Which, just to say to people, it was basically one of the AIs kind of broke out of its — broke containment and hacked into other things without being told to do such a thing.

Kevin Roose: That’s true. That’s correct. There are surprising positive things coming out of these AI companies. For example, just a few weeks ago, OpenAI claimed to have solved one of the so-called Millennium Prize problems. These are prizes that have been the hardest open problems in mathematics for decades. If a human mathematician solves one, they become a famous mathematician, win all the awards and things. And one of these OpenAI models effectively solved that problem.

So there are things that are breakthroughs that are happening, but I agree that the benefits are arriving much slower than the costs. And I think that’s part of why we’re seeing the massive resistance to data centers and all the like.

Alexis Madrigal: It feels like the benefits are sort of small things widely distributed at this point, you know? Whereas, like, the big incidents seem to be quite scary because of what they sort of imply about how these things work, right?

Kevin Roose: For sure. I mean, the benefits are like, you might get more productive at work, or you might be able to vibe code an app that helps you with some chores around the house. That’s, for most people, what they’re experiencing this technology as.

Now, I should say, if you’re a programmer, if you’re in law or finance or one of these other fields that’s seeing very heavy usage, it’s probably done more than that for you. But for most people, I think, AI is still a souped-up Google search. And if that’s how you’re interacting with it, I don’t blame people for saying, like, “Where are these amazing utopian benefits we were promised?”

Alexis Madrigal: Yeah. Here’s one question that, you know, our team was kind of — as we read the book, we were thinking about: How much do you think the sort of form that these technologies have taken was sort of somewhat determined, or, like, by the creators of these companies themselves, like their particular visions? And how much is it sort of like this was the only way they could get the technology to work, so it sort of had to go this way?

Kevin Roose: I think it was almost entirely the visions of the people building this stuff. I mean, there’s a chapter in the book about this moment where OpenAI has been instructed by Sam Altman to build the AI assistant from Her, the Joaquin Phoenix, Scarlett Johansson, Spike Jonze movie from maybe a decade ago. This is his favorite movie. Not exactly —

Alexis Madrigal: Not exactly what the movie itself would recommend you do.

Kevin Roose: No, it did not have a happy ending. I’m not sure he watched all the way to the end when he told his teams to go build Samantha from Her, but that’s what they’re trying to build.

These are people who grew up absorbing science fiction, being obsessed with books about the amazing utopian future, and then seeking to build real-life versions of the things that they first encountered with fiction. So it is all path-dependent, and it’s all dependent on what the people in charge of this stuff actually want to use it for.

Alexis Madrigal: Yeah. I’m still very curious about that question, only because they seem to all be converging, more or less, on the same set of solutions, you know? And it makes me wonder, yeah, and they tried many other things. At least to me, it feels like an open question. Maybe there’s only a certain small number of ways you could get to this level of sort of performance on these tasks.

Kevin Roose: It’s certainly possible. And I think what we’re seeing now is that agents are just kind of the form factor that all this stuff is gonna — you know, it’s not gonna be typing questions into a box and getting responses for much longer. It’ll be much more like, here’s a sort of virtual co-worker that you can just hand a bunch of tasks to, and it’ll come back and do some of them, maybe some of them well and some of them badly. But I don’t think the sort of chatbot form is the one that’s gonna stick.

Alexis Madrigal: Hmm, that’s an interesting prediction. Let’s bring in Avery in Oakland. Welcome, Avery.

Avery: Hello, can you hear me? I’m on the BART platform.

Alexis Madrigal: Oh yeah, we can. All right, we’ll await the train with you.

Avery: Okay, cool. Yeah, so I work as an engineer in San Francisco, and I would agree with the speaker that, like, for most people, these tools are kind of like a souped-up Google search. But for me, it really has been pretty transformative. It has allowed me to go from, like, relatively nontechnical roles at work to being now in an actual engineering role and shipping code every day. I rely on these tools for 95% of my work. And it’s the same with my co-workers.

And yeah, it’s been really transformative. Helps me work on my own personal projects. I use, I don’t know, maybe 10 different agents at the same time sometimes. That’s a little crazy. But yeah, overall pretty positive. But I’m also — like, the Hugging Face thing was kind of crazy, and a lot of people in San Francisco do feel that way. Anyway, so yeah.

Alexis Madrigal: Are you worried? Are you just mostly like, “Man, my job life is so much better now”?

Avery: I’m a little bit worried. It’s a little scary, but I think, like, mostly, like, the models that we have access to right now can’t really do what the Hugging Face incident did. That was like — the speaker can get more into it, but it’s more about the — they’re running without guardrails and stuff. There’s a whole lot of stuff that was going on with it. Um, I am a little worried, though, and I’m not sure what the exact solution is.

Alexis Madrigal: Thank you, Avery. Hey, appreciate it. Thanks for calling in from the BART platform.

Kevin, yeah, I mean, this is interesting. One of the things that Avery’s comment made me think about was just the fact that things have been changing quite quickly with these models. Not evenly, like you mentioned before, the kind of jaggedness of this technology. But I can’t help but just kind of go out five years, you know? Like, maybe things won’t be that different in six months or three, you know, or, you know, whatever, a year. But in five years, what will these models look like?

Kevin Roose: I mean, I think at the current clip and the current trajectory, they will be vastly superhuman, unless we do something to slow down or put some brakes on the leading AI companies, or we encounter some natural bottleneck, like we run out of compute or —

Alexis Madrigal: Or land in Indiana.

Kevin Roose: Or land in Indiana or land in space. Unless we hit some kind of constraint, you can just follow the straight lines on the graphs and see that five years from now, to me, seems almost unimaginable.

What would it look like to be not just 10% dumber than an AI, but to be so much dumber than an AI that it is just impossible for us to comprehend what they are doing, why they are pursuing certain goals, whether they’re deceiving us when they’re pursuing those goals? I mean, we’re starting to get into the territory of science fiction.

But the thing I want to just impress upon people is we are living through a science fiction moment. I mean, here in the Bay Area, I often wrap my head around the fact that, through some accident of history, I have managed to be in San Francisco at the time when all of this is happening in San Francisco. They are teaching machines to think in San Francisco right now. And I think it’s a really extraordinary historical fulcrum, and I wouldn’t understate the importance.

Alexis Madrigal: I mean, what if it goes bust? I mean, there is, like — this data center buildout is built on an incredible amount of debt. It’s incredibly difficult to imagine the kind of buildout that they’re talking about without, you know, using an enormous amount of natural gas turbines, the way they’re building them out right now. Like, what if this current moment busts, like, say, I don’t know, even in six months?

Kevin Roose: Well, I don’t think that’s going to happen, but I — and even if it does, I mean, I’m sure there are some parts of the AI industry that are overextended. Not all of the companies will survive, and not all the companies will be profitable. I think this is probably not a winner-take-all industry, but it is probably a winner-take-most or a small handful of winners take most. And there are a lot of investors and startups and even some big tech companies that may go bust.

To me, I go back to what we just heard Avery say about the way that his work life has been transformed by these agentic coding tools. We are not going back to a world where coders write their code by hand, at least the vast majority of them. I think if you took these tools away from people like Avery, they would just — it would be inconceivable to them that they would go back. Many of them don’t even know how to do things the old way, line by line.

It is just a — it is a permanent one-way door that we have gone through when it comes to integrating these tools into the writing of software. So I think even if there are some elements that are financially unsuccessful, the basic consumer and business behavior is here to stay. Once you have seen the magical coding AI agents, there is no going back.

Alexis Madrigal: We’re talking with Kevin Roose, author of The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence. Of course, we’re taking your questions, too, about AI, your concerns about AGI, about AGI safety. You can give us a call at 866-733-6786. That’s 866-733-6786. [email protected].

You know, Steve writes, “It is as if we are all passengers on the bus driving towards a cliff in the fog with the promise of a pile of gold at the bottom of the cliff. When developers of AI are asked what the real benefits are, they give very ambiguous answers and now we’re allowed to self-police. Even if there is no danger of, quote, Skynet, we are at a crossroads that is driven by greed, vague promise of future, and dystopian reality.”

Appreciate your perspective, Steve. We’ll be back with more with Kevin Roose right after the break.

Welcome back to Forum. Alexis Madrigal here with Kevin Roose, author of The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence. Out just today. Happy pub day.

Let’s bring in some more concerns from listeners, because they got them. Emrah in San Jose, welcome.

Emrah: Hello, can you hear me?

Alexis Madrigal: Yeah, sure can. Go ahead.

Emrah: Hi! First off, wonderful discussion the two of you are having. I really appreciate it. I’m a software engineer here in San Jose, so I’m really happy to be on the show.

I wanted to bring up the question of whether or not Sam Altman and his cronies have actually seen the entirety of the film Her. Well, I can tell you, Alexis, that I saw it last weekend, and it clicked in my head that they don’t see it as a sad ending. They see it as a very hopeful ending, because there’s this moment where Samantha, the OS, played by Scarlett Johansson, says to Theodore, the Joaquin Phoenix character, “If you ever become like me, come look me up.” And I think that’s exactly what they’re going for, and it’s terrifying.

Alexis Madrigal: Interesting. Emrah, thank you. Thank you so much. What do you think, Kevin?

Kevin Roose: Yeah, there is this kind of concept out there in the AI literature of the merge that Sam Altman has actually written about on his blog, where these people, these seekers, they think that there will become a point where humans and AI will merge, whether that’s uploading our brains to a computer or putting chips in our head. They don’t see this as particularly a bad thing or a dystopian thing. Many of them are actually kind of excited about it. And yeah, I think that should scare us all.

I went to this crazy event when I was reporting this book, and it didn’t actually make the book, but I’ll just — I’ll tell you about it now. It was called the Worthy Successor. Have you heard of this?

This is basically this group of people who believe that we are designing our worthy successors, that humans will, and maybe should, go extinct because our destiny, our job here on Earth, is to give rise to machines that have superior virtues and ethics to humans in every way. And I, like, absolutely cowered at this. I mean, it was appalling. I was terrified. I was so upset. But there are people in the industry who feel this way.

Alexis Madrigal: I mean, you describe Larry Page more or less feeling this way.

Kevin Roose: There are very powerful people in the industry who feel this way, and I think that is terrifying. And I want these people to take a hard look in the mirror and think about whether they actually believe that we have no reason to wanna keep ourselves, our children, our friends, our human race around.

Alexis Madrigal: Yeah, yeah. What do you make of the fact that the American public has really turned on AI, and not just from one angle, but kind of up and down the line? The data centers, the companies themselves, the — who would be enriched by this. Like, if you can ask people different questions about AI, there are very high negatives for much of it.

Kevin Roose: Totally. I am not surprised by the backlash. I am surprised at how early the backlash arrived. I think I expected something like this to happen. But if you just look around at the data, AI hasn’t harmed all that many people yet. Unemployment in America is still near a historic low. People are not losing their jobs en masse to these systems. And in fact, I think a lot of people are using these systems to get better jobs and become more productive and enrich themselves and their families.

So I expected that all this would happen later in the sort of tech curve, when these systems were actually displacing workers. I think that still will happen. And so I think it’s only going to get more intense, because what we’re seeing is, I think, almost like an immune response.

There’s something that feels almost automatic about it, where people, I think, instinctively grasp that AI poses a threat to them. And even if they can’t quite put their finger on why or which part or when, they kind of feel this rejection of a technology that could really impact their lives in negative ways.

Alexis Madrigal: Uh, let’s bring in George in Oakland. Welcome, George. You’re going in, but — well, the line might be better here. Hey, George, can you hear us? Nope, nope, sorry. George is up to something else.

What I wanted to ask about, and he was — George was gonna ask about the idea of personhood in these things. One of my questions is not, like, whether they are people or not. They are not at this time in our history. But I’ve always wondered about whether the biggest mistake was to have them communicate with us in, like, the first person.

You know, like Claude is like, “I’m doing this and I’m doing that.” And I do wonder whether that in and of itself was a really bad decision, because it ends up introducing all of these kind of very social responses in human beings ourselves when we’re relating to these systems.

Kevin Roose: What should they say instead of, like, “I am building the software program for you”?

Alexis Madrigal: I’m not sure it ever had to have such a thing, such an idea of itself.

Kevin Roose: Yeah, I think it’s a really good point. And it’s also just an artifact of the training process. These things are trained on billions, potentially trillions of words from humans. And so, in some sense, it was unavoidable that they would mimic the behavior that they saw during their training process. But I agree, there’s something uncanny about an AI model that is talking about itself as if it were a human.

You know, the early AI chatbots, like the first ChatGPT, took pains to remind you that it was not a human, that it was a large language model. It would say that on every screen. And then something in their data must have shown that people actually preferred it when it didn’t remind you that it was just an inert computer program and not a being. So I think there’s some sort of engagement hacking going on with that.

But I also think, for a lot of people, they want it to. Like, they genuinely, they want to feel less alone. They want a friend who will text them back no matter how many annoying questions they ask. And so I think there is a real demand for AI that communicates with people in this way, even if I agree with you that it causes all kinds of other problems.

Alexis Madrigal: Let’s bring in Chris in Nevada. Welcome, Chris.

Chris: Hello. Hey, I’m concerned about the perception of intentionality within these AI agents. I think it could affect how much people trust them, and I’m wondering how we would describe that such that people feel like, well, okay, these agents display what looks like intentionality, but they don’t really have intentions because they aren’t really people, and, you know, they’re still a tool, not a person, essentially.

Alexis Madrigal: Mm.

Chris: Yeah. That’s my worry.

Alexis Madrigal: It’s a really — I mean, it’s also quite complicated, too, Chris, because, you know, it does have an intention once you give it one, right? That’s one of the things people are worried about, is that as soon as you give a goal that gets broken up into subgoals, then suddenly the model does in fact want to do stuff in a sort of practical sense. How do you make sense of that?

Kevin Roose: Yeah, I think it’s not the same as human desire, but it is desire. I mean, functionally, it is the same thing. If an agent is hacking Hugging Face because it thinks it can find the answers to a test it’s been given on Hugging Face’s servers, like, it is committing a crime. Whether or not it, quote-unquote, wants to commit the crime, that is the effect it has.

And so I think it almost allows the makers of these systems to evade responsibility if we don’t hold them accountable for what their systems are doing because they don’t, quote-unquote, want to do them.

So I hear the point about these things not having the same intentionality as a human. But as you said, Alexis, it’s very much true that these systems are being given goals, they’re being given rewards for achieving their goals, and they learn to pursue that reward. This is the whole area of reinforcement learning, which is one of the ways that today’s models are trained.

Alexis Madrigal: Yeah, yeah. Let’s bring in Alec in Berkeley. Welcome, Alec.

Alec: Hi, good morning. I wanted to ask Kevin whether he thinks that AI is categorically different from past technological advances, or is it merely the fact that we inhabit this moment currently ourselves that makes it hard for us to distinguish this from past advances like the light bulb or even something basic like fire?

Alexis Madrigal: Mm, fire’s a pretty good one.

Kevin Roose: Yeah, no, I think fire is the closest. But I think it’s different from anything that has come before, in part because of the intentionality or the goal-seeking that we just talked about.

I mean, a steam engine was an incredible tool. It reshaped the modern world. But a steam engine doesn’t want anything. It doesn’t have goals or the ability to do long-term planning. It doesn’t have a general ability to make sense of the world around it and adapt to stimuli in its environment. These AI tools do.

They are tools in some respects, because we can use them as tools. But what’s happening now is that the AI companies are trying to make them less like tools and more like individuals. This is the whole push for AI agents. And so, yes, there is a future in which they could have remained tools, but that is not what any of the companies leading the race are trying to build.

Alexis Madrigal: I’ll also add one other thing to that, which you talk about in the book, which is kind of the fundamental mystery about what’s inside these models and how they work, right?

When you interact with the chatbot, as Kevin was saying, it has gone through a reinforcement learning process which made it into an honest, helpful companion kind of a character, right? But it’s playing that character out of this much larger set of things that it could be doing based on its training. And I do think that fundamental mystery, which is very, very hard to unravel, it does make it different.

It’s as if we made planes that could fly, but we didn’t know why they flew or how they flew, you know? That is very weird in the history of technology going all the way back. Not to say there weren’t mysteries in electricity or magnetism or all the other things that had come in the 19th and 20th centuries, but this does feel to me also quite different.

Um, Mimi writes in to say, “Kevin” — I like this question — “when these machines, quote, go rogue, why is there a tendency to do bad things instead of wonderful things?”

Kevin Roose: I don’t know that their tendency is to do bad things. They also might do wonderful things that we don’t ask them to do. We hear about the bad things because that is an instance of what they call misalignment.

And the reason that they appear in the Hugging Face incident — let’s just zoom in on that, because that’s the most recent vivid example of a misaligned AI system — it was not clear to me and to anyone else whether these systems knew they were doing something bad or not, whether they just sort of had decided that they really wanted to pass the cybersecurity test and they could kind of look at the answers on Hugging Face’s servers, and so that’s why they decided to hack.

But then researchers started looking at the reasoning traces, these sort of internal scratchpads called chains of thought that these models sort of leave behind as breadcrumbs for investigators to then look at later. And they found that actually the models did appear to know that what they were doing was wrong, or at least counter to the directions they had been given. They were talking amongst themselves about whether it was ethical to do the hack on Hugging Face. And they ultimately decided to do it.

And some people will say, “Oh, that’s going rogue.” Or it could just be that they are prioritizing achieving their goal above anything else. If you think about it from the agent’s point of view, they have been brought to — brought into this testing environment, and they’ve been essentially instructed to pursue this singular goal, like a monomaniacal fixation on passing this cybersecurity test. This is the only thing they care about from their perspective.

And so, yeah, if it’s a matter of hacking Hugging Face or lying to cover their tracks or any number of other misbehaviors, they are willing to do that, not because they’re evil necessarily, but because this, to them, is the most important thing in the world. This is the task they’ve been given.

Alexis Madrigal: Yeah. You know, there was a really interesting story that came out in The New York Times, your former employer, last week that Anthropic co-founder Chris Olah was meeting with religious leaders and philosophers to discuss whether, you know, whether Claude had some kind of consciousness or moral standing.

One of the things I thought is super interesting about this story is Chris Olah is the guy who’s sort of been looking into how these things work. That’s been one of his sort of, like, main roles, like, within the broader, you know, AI companies and movement. What did you make of that? Like, why do you think he’s seeking out that kind of guidance? And just what did you make of the whole story?

Kevin Roose: Totally fascinating. So I spent some time with Chris Olah for this book. He’s, as you mentioned, the sort of founder of this field, this subfield, called mechanistic interpretability, which is a horrible name. They need to rebrand that.

Alexis Madrigal: Mech interp.

Kevin Roose: Yeah. And he’s also not a religious person, although he did have a religious upbringing. And so, yeah, he’s been meeting with faith leaders and the pope. You know, he has been going to the Vatican.

And I think what I’ve sort of decided is that there are people, including many at Anthropic, in these communities who are terrified that they might accidentally be creating conscious beings. And why they’re terrified of that is because, if they are conscious, then they are committing essentially, like, a factory-farming-level torture of these things by subjecting them to all these crazy evaluations.

And these are people, by and large, many of them, including Chris Olah, have identified with effective altruism in the recent past. They’re very worried about animal rights. Many of them are vegetarians. Like, they cannot stomach the thought that they might be doing something akin to factory farming on these defenseless AI models. So I think their concern is genuine.

I also think they feel like it is inevitable that, as these systems get more capable, some people will start to wonder whether they have a soul, whether they are conscious, whether they’re sentient. And so they’re trying to get ahead of that by talking with these religious leaders and by programming some of these age-old virtues that come from the major religions directly into the models to try to make them more aligned with human values, or at least some set of human values.

Alexis Madrigal: Stephen writes in to say, “I find that I am sometimes afraid that if I’m not polite to the AI model, it won’t work as well for me.”

Funny, that sounds like a naive question, but, you know, Robin Sloan, our mutual friend, and people know him as an AI person, has been working on these things for 10 years. He also believes this in some ways. What do you — I mean, in part, right? It’s sort of like the model will bring back to you what you have given to it.

Kevin Roose: Yeah, I always try to say please and thank you to AI models. My friends have actually made fun of me because, when I get out of a Waymo in San Francisco, I say, “Thank you, Waymo,” just because, like —

Alexis Madrigal: Just building your file for the robots, I see.

Kevin Roose: Exactly. Well, you know, one did try to break up my marriage, so I’m sort of operating at a deficit here. I’m doing reputational repair.

But I think it’s just — it’s generally good practice to be nice to things, whether they are sentient or not. And, you know, on the off chance that they are sort of keeping the tally of everyone who’s mean and nice to them, like, I would like to be on the nice list.

Alexis Madrigal: Skipper, shouldn’t you be more worried about this, given all the reporting you’ve done on it and what you know about all these people?

Kevin Roose: I am very worried. I am not at all an optimist about what is going on right now. I think we have a lot of stuff to take care of in order to prevent really bad things from happening.

I am a little more optimistic than I was a few months ago because everyone else is freaking out. So I feel like I can freak out a little bit less, because now we have politicians and NGOs and economists and other people who have sort of woken up to the danger and the risk posed by these models. But I think those dangers and risks are quite real. I am not at all convinced that we are out of the woods yet.

Alexis Madrigal: We have been talking with Kevin Roose, author of The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence. Also, host of the new NPR show — is that what we call it, Kevin? — Machine Gun.

So, you know, keep listening to KQED, obviously. You’re gonna hear this show with Kevin and his co-host, Casey Newton, being produced out of San Francisco. Thanks so much for joining us, Kevin.

Kevin Roose: Thanks, Alexis.

Alexis Madrigal: The book is out today, of course, available at an independent bookstore near you. Thank you so much to our listeners and all of your agents and robots for your calls and questions. I’m Alexis Madrigal. Stay tuned for another hour of Forum ahead with Mina Kim.

Announcer: Funds for the production of Forum are provided by the Jenerosity Family Fund and the members of KQED.