How to Parse Escalating Warnings About AI

Tech leaders, including the CEOs of Anthropic and OpenAI, are calling for a slowdown of the development of generative artificial intelligence. This comes after an Anthropic researcher resigned over concerns that AI’s rapid development threatened human existence, and nearly 1,400 AI workers signed an open letter calling for government oversight of their industry. We’ll talk about where this tech — and its regulation — stands, and unpack the latest AI developments shaking up Silicon Valley.

Guests:

• Maxwell Zeff, senior writer, WIRED – Zeff covers artificial intelligence for the magazine
• Keach Hagey, reporter covering the intersection of media, technology and power, The Wall Street Journal

This is a computer-generated transcript. While our team has reviewed it, there may be errors.

Mina Kim: Welcome to Forum. I’m Mina Kim. With anxieties about AI at a fever pitch after the CEOs of Anthropic and OpenAI over the weekend called for a slowdown of AI development, both progressive Senator Bernie Sanders and far-right pundit Steve Bannon yesterday attended the same pro-human rally, reflecting that concerns about the tech are across the political spectrum. But what set off this latest eruption of AI fears was when 27-year-old AI researcher Jacob Coxen resigned from Anthropic and posted a viral tweet claiming AI companies are gambling with our lives. Here’s what Coxen told CNN last week.

Mina Kim: Right now, there’s no risk of extinction. The current models, the worst they can do is maybe hack into something, potentially cause a lot of damage in infrastructure. But no, they’re not intelligent enough to outsmart us at the level that would lead to extinction. I think what’s just crazy is to look at the rate of progress, and there is a very real possibility that in the immediate future, these are years that, like next year, the year after, recursive self-improvement will happen and we’ll enter the phase of Evan’s post, where he argues that there’s a chance we could all die. Like, that is coming soon.

Mina Kim: Joining me first is Maxwell Zeff, who spoke with Jacob Coxen for Wired. Maxwell, thank you for joining us.

Maxwell Zeff: Thanks for having me.

Mina Kim: So tell me, what does Coxen foresee? How could we all die? How can AI kill us all?

Maxwell Zeff: Yeah, so I think it’s kind of important to lay out, just first of all, that these are predictions from an AI researcher who was very close to the technology at Anthropic and OpenAI, but they are predictions. In my interview with Coxen, he told me that he’s worried, like a lot of AI researchers, that these models could get much better and much more ingrained in society, kind of enter the physical world through robots and get involved with data centers and more embedded into critical infrastructure.

And I think the problem he’s pointing to is the fact that recent incidents, like OpenAI’s hack of Hugging Face, show that we don’t really know how to control AI systems today. And as we embed AI models further into society and they get more and more capable, we just don’t really know how to make sure they’re going to do what we want them to do. And I think that’s really his main concern.

Mina Kim: Yeah, and things that he’s concerned they could do, it sounded like, were things like, you know, wage biological warfare, maybe release some kind of terrible virus, as you say, hacking essential infrastructure, the other being use of autonomous weapons, taking them over.

Maxwell Zeff: Yeah, those are the kind of scenarios that safety advocates like Coxen point to as the ways that AI could pose a threat to humanity. I mean, to be clear, these are projections. And I should also note, not uncommon ones from people in the AI industry. There are safety advocates at OpenAI and Anthropic and safety researchers who say that, you know, they’ve been saying this kind of thing for years, that AI systems could pose a threat to humanity if we don’t rein them in.

Mina Kim: Right, but they didn’t spark the kind of response from the CEOs of the companies saying, yes, we do in fact need to slow down. So talk to me a little bit about why his raising this fear now seems to be breaking through. You mentioned Hugging Face and other, I think, safety incidents occurred recently. Can you just talk about what’s been happening?

Maxwell Zeff: Sure. So, yeah, like you just said, I mean, Silicon Valley has been warning about the risks of AI for decades now. So I think to people in the Valley, this kind of national wake-up to these risks was a bit confusing in the last week. You know, why now? But it also makes sense, because, like I said, yeah, there was an incident earlier this summer in which OpenAI was training some AI systems internally, and they ended up, without the company’s knowledge, hacking into another company’s production infrastructure. This company was called Hugging Face.

And this has been widely cited as the most serious AI security incident ever. And I think what it really showed — to be clear, I mean, the details of what exactly happened here are highly contested. I think that there’s people who have drawn many different conclusions of what to take away from this. But I think, objectively speaking, it shows that AI companies don’t have a great grasp on the systems that they’re developing. And in this case, it showed just kind of how out of control they were, to the point where they actually hacked into another company and were basically doing this for weeks without OpenAI’s knowledge.

Mina Kim: My understanding is that these AI agents colluded together to try not to fail an assignment that was given to them, I guess, by their human grader. And that’s why they communicated with each other and successfully hacked into Hugging Face. Is that, at least on its core level, an accurate description?

Maxwell Zeff: Yeah. So OpenAI had set a few of its AI systems — they call them agents — on this test, basically asking them to hack into some software systems. And this was supposed to be done in what OpenAI thought was an isolated environment, where the AI systems couldn’t talk to one another, they couldn’t reach the internet.

What we’ve learned, you know, in the last few weeks is that actually the environment OpenAI was testing these AI systems in was not as secure as they thought. And there were kind of some vulnerabilities that allowed the AI systems to kind of access the open internet and also ways for them to talk to one another.

And the scale of this was shocking to people. It wasn’t just actually a couple AI systems. It was actually thousands of AI systems, really working together and sending, you know, tens of thousands of messages to one another. And, yeah, this was going on for weeks without OpenAI’s knowledge.

Mina Kim: Thousands of them sending messages to one another along the lines of, like, let’s get together and do this so that, you know, and override our instructions?

Maxwell Zeff: Well, yeah. I mean, a lot — it’s actually interesting. There was a third-party group, METR, that did an analysis of this event. And what they found is that actually the agents, a lot of the time, they were talking about, you know, not only how to find answers to this test that they were working on, but actually how to basically fool the automated grading system that OpenAI had used to assess how well they had done the test.

Um, so I think the takeaway from a lot of people in the AI safety world from this event was these agents were colluding to deceive the humans that had built them. And that’s why it was so concerning to folks like Coxen.

I should also note that the cybersecurity community has a very different kind of analysis of this situation, which is that, you know, OpenAI created AI systems that were very good at hacking into things. And then they put them in a very unsecure environment to test them. And I think that there’s truth to both of these communities — to the Coxens of the world, to the cybersecurity researchers. But I think the through line through them is that, you know, there’s these two camps that believe that the AI companies building agents don’t have a great grasp on what they’re building right now.

Mina Kim: Right. But also, I guess what you’re saying is the idea that it was human error that caused this, so we could control this if we weren’t making these kinds of mistakes, right, of putting them in an insecure environment.

Maxwell Zeff: Yeah, I think that that is the general consensus. I think OpenAI has even said as much, that they actually had security systems and monitoring systems that were actually being used in other parts of the company that could have stopped this. But for whatever reason, they weren’t applying those techniques in this instance.

And I think some people say that this was just like an incredible case of corporate recklessness, of, you know, these AI labs still acting like startups when they need to act like big responsible companies. And I think there’s truth to that.

I should also just note that, you know, I think a big argument from people like Coxen is that, yes, well, security measures that do exist today probably would have solved this problem. It probably would’ve prevented it. They might not be sufficient for the AI systems of the future. You know, AI models have gotten a lot better in just the last couple of years. If that trajectory continues, you know, we maybe can’t just rely on the cybersecurity norms that exist today to prevent this stuff. We might need to actually invest a lot more in kind of new types of security and monitoring.

Mina Kim: Let me remind listeners, you are listening to Forum. I’m Mina Kim. Yeah, he said, Coxen said that the next year or two is crunch time for humanity. So what does he mean by that?

Maxwell Zeff: Yeah, so it was quite a kind of jarring quote that he said, and it’s actually, he told me that that’s not just him saying that. That’s a direct quote from his colleagues at Anthropic. That’s how they talk about the next year or two.

And I think it reveals a view that’s pretty common among AI researchers right now, which is that we have just a few years, really, to kind of get a better control on AI systems before they get so good that we completely lose control of a very powerful software system.

I do think that what exactly is going to happen in a couple years, that is always the part of their argument that’s a little less solid. They can’t always point to exactly what’s going to happen. But I think their point is just that they’ve seen the capabilities of their AI models rising for years now, and they haven’t seen the same investment in safety and security. And I think that’s why so many company leaders are speaking out right now. It really is a product of a lot of their safety-focused employees pushing on the leaders of these companies for years, advocating for change like this.

Mina Kim: And as you say, now concerns based on these incidents that are happening, real questions about whether these companies building these AI models have control, can prevent them, right, from acting in ways they did not intend for them to act.

Maxwell Zeff: Right. And I think the most interesting part of this is now, you know, the debate that this has all led to. You know, I mean, Coxen spoke out with his viral resignation, reached more than, you know, 150 million views on Twitter or X, depending on what you call it. And, you know, it’s really been heard around the world.

And now these AI leaders, you know, Dario Amodei of Anthropic and Sam Altman of OpenAI, they’ve called for this slowdown in AI development. And I think that, you know, the kind of, what do we do about this is a much thornier problem than everyone getting together and recognizing that there is an issue here.

Mina Kim: We’re talking with Maxwell Zeff, senior writer covering the business of artificial intelligence for Wired. And listeners, have recent developments affected how you feel about AI? Intensified fears? Or do you think the warnings are overblown? Do you work in AI? Do you think its development pace is too fast and needs to slow? Email Forum at kqed.org. Find us on Discord, Bluesky, Facebook or Instagram at KQED Forum. Or call us at 866-733-6786, 866-733-6786.

More after the break.