When Technology Can Be Used to Build Weapons, Some Workers Take a Stand

On the night of Jan. 16, Liz O’Sullivan sent a letter she’d been working on for weeks to her boss, Matt Zeiler, the founder and CEO of Clarifai, a tech company.

“The moment before I hit send and then afterwards, my heart, I could just feel it racing,” she says.

The letter asked: Is our technology going to be used to build weapons?

With little government oversight of the tech industry in the U.S., it’s tech workers themselves who increasingly are raising these ethical questions.

O’Sullivan often describes technology as magic. She’s 34 — from the generation that saw the birth of high-speed internet, Facebook, Venmo and Uber.

“There are companies out there doing things that really look like magic,” she says. “They feel like magic.”

Her story began two years ago, when she started working at Clarifai. She says one of her jobs was to explain the company’s product to customers. It’s visual recognition technology, used by websites to identify nudity and inappropriate content. And doctors use it to spot diseases.

Clarifai was a startup, founded by Zeiler, a young superstar of the tech world. But shortly after O’Sullivan joined, Clarifai got a big break — a government contract, reportedly for millions of dollars.

It was all very secretive. At first, the people assigned to work on the project were in a windowless room with the glass doors covered.

O’Sullivan would walk by and wonder: What are they doing in there?

Zeiler says the contract required secrecy, but everyone working directly on the project knew what it was about. “We got briefed before even writing a single line of code,” he says. “And I also briefed everybody I asked to participate on this project.”

NPR spoke to one employee who worked directly on the project. That person, who requested anonymity for fear of retaliation, says many of the workers in that room were not entirely clear what the technology they were working on was going to be used for.

In the months that followed, former employees say, information started trickling down. It turns out, they were working with the Department of Defense.

Then, people working on the project got an email that outlined some details. The text included a brief reference to something called Project Maven.

The Pentagon told NPR that the project, also called Algorithmic Warfare, was created in April 2017. Its first task was to use computer-vision technology for drones in the campaign against ISIS.

“This could be more effective than humans, who might miss something or misunderstand something,” explains Ben Shneiderman, a computer scientist at the University of Maryland. “The computer vision could be more accurate.”

Shneiderman had serious ethical concerns about the project. And he wasn’t alone. Many people in the tech world were starting to wonder: What will the technology we’re building be used for down the road?

O’Sullivan says this question began to haunt her too.

The big fear among tech activists is that will this be used to build autonomous weapons — ones that are programmed to find targets and kill people, without human intervention.

The Department of Defense’s current policy requires that autonomous weapons “allow commanders and operators to exercise appropriate levels of human judgment.”

It’s a definition many find murky. And last year, tech workers began to ask a lot of questions. “It’s a historic moment of the employees rising up in a principled way, an ethical way and saying, we won’t do this,” Shneiderman says.

In 2018, Microsoft employees protested their company’s contract with Immigration and Customs Enforcement. And several thousand employees demanded that Google stop working on Project Maven. Google did not renew its contract with the project.

Last June, Clarifai CEO Matt Zeiler also weighed in. In a blog post, he explained why the company was working on a military project.

O’Sullivan read that with interest. “You know, the people running these companies are sort of techno-Utopians,” she says. “And they believe that tech is going to save the world and that we really just have to build everything that we can, and then figure out where the cards fall. But there are a lot of us out here saying, should we be building this at all?”

Former Clarifai employees told NPR that at the office, the mood got tense.

There were plenty of people who felt comfortable working on Project Maven. Others resented that it had been so secretive. And some just found it morally troubling.

As the months went by, O’Sullivan says she realized she couldn’t change the direction of the company. So at the beginning of this year, she wrote that letter to Zeiler and sent it to the whole staff.

“We have serious concerns about recent events and are beginning to worry about what we are all working so hard to build,” she wrote.

She went on to ask a bunch of questions. Many of them are the same ones being asked across the tech world today.

“Are you going to let us know who we’re selling our stuff to?”

“Are you going to vet how it’s used?”

“Do we care if this is used to hurt people?”