Teachers Are Using Software To See If Students Used AI. What Happens When It's Wrong?

Ailsa Ostovitz has been accused of using AI on three assignments in two different classes this school year.

“It’s mentally exhausting because it’s like I know this is my work,” says Ostovitz, 17. “I know that this is my brain putting words and concepts onto paper for other people to comprehend.”

Ostovitz, a junior at Eleanor Roosevelt High School in the Maryland suburbs of Washington, D.C., shared with NPR one of the accusations she received from a teacher. The message, from September, included a screenshot from an AI detection program showing a 30.76% probability Ostovitz had used AI on a writing assignment that included a description of the music she listens to.

“I write about music. I love music. Why would I use AI to write something that I like talking about?” Ostovitz says.

Ostovitz reached out to her teacher about the assignment via the school’s online learning platform. “I said, seriously, I didn’t use AI. Can you try a different detector?”

The teacher didn’t respond, and docked Ostovitz’s grade.

Ostovitz’s mom, Stephanie Rizk, says her daughter is a high-achieving student who cares about doing well in school and she was alarmed when the teacher jumped to conclusions about Ostovitz’s work so early in the school year.

“Get to know their level of skill, and then maybe your AI detector is useful,” Rizk says.

Rizk told NPR she met with the teacher in mid-November and the teacher said they never saw her daughter’s message.

The school district, Prince George’s County Public Schools, made clear in a statement that Ostovitz’s teacher used an AI detection tool on their own and that the district doesn’t pay for this software.

“During staff training, we advise educators not to rely on such tools, as multiple sources have documented their potential inaccuracies and inconsistencies,” the statement said.

PGCPS declined to make Ostovitz’s teacher available for an interview. Rizk told NPR that after their meeting, the teacher no longer believed Ostovitz used AI.

But what happened to Ostovitz isn’t surprising.

More than 40% of surveyed 6th- to 12th-grade teachers used AI detection tools during the last school year, according to a nationally representative poll by the Center for Democracy and Technology, a nonprofit that advocates for civil rights and civil liberties in the digital age.

That’s despite numerous research studies showing that AI detection tools are far from reliable.

“It’s now fairly well established in the academic integrity field that these tools are not fit for purpose,” says Mike Perkins, a leading researcher on academic integrity and AI at British University Vietnam.

Perkins found that some of the most popular AI detectors — including Turnitin, GPTZero and Copyleaks — flagged some things as AI that weren’t, and vice versa. Their accuracy rates dropped even further when AI text was manipulated to appear more human.

“We saw some really concerning problems with some of the most prolific AI text detection tools,” he says.

Despite those problems, NPR found that school districts from Utah to Ohio to Alabama are spending thousands of dollars on these tools.

Why one of the nation’s largest districts uses AI detection software

Near Miami, Broward County Public Schools is spending more than $550,000 on a three-year contract with Turnitin. The long-standing ed-tech company has historically provided schools with plagiarism detection software; in 2023, it introduced an AI detection feature. When educators put student work through this tool, it generates a percentage, which reflects the amount of text the software determines was likely generated by AI. One caveat: According to the company, scores of 20% or lower are less reliable.

“The Turnitin tool is something that helps us facilitate conversation and feedback, not grading,” says Sherri Wilson, director of innovative learning for the Broward school district, which enrolls more than 230,000 students and is one of the largest school districts in the country.

Wilson says the district is “totally aware” of the research showing AI detection tools, including Turnitin, aren’t 100% accurate or reliable.

Turnitin also acknowledges this: On the company’s website, it says, “our AI writing detection may not always be accurate … so it should not be used as the sole basis for adverse actions against a student.”

Turnitin wrote in a statement to NPR that it’s more important to avoid falsely accusing students of cheating than to catch all AI writing.

Wilson says the Turnitin tool is still valuable because it saves teachers time by quickly scanning student work for suspected AI use.

Another reason that Broward teachers have access to the tool, Wilson says, is that the district participates in academic programs, such as International Baccalaureate, or IB, in which student work must be authenticated by teachers before it is sent out for external review.

Both of the programs Broward offers, IB and International Education at Cambridge, told NPR that schools are not required to use AI detection software as part of the authentication process. Nonetheless, Broward told NPR in a statement, “we have chosen to provide our teachers with [Turnitin] as one of the tools to meet the requirements.”

But Wilson says teachers are the ultimate authority on whether a student’s work is their own — not the AI detection tool.

“They’re using these tools as feedback to then have those teachable moments with students,” she says.

Why one teacher uses AI detection tools

Language and literature teacher John Grady says, for him, AI detection tools provide “a jumping off point” to start a conversation with a student who may have used AI.

“It’s certainly not foolproof,” he says. “But it gives you something to hang your hat on.”

Grady teaches at Shaker Heights High School, part of the Shaker Heights City School District outside Cleveland. The district serves roughly 4,400 students, and is paying GPTZero, another AI detection software company, about $5,600 this year for annual licenses for 27 of the district’s teachers. The tool calculates a percentage likelihood that a student’s work is AI-generated.

Grady says he puts all student essays through GPTZero; if the tool shows more than a 50% likelihood AI was used for the assignment, Grady digs deeper. That includes using revision history tools to see how much time a student spent on an assignment, and how many edits they made during the writing process. If it appears that a student made only a few edits and spent hardly any time writing, he’ll check in with that student.

“And I’ll say, ‘Hey, this flagged. Can you talk to me about why?’ I’d say the bulk of the time, like 75%, if it was AI, they’d be like, ‘Yeah, I did.’ And I’m like, ‘OK, well now you’ve got to rewrite it with less credit,'” Grady says.

Edward Tian, co-founder and CEO of GPTZero, says this is how educators should be using his company’s tool.

“We definitely don’t believe this is a punishment tool,” Tian says. “This needs to be a tool in the toolkit and not the final smoking gun.”

He says it’s important to understand that a GPTZero probability score under 50% means it’s more likely the text was human versus AI-generated. He says scores over 50% warrant closer examination — like what Grady describes.

Tian doesn’t dispute the research that shows GPTZero isn’t always reliable. But he notes that there are educators, like Grady, who still find it valuable for the information it provides.

He says that tools like his offer a “signal on what’s happening in your classroom” but that teachers should always follow up with students if that signal shows something concerning.

The AI detection skeptics

Shaker Heights junior Zi Shi, whose first language is Mandarin, says his writing style can sometimes look like AI “because of the repetition of words I use. I feel like it’s because of how limited my vocabulary is.”

Shi — who isn’t a student of Grady’s — says he’s still working on his writing skills and he’s concerned that AI detection software might be biased against non-native English speakers like himself.

Some educators share this concern, though the research so far is limited and contradictory.

Shi says an assignment he completed for his English class earlier this fall was flagged by GPTZero as possibly AI-generated. He says his teacher suggested that his use of an online tool called Grammarly may have triggered the detection software. Grammarly uses AI to correct grammar and, if prompted, generate text. (The teacher confirmed Shi’s account with NPR.)

Shi says he only used Grammarly to clean up his writing and that he wrote the assignment himself. “It was definitely disappointing to see the comment of it being flagged as AI,” Shi says.

Shi thinks AI detectors should be thought of as a “smoke alarm, where it’s a sign, or warning. But, you know, sometimes it could be like a false alarm.”

He questions whether the school district should be spending thousands of dollars on AI detection software. He says that money could be better spent on professional development for teachers.

Carrie Cofer, a high school English teacher in the Cleveland Metropolitan School District — just a few miles from Shaker Heights — shares that view.

Last year, as an experiment, she uploaded a chapter of her Ph.D. dissertation into GPTZero. “And it came up with like 89% or 91% AI-written, and I’m like, ‘Oh, no, I don’t think that’s right, because it was all mine,'” Cofer says.

Cofer is helping her district shape its AI policy and guidelines; she says Cleveland schools don’t currently pay for AI detection software and she’d advocate against it.

“I don’t think it’s an efficacious use of their money,” Cofer says. “The kids are going to get around it one way or the other.”

Some workarounds that students could turn to include using AI detection software themselves, to workshop assignments so they don’t get flagged, and using “AI humanizer” programs, which claim to make AI-generated writing appear more human.

Ultimately, she says, teachers will need to adapt to AI by changing how they teach and assess student learning.

Back in Maryland, high school junior Ailsa Ostovitz is also adapting. She now runs all her homework assignments through multiple AI detection tools before she turns them in.

The writing is her own, she says, but she’ll rewrite sentences the software identifies as possibly AI-generated, an extra step that adds about half an hour to every assignment.

“I think I’ve definitely become more vigilant about presenting my work as mine and not AI,” she explains.

She doesn’t want to take any chances.

This reporting was supported by a grant from the Tarbell Center for AI Journalism.

Edited by: Nicole Cohen
Visual design and development by: LA Johnson
Audio story produced by: Lauren Migaki