AI Text Detectors: Can You Trust Them?

A master's student turns in a thesis she wrote alone over six weeks. Her university's detection tool displays "68% likely AI-generated." She is summoned to a meeting. She has no dated draft and no edit history, because she writes in a single file that she overwrites with every save. She ends up having to prove she didn't do anything.

This kind of situation is regularly reported in education. It comes from a misunderstanding about what a detector measures.

What a detector does

A detector doesn't find the trace of a tool in your text. There isn't one, unless the provider has added a deliberate watermark, and most texts don't carry one. The detector is itself a model trained to recognize regularities: sentences of similar length, highly predictable words, a smooth style. From these it derives a probability.

The problem is that these regularities aren't unique to machines. A bureaucratic text, a school essay, a factual summary, or a text written in a language that isn't the writer's own resembles what an AI produces. Researchers have in fact shown that texts by non-native English speakers were flagged wrongly more often than those by native speakers. Conversely, a text that was generated and then lightly rewritten, or produced with a prompt asking for a more irregular style, passes without difficulty.

OpenAI published its own text detector in 2023 and withdrew it a few months later, citing its low accuracy. When a model's own maker can't reliably recognize its texts, it's better not to treat a detector as proof.

A percentage is not a verdict

Two errors are possible, and they don't cost the same.

• The false positive: a human text is flagged as generated. This is the student's case, and it's the more serious one, since it accuses someone of cheating.
• The false negative: a generated text passes as human. It makes people believe the tool works better than it does.

A score of "68%" doesn't mean that 68% of the text was written by an AI, any more than it means there's a 68% chance the text is generated. Tools don't always explain this clearly, and several vendors themselves specify that the score shouldn't be the sole basis for a penalty. Read the documentation of the tool you use: it's the most telling document.

Detectors aren't equal to one another, and two tools can give different scores to the same text. Run a test with a text whose origin you know.

If you're a teacher or administrator

Use the score, if you look at it, as a signal to ask a question, never as proof. What works better:

• ask for a version history, drafts, notes, or sources;
• talk with the author about their text in person;
• adapt the assignment so that it rests on a personal process (a lived experience, a case from class, a step to show) rather than on a generic text;
• set out in advance, in writing, what is allowed with AI and what isn't.

The interview is the most productive. An AI can help you prepare it from the assignment itself:

Here is the assignment of an 11th-grade student on [SUBJECT]:

[TEXT OF THE ASSIGNMENT]

Prepare five oral questions to check that the student understands what they wrote: two about their argumentation choices, two asking them to explain a passage in their own words, one asking them to extend an example. Make no judgment about the origin of the text.

The last sentence matters. The goal is to open a discussion, not to build a case. Also avoid pasting the assignment of an identifiable student into a tool whose retention rules you don't know. More ideas in the guide to prompts for teachers.

If you're falsely accused

Stay factual and gather what you have: drafts, document history (online word processors often keep versions), notes, sources consulted, exchanges with a supervisor. Offer a meeting to explain your work, which is often the best defense. Ask which tool was used, what score, and on what basis, and check what the tool's documentation says about the reliability of its score. If the procedure is serious, get help from a representative (student union, ombudsperson, legal advice).

Going forward, get into the habit of writing in a document that keeps its edit history, and of keeping your notes. This is also good practice if you use AI in an authorized way, for example to rephrase: keep both versions. The article for students explains how to use these tools without putting yourself in difficulty.

And if you want to write a text that doesn't "look like AI"

Don't try to fool a detector: it teaches you nothing, and the rule of your institution or your client stays the same. If AI helps you, what makes a text truly yours is the material you put into it: your examples, your verified figures, your way of saying things, what you find in the article on your style. The biases of tools, including detectors, are covered in the article on AI bias.

For next time

If your institution or company uses a detector, ask in writing how the score is used and whether it can be the sole basis for a decision. And if you're writing an important document, turn on version history today.