Checking an AI Response: What to Verify, and How

In 2023, in New York, lawyers were sanctioned by a federal judge for filing a brief that cited court decisions that didn't exist. They had been produced by ChatGPT, and the Mata v. Avianca case became the classic example. The references looked impeccable: a court name, a number, a date. Nobody had read those decisions before citing them.

This story teaches a simple lesson. What fools us is the form: a well-written answer triggers our trust. Checking everything is impossible and pointless. You need a way to decide what to verify, with how much effort and against which source.

Start by estimating what an error would cost

Before checking anything, ask yourself one question: if this answer is wrong, what happens?

A flop of a slogan on a poster shows up immediately and costs little. A wrong VAT rate on a tradesperson's quote can cost a lot and go unnoticed. Incorrect allergen information on a restaurant menu can affect a customer's health. The verification effort should follow that risk, not the length of the text.

Three levels are enough in practice. When an error has no consequence, a quick read is enough. When it puts your credibility on the line, as with an article or a client presentation, you check every fact. When it touches law, money or health, the AI is only a starting point: the reference is the official source or the relevant professional.

Checking by type of claim

An AI text mixes sentences that don't call for the same check. An opinion or a rephrasing is judged by reading. A factual claim needs a source. This table shows where to focus your attention.

On that last point, a quick test is to ask for the exact quote supporting each idea in the summary, then search for those sentences with Ctrl+F in the original.

Sources: open, read, compare

Tools connected to the web, like Perplexity and its sourced searches, have improved things, but a citation displayed next to a sentence isn't proof. The link may exist and say something else, or say the same thing with caveats that the answer made disappear.

For a source, four questions are enough:

1. Does the document really exist?
2. Does it say what it's made to say, with the same nuance?
3. Who published it, and do they have a stake in the matter?
4. Is it recent enough for your topic?

Beware of false confirmation too. Several sites repeating the same information may all rely on a single original source, sometimes itself produced by an AI. Several copies of the same text aren't worth two sources. Trace it back to the first publication.

What the AI can do to help you verify

Asking "are you sure?" doesn't achieve much. The model may apologize and change a good answer, or defend a bad one. As our article on how language models work explains, it has no direct connection to the facts.

It is, however, very useful for preparing your checking work: isolating what needs to be verified and indicating where to look. Paste the text into a fresh conversation, so it reads it without trying to defend its own answer.

You're going to help me proofread the text below before publication. Don't rewrite it and don't correct it.

Build a table with these columns:
1. Verifiable claim, copied word for word.
2. Type: number, date, name, quote, legal or tax rule, technical fact, other.
3. Severity if it's false for [AUDIENCE OR USE]: low, medium, high.
4. Source to consult to verify it, describing the type of organization or document.

Sort the table by decreasing severity. Never state that a claim is true: your task is only to show me what I need to check. At the end, add the implicit claims, meaning what the text assumes without saying so.

Country and date concerned: [COUNTRY, DATE].

Text:
"""
[TEXT TO VERIFY]
"""

This prompt doesn't replace your own checking. It can miss a claim, and the sources it suggests should be taken as leads. Above all, it keeps you from rereading by instinct.

Using a second tool has value, as long as you don't trust it blindly. If two models disagree, you know where to dig. If they agree, they may have learned the same error from the same texts. You'll find how to reduce these errors upstream in our guide to limiting hallucinations.

The traps that survive a quick read

Some errors get past even careful proofreaders.

True and false in the same sentence. A correct date paired with a wrong place wins the reader's trust. If you verified half the sentence, you haven't verified the sentence.

What's absent. The hardest thing to spot is what's missing: an exception, a condition, a deadline. A rule presented without its threshold of application may be right in general and wrong for your case. Ask the question: "In which cases does this rule not apply?" then check the answer.

Details that are too perfect. A very precise number with no source, a quote that lands at exactly the right moment, a conveniently named study are warning signs.

Fatigue. After five correct answers, your attention relaxes. That's exactly when the error slips through, especially when you use the tool every day. A short written checklist helps you keep going, because it doesn't depend on your alertness in the moment.

Make it a habit with your next text

On the next text the AI writes for you, highlight every number, proper noun and date, then open at least one source for the three items that would hurt you most if they were wrong. Note what you find. After a few texts, you'll know your tool's weak points on your topics, and you'll know exactly where to look.