Scoring AI Answers With a Rubric

The manager of a campground replies to customer reviews, especially the bad ones. She has written two prompts for the AI to draft those replies and can't decide between them. She tries them on one review, finds answer A "warmer" and B "more professional," and picks A. The next day, on another review, she rereads B and changes her mind. Her judgment depends on her mood at the moment, the review in front of her, and what she read just before.

A scoring rubric fixes that. It doesn't make you objective, but it makes you consistent: you look at the same things, in the same order, every time.

What a good rubric contains

Four or five criteria, no more. Beyond that, you stop applying them seriously. Each criterion must be concrete, meaning another person would reach the same verdict as you when reading the answer.

Here is a rubric for the campground's replies:

What makes the rubric usable is the description column: for each score, you know what justifies it. A scale of 1 to 10 with no description is a feeling with a number on it, and your 7 today isn't tomorrow's.

Write the criteria before looking at the answers. Otherwise you'll adjust them, without realizing it, to favor the one you liked at first sight. It's the same discipline as in the article on testing a prompt, which also covers choosing the cases.

Comparing two prompts across several cases

The principle: you take five real reviews, run prompt A and prompt B on each, score the ten answers with the rubric, and add up. Five cases is few, but it's five times more than one, and above all you can see whether the gap comes from a single difficult review or repeats. Include at least one aggressive review and one unfair review.

You can also score without knowing which prompt produced what. Ask someone close to you to shuffle the answers, or number them at random. It's a detail that seems excessive until the day you catch yourself giving one extra point to your favorite prompt.

Having AI do the scoring: possible, with safeguards

Scoring ten answers by hand takes time. AI can do a first pass, as long as you don't let it have the last word. Here is a prompt to run in a fresh conversation, because an AI tends to find good the answer it has just written.

You are evaluating a campground's replies to customer reviews, using the rubric below. You rewrite nothing.

Rubric:
- Acknowledges the specific problem: 0 if not mentioned, 1 if mentioned in general terms, 2 if repeated in the customer's words.
- Accurate facts: 0 if a fact is invented, 1 if no invented fact but nothing concrete, 2 if the facts come from the review or the notes.
- Action offered: 0 none, 1 vague, 2 specific.
- Tone: 0 defensive or servile, 1 correct but stiff, 2 direct and polite.
- Length: 0 over 120 words, 1 from 90 to 120, 2 under 90.

Customer review:
"""
[REVIEW]
"""

Campground notes (the only permitted source of facts):
"""
[NOTES]
"""

Reply to evaluate:
"""
[REPLY]
"""

For each criterion, give the score and quote the passage of the reply that justifies it. If you hesitate between two scores, choose the lower one. End with the list of facts in the reply that appear in neither the review nor the notes.

Three precautions:

• Review a sample. Take three replies scored by the AI, score them yourself, and compare. If your scores often diverge, your rubric is too vague or the AI isn't applying it the way you mean.
• Beware of length. Automatic evaluators sometimes have a bias toward longer, fuller answers. Put length in the rubric, as here, so it doesn't pass for quality.
• On facts, check yourself. AI is useful for spotting facts missing from the notes, but it can let some through. As in the article on verification, checking figures and names remains a human job.

Evolving the rubric

The first version of your rubric will have flaws. You'll see it when two answers get the same score even though one is clearly better: a criterion is missing. Or when a score seems unfair to you: the description is poorly worded. Fix it, run the same answers through again, and keep the rubric with your prompt.

Once you have a rubric that works, it also helps you evaluate what you buy. On a prompt from the marketplace, you can apply yours to the sample result shown on the listing before deciding.

To do now

Take a task you regularly hand to AI. Write three criteria that you always check without thinking about it, with what earns 0 and what earns 2. Apply them to three real answers without looking at the prompt that produced them. Then adjust the criteria where you hesitate, and keep the rubric next to the prompt.