Extracting information from messy text

The secretary of a chess club is organizing a bus outing. He asked by email who was coming, and thirty-four replies have arrived, like these: "There'll be two of us, my wife is vegetarian," "I'm coming after all, sorry for the delay," "Maybe, I'll confirm Thursday," "It's a no for me, unless the bus leaves after 9 a.m." He needs a number of seats, a number of vegetarian meals, and the list of people who haven't given a clear answer.

The AI can do this kind of sorting, as long as you ask it for something other than "summarize these replies."

Say what to look for, field by field

An extraction is a table you fill in from a text. So start by drawing the table: one row per person, and one column for each piece of information you want. For each column, specify the allowed values, otherwise you get "yes," "Yes, gladly," and "present."

Then plan for the cases that don't fit the boxes. In the example, "maybe" and "no unless" exist. If you say nothing, the AI decides for you, silently.

Here are replies from members of my chess club to the invitation to a bus outing. One reply per block, separated by a blank line.

"""
[PASTE THE REPLIES]
"""

Build a table with one row per person and these columns:
- Name (as it appears in the reply; if absent, "unknown")
- Attendance: yes / no / uncertain
- Number of seats (1 by default; more if the person announces more)
- Vegetarian meals: number of meals, or 0
- Condition or remark, if the person sets one
- Quote: the exact words from the reply that justify the row

Rules:
- "Maybe" and any conditional reply are "uncertain," with the condition noted.
- If a person writes twice, keep the last message and flag the change in the remark.
- Invent nothing. If a piece of information isn't in the reply, leave the cell empty.
- After the table, give the total seats for the "yes" rows, the number of vegetarian meals, and the list of "uncertain" rows.

Require a quote for every row

The "Quote" column is the most useful tip in this article. For each row, the AI has to copy out the words that led it to its conclusion. You gain two things: you spot a doubtful interpretation right away by reading the quote next to the result, and the AI invents less, because a quote has to exist in the text. That doesn't make it infallible, since it can quote a sentence that exists and misunderstand it, but it's the quickest check there is.

The cases that trip up extraction

Negation and conditions. "I couldn't not come" is a textbook case. More common: "no, unless..." Plan a category for the in-between.

Corrections. When the same person sends two contradictory messages, the rule "the last one wins" has to be written down. If your messages aren't dated, the order you paste them in is what counts, so paste them in order.

Information about someone else. "My wife is vegetarian" concerns a person who didn't write in. Specify whether you want those cases counted, otherwise the wife's row will never exist.

Implicit values. If "there'll be two of us" means 2 seats, that's an inference. Say whether you accept it or would rather the AI leave the cell empty.

Date formats. To extract deadlines from a text, ask for a single format (MM/DD/YYYY) and add "keep the original text if the year is missing." Otherwise it will pick one.

Check before using the table

Out of thirty-four replies, the AI may miss one or two, especially in the middle of a long text. Three checks take five minutes:

1. Count the rows. There should be as many rows as replies. Ask the AI to tell you the number of blocks it read.
2. Reread the uncertain and conditional cases, which are also the easiest to misclassify.
3. Take five rows at random and compare them with the original. If you find a wrong one, reread the whole table.

For very large volumes, the guidance in the article on long documents applies: split into batches of a few dozen messages, and merge the tables afterward. The final calculation (total seats, meals) is also worth redoing in a spreadsheet. Long additions aren't a language model's strong suit, and the table lets you check them.

Which output format to choose

A text table can be copied into a spreadsheet without much trouble, and is enough for a one-off task. If you need to reuse the data elsewhere, for example in a CSV file or in a tool, ask for a stricter format: the article on JSON and CSV explains how to describe the fields and validate the result.

Think about personal data

Email replies contain names, and sometimes dietary requirements or health information revealed indirectly. Before pasting them into a tool, remove what isn't needed for your task (signatures, addresses, phone numbers), and reread what the article on privacy says. For a club or association, also look at what your internal rules provide.

To test on your own case

Choose a batch of messages you have to go through by hand this week. First write out the columns and the rules for ambiguous cases. Run the extraction on just ten messages, compare with your own tally, fix the rules that went astray, then process the rest.