Writing · essay
Measuring whether AI drafts are actually good
The best quality signal for AI drafts is how much survives human editing. How to measure it per prompt version, plus the guardrails that keep drafts safe to review.
Ask a team whether their AI drafts are good and you'll usually get an opinion. "They're pretty good." "They need a lot of editing." "Better than last month, I think." Those opinions might be right. They're still opinions.
If it isn't measured, you're guessing. And AI drafting is one of the easiest places to stop guessing, because the evidence is created every time a person reviews a draft. You just have to keep it.
I've built a human-approved reply-drafting workflow where AI prepares the draft, a person edits and approves it, and the system measures how much of the draft survived. That one signal has been more useful than any amount of reading drafts and forming impressions.
The signal: how much of the draft survives
The idea is simple. For every draft, you have two pieces of text:
- What the AI produced.
- What the human actually sent.
Compare them. If the sent version is nearly identical to the draft, the AI did the job. If the human rewrote most of it, the draft was little more than a prompt for the person to start over.
You don't need anything exotic to measure this. A standard text-similarity comparison between the two versions gives you a score for each draft. A character-level or word-level similarity ratio from a common diff library is enough to start. The exact method matters less than using the same one consistently, so scores are comparable over time.
A few practical notes:
- Normalize before comparing. Strip whitespace differences, signatures and quoted reply history, or they'll distort the score.
- Record rejections separately. A draft that was thrown out entirely is different from one that was heavily edited. Track both.
- Save the pair, not just the score. When you want to understand why scores dropped, you'll need to read the actual edits.
- Don't treat the score as truth. A high survival score means the reviewer accepted the draft. It doesn't guarantee the draft was correct. It measures fit, not accuracy.
Track it per prompt version
The score gets far more useful when you tag every draft with the version of the instructions that produced it.
Prompts change. You tighten the tone guidance, add examples, change what context gets passed in. Without versioning, you have no way to tell whether a change helped. With it, you can compare survival scores across versions and see the effect directly.
What to store with each draft:
| Field | Why |
|---|---|
| Prompt version | Compare quality across instruction changes |
| Draft text | The AI's output |
| Final text | What the human sent |
| Similarity score | The survival signal |
| Outcome | Approved as-is, edited, or rejected |
| Category of request | Quality often varies by type of message |
| Reviewer | Different reviewers edit differently |
Breaking results down by category is where the real insights show up. It's common for drafts to be strong on routine requests and weak on a specific kind of question. The average hides that. The breakdown tells you exactly where to improve the instructions, add context, or decide that a category shouldn't be drafted by AI at all.
Reviewer variation matters too. Some people edit heavily out of habit, others barely touch anything. Look at trends within the same reviewer before drawing conclusions across people.
Guardrails before the draft reaches a person
Measuring quality after the fact is half of it. The other half is keeping bad drafts from getting close to being sent. These are the guardrails I think every drafting workflow should have.
Fact sheets. Most wrong answers come from the model filling gaps with plausible guesses. Give it an approved fact sheet: the business details, policies, hours, services and standard answers it's allowed to use. Instruct it to use only those facts and to flag the draft when the answer isn't there, rather than improvising.
Linters for risky claims. Some content is dangerous no matter how good the rest of the draft is. Run automated checks on every draft before a human sees it, and flag anything that contains:
- Prices, fees or dollar amounts
- Dates, deadlines or timeframes
- Promises or guarantees ("we will," "guaranteed," "always")
- Outcomes or results that can't be promised
- Names, phone numbers or addresses that didn't come from the fact sheet or the original message
The linter doesn't need to be clever. Simple pattern matching catches a lot. Its job is to put a visible flag in front of the reviewer so those parts get read carefully, not skimmed.
Scrub personal data before it reaches the model. Only send the model what it needs to draft a good reply. Strip or mask personal details that aren't necessary: account numbers, dates of birth, medical or legal specifics, anything sensitive. This limits what could leak through logs or third-party processing, and it's a sensible default whatever your provider's data policy says.
Human review, always. Nothing goes out without a person approving it. The guardrails and the metrics exist to make that review fast and focused, not to replace it.
What to do with the numbers
The survival score is a tool for improvement, not a grade.
- When scores drop after a prompt change, roll back and read the edits to understand why.
- When one category keeps scoring low, look at what reviewers are changing. It's often missing context, not bad writing.
- When scores are consistently high in a category, that's a candidate for lighter review, but only if the risk of a mistake is low. A consistently accepted draft on a low-stakes topic is different from one containing prices or commitments.
- When reviewers start approving everything unchanged, check whether the drafts improved or the reviewers got tired. Spot-check approved drafts regularly.
This is the same principle behind the rest of how I use AI in operations: AI prepares, humans approve, and the system learns from the gap between the two. I wrote about the broader approach in AI as an operating layer, not a chatbot and about the review-queue mechanics in Scheduled AI agents with a human in the loop.
Measure the edit
Every time a person edits an AI draft, they're telling you exactly what the AI got wrong. Most teams throw that information away.
Keep the draft. Keep the final version. Compare them. Tag them with the prompt version. Add fact sheets, risky-claim checks and data scrubbing so the drafts are safe to review in the first place. Then let the numbers tell you whether things are getting better, instead of asking people how they feel about it.