Only one section of the document actually needed AI

Last fall I was the developer on a project with a small legal team. They were building software that reviewed medical billing before it went out the door. For every case, an attorney read the patient records and wrote a legal opinion on whether the work was documented well enough to bill. Each opinion ran about fifteen pages.

The plan everybody had in their head, mine included, was that AI would write those opinions. Records go in, fifteen pages come out, and an attorney reviews it and signs.

Then one Monday the lead attorney got on our call. He'd spent the weekend writing five of these opinions by hand, and he walked us through one section by section.

What he noticed doing it by hand

Somewhere in there he said the thing that changed the project. The records looked about 90% identical from one patient to the next: past medical history, the evaluation, prior treatments, current medications. They had the same fields in the same order every time.

So most of the opinion didn't need a model. The facts could be mapped straight from the records into a template. The legal standards section cited the same law every time, so it could be written once. The only place we needed AI was the section where this patient's facts get applied to those standards and someone decides whether the claim holds up. In his words: "We can do mapping on everything except that Roman numeral five."

It took me a minute to see how much that changed. We'd been treating all fifteen pages as one big AI job. After that call, the AI job was one section. The rest was getting the right fields out of the records and into the right places.

Why the smaller job is easier to trust

When a model writes fifteen pages, the reviewer has to read all fifteen before trusting any of it. When it writes one section, and everything around that section came straight from the records, the reviewer knows exactly where to look.

It also changes how you get it right. He told us he was adjusting his prompt after every opinion, and he expected to keep doing that for the first 30 or 40 cases, because every case was a little different. That's a lot of rounds. Each round goes faster when you're tuning one section instead of a whole document.

A mapped field comes out the same way every time, so you check it once and move on.

Who should be tuning it

That same morning he'd built a custom GPT to test his prompt himself, because he couldn't wait on us. I understood why. A medical group with 200 doctors had just sent over 30 patient files, and he needed to start producing opinions.

On that call I suggested giving him a place where he could edit the agent directly, with the app calling that same agent, so whatever he changed showed up in the product. Looking back, I'd push for that on day one. The person who knows what a good opinion looks like should tune it in the place the software actually uses, instead of handing a developer a long prompt at the end.

Try this before you build anything

If you want AI to produce a document for your business, try this first:

1. 1.Put three or four finished versions side by side.
2. 2.Mark everything that was copied from somewhere else, like a form, a profile, or a record.
3. 3.Mark everything that's the same text every time.
4. 4.Whatever's left is the AI part. On that project it was one section out of fifteen pages.

If you run a recruiting desk, that document might be the write-up you send a client about a candidate. If you run a coaching program, it might be the onboarding notes or the check-in summary. Some of it comes from a form someone filled out. Some of it is the same paragraph every time. Somewhere in there is the part where you're actually making a call about this person. Hand that part to a model, and have a person read what it wrote.

We only saw the 90% because he wrote five opinions by hand first. If you can, do the same before you commission anything.