AI data extraction from PDFs: why I make AI pull the facts into JSON before it judges anything

In October a developer on my team was fighting with an AI agent we'd set up for a client. Its job was to bring in new leads, sort each one onto the right list once a deal was confirmed, and create a buyer record when the sale got going. Three jobs, one agent. It kept getting things a little wrong, and it was hard to tell which part was the problem.

On our check-in he walked me through what he'd figured out. "Instead of having one agent that does everything, because it's a few things that it's trying to do, I think that's where it's getting kind of messy." So he split it. The first agent only brings in the lead information and puts it on the open inquiries list. A second agent gets handed the lead when the deal is confirmed and routes it. And the third step, he said, might not need AI at all. "Maybe it's just an automation."

That's the whole idea of this post. People and AI agents are working in the same files every day now, and the businesses getting good results are the ones giving each of them a distinct job. The AI does one thing per step. A person owns the calls that need judgment. And the step I put first, almost every time, is getting the facts out of the document before anyone, human or AI, decides what they mean.

Why does AI get messy when one agent does several jobs?

Every job you add puts more instructions in front of the model at once, and when the result comes back wrong, you can't tell which job broke.

An agent reads everything you gave it every time it runs. The instructions for scraping, the rules for routing, the steps for creating a buyer, and the lead itself all sit in the same context. The more that's in there, the more chances the model has to blend them. A rule meant for routing bleeds into how it records the lead. That's what "messy" looked like on that client's lists.

I said the same thing to an advisory firm back in April, when they asked about handing work to a general-purpose AI assistant. "If you want to do that repeatedly in a really reliable fashion, you need to build a dedicated process." AI works on probability. Change one word in the instructions and you can get a different result from the same process.

The bigger problem shows up when you go to fix it. Say the buyer record comes out wrong. Was it the scrape, the routing or the record step? With one agent doing all three, you go hunting. With one job per step, you open the output of each step and see where it went sideways. And notice that the third step ended up as a plain automation. Once the jobs were separated, it was obvious that one of them never needed a model.

What's the difference between extracting data and analyzing it?

Extraction copies what a document says into labeled fields. Analysis decides what those facts mean. They're two different jobs, and the second one is where all the nuance lives.

The clearest place I've seen this is the litigation system I built for a law firm, the one that drafts responses to motions to dismiss. Documents land in a shared drive and the text gets pulled out. Then an AI step works through what was filed and pulls out the arguments and the facts. When a friend asked me in August how it worked, I described the knowledge base behind that step as the thing that "would help shape the particular language and its strategy for extracting arguments and facts from the docket." Case law research is a separate agent. An attorney reviews the draft at the end.

Court work is a good example because the analysis is so nuanced. Whether an argument holds up depends on the facts, the court, the case law, and sometimes one word in the complaint. If the same step that's reading the complaint is also deciding whether a claim survives, a misread date or a missed count turns into a well-reasoned wrong answer. It reads fine. That's the dangerous part. When extraction runs first, someone can check the dates and claims against the document before any of it gets argued.

The lawyer I wrote about in the post on using ChatGPT without fake cases works the same way by instinct. He lets ChatGPT help him think, and he refuses to let it be the source of a citation. The facts come from somewhere you can check, and the thinking happens on top of them.

You see the same split in plain documents too. On a medical billing review project, most of a 15-page opinion was facts mapped straight from the records into a template. Only one section needed AI, the part where facts get applied to the rules. Everything else was extraction.

Why should AI hand its extraction off as JSON?

Because JSON labels every fact in a way software can read, so a database, a simple check or the next agent can pick it up without guessing what anything means.

If you've never seen it, JSON is plain text where every piece of information gets a label, like "date_filed": "2026-03-04", so a program knows exactly what each value is.

When that same friend asked what guardrails I'd put on the legal agents, my answer started here: "For the most part, there was a JSON schema validation that happened, and I modularized a lot of the work." The schema is the list of fields that have to come back and the shape each one has to take. A date has to be a date. A case number can't be empty. Because the extraction comes back labeled, the yes-or-no check after every AI step can do its job. If the fields are all there in the right shape, the work moves on. If not, it goes back through.

Here's roughly what one of those extractions looks like.

Look at date_served. It says null, which means empty, because the service date wasn't in the document. The extraction step is told that if something isn't there, it leaves the field empty. It doesn't guess. That empty field is a flag a person can see and go fill in. The response deadline gets worked out later, by plain business logic, from the service date. That's calculator work, and a model shouldn't be doing it.

Once the facts are labeled, they can go two places. They can go into a database, the same records your team already works from. Or they can go to the next agent, which now reads a handful of clean fields instead of the whole filing. The advisory firm in April had a version of this. We talked about what happens when a client sends back a filled-out PDF application. "That is where we would use AI to extract all the data points," I told them, and those data points land in the same table as every other client. The step after that reads the table, never the PDF.

Where does the person come in?

The person owns the decisions that carry risk or a name: the nuanced call, the approval on what goes out, and when the AI runs at all.

On the April call, when we got to how that PDF process would kick off, the advisor pushed back. "I almost want to control when the robot decides to do it." He wanted to key the client into his CRM and press a button. That's a distinct responsibility, and it belongs to him. The AI does the extraction and prepares the draft. He decides when it runs.

In the legal system, the attorney owns the end. Every draft goes past a person with a license before it goes anywhere. The AI's job is to get them a first draft built on facts they can check, fast.

I said something on that April call that I still believe. "We are called to discern things. We are called to make decisions, and an AI is going to do it based on probability." AI is very good at the math part, better than a lot of people on some tasks. What to do with the answer is still a human job. In practice, that means before I build anything, I write down which steps are AI, which are plain rules, and which belong to a person, and I don't let them share a step.

What this means for your business

You don't need a courtroom for this to matter. A renewal date in a coaching agreement, the last job title on a resume, a total on an invoice, the week-six check-in on a client's program. All of it is extraction first. On a coaching platform we're building right now, each client's check-in call gets tied to their enrollment by week. The next step is an AI briefing before the coach's next call, and that briefing reads labeled check-ins instead of raw recordings.

Here's something you can try this week. Take one thing you ask AI to do in a single prompt today. Write down every job hiding inside it. Draw a line after the first job that only copies facts out. Have that step hand back labeled fields, then check them. Then let the next step do the thinking, and decide which person signs off at the end.

This is the reason Skaiward centralizes a service business's data. The extracted facts need somewhere to land where your AI agents and your people are working from the same records: a human-agent operating system. The AI reads from it, writes back to it, and a person can always open it and see what the AI saw.