Why AI search returns horrible results, and how we fixed it

A law firm had an AI assistant sitting on top of years of its own case history, loaded up so the attorneys could ask a question and get back the parts of old matters that were relevant. It kept returning the wrong material, and I got brought in to fix it.

Another developer had built it. By the time I got there, the problem was easy to describe. You'd ask it something, and what came back wasn't the right material at all.

Swapping the model wouldn't have fixed it. The problem was underneath, so that's where we went.

How this kind of search actually works

This part matters for everything after it. A system like this doesn't read your whole archive every time you ask a question. Ahead of time, every document gets cut into pieces, and each piece gets stored in a database built for this kind of search. That project used one called Pinecone. When somebody asks a question, the system pulls back the handful of pieces that look closest to it and hands those to the AI to write an answer.

So the AI can only answer from whatever pieces come back. If the wrong pieces come back, you get a confident answer built on the wrong material.

That puts two things under the microscope. How the documents get cut up, and what's written on each piece so it can be found later.

How we cut the documents

There are two common ways to cut a document. You can cut it by length, every so many characters, with a little overlap between pieces. Or you can cut it by meaning, so each piece is one complete thought.

Both have problems. Cut by length and you slice an argument in half, and now neither half makes sense on its own. Cut by meaning and some legal reasoning runs long, way longer than a piece can be.

So we did both. We'd look at a piece and ask whether it was complete on its own. If it was, it stayed whole. If it was one of those long chains of reasoning that wouldn't fit, that's when we'd fall back to cutting by length. We kept the length cuts to a minimum, because those were the ones that led to more made-up answers.

The labels were the bigger fix

The labels mattered even more than the cuts. Every piece of a document can carry some information about itself. What topic it's about. What the document was called. That's what the search uses to tell one piece from another.

In that system the labels were incomplete. Topics were missing. Document names weren't consistent. There was a lot of detail that could have been on there and wasn't.

The other half of it is future-proofing the labels. You don't only label for the documents you have today. You have to think about what's going to get added six months from now and whether your labels will still sort it correctly. If you set up the labels around this year's files, next year's files show up and don't fit anywhere.

Splitting the library by job

There's a rule I build by here. When you have several AI agents in one system, each doing a different job, you don't give them all one giant pile to search through. You give each one its own library, built around the process it runs or the decision it has to make.

Split it that way and each agent is searching a smaller pile that's actually about its job, so what comes back fits the question it's answering.

Why this matters for every business with years of notes

That project was a law firm, but the same thing applies to every business that's been collecting notes on people for years. A recruiting firm with a decade of candidate notes, call summaries and placement history is sitting on the same kind of archive. So is a coaching program with years of client check-ins.

Everybody wants to point AI at that archive and ask it questions. Which candidates have we talked to before for a role like this? What did this client say the last time they were stuck? The AI part is the easy part to set up. Whether the answers are any good depends on how that history gets cut up and labeled before the AI ever sees it.

So when I scope one of these, that's where I spend a lot of the time. What's in the archive. What each record should be labeled with. And what's going to get added next year, so the labels still hold up when it does.