How a practicing lawyer uses ChatGPT without citing fake cases

Back in January I was on a call with a practicing lawyer, and I asked him something I'd been curious about for a while. How much does he actually trust ChatGPT?

He'd been walking me through how he puts together requests for admissions. He tells it what he's after, lists the admissions he already has, and asks what he's missing. Then he said the part that stuck with me. "It's more of a brainstorming tool, the way that I use it, than it is a drafting tool." And why: "I'm afraid that it's going to draft something that isn't true."

He uses it all the time. He's also built his routine around one specific fear, which is a case in his filing that doesn't exist. They're worth stealing, whether you're a lawyer or anybody else who puts AI output in front of a client.

Why does ChatGPT make up case citations?

ChatGPT writes what a good answer usually looks like, and a citation that looks right is part of that, whether or not the case exists.

He walks in expecting it. He told me he goes into ChatGPT assuming it'll "plug in things that I want to see or want to hear but aren't necessarily true."

That's the right instinct, and I told him so. You can tell the AI not to hallucinate. By its nature it's still probabilistic. It's the same with people. You can tell a person to draft something and make sure there are no mistakes, and you still read it before it goes out. As I put it on the call, "You can't trust humans either."

He also explained why the lawyers around him are careful. His state bar publishes who's been disciplined and what they did. He's seen the stories about lawyers who filed AI-written cases, where opposing counsel stood up and told the judge, "These cases aren't real." That gets referred to the bar, and the bar hands out a public admonishment. In his words, "I think that has scared a lot of the more responsible attorneys away from just trusting ChatGPT."

How does he use ChatGPT without letting it invent cases?

He treats it like a paralegal. It brainstorms arguments, works out deadlines and drafts emails, and it's told to leave citations blank.

When he asks it something like what the main flaws are in the other side's arguments, he adds a line to the prompt: "make sure to give me no hallucinations." That line on its own does nothing. The model doesn't make things up on purpose, so asking it not to won't stop it. The line he adds right after is the one that works. He tells it that if it doesn't know the citation, leave it blank, or tell him to go find it. That gives the model a better option than guessing, and it turns a gap into something he can see.

So what comes back reads like a note from a junior person. There's case law in his state that says this, go do the research. And he goes and does the research, instead of letting the model fill in a case name that sounds about right.

He walked me through one where that pays off. He's suing somebody who doesn't have any money, so he wants to know how to reach the employer. He gives ChatGPT the facts and asks what causes of action could tie the employer in. It comes back with angles like respondeat superior, or negligent hiring if the person had been written up before. Some of those he'd use. Some don't apply. Either way, "it gives me a different angle." Then the case research is his.

He uses it the same way for deadlines. He gives it the docket, when the complaint was served, and asks the question he'd ask a paralegal: what's our deadline to respond before they can move for default? And for formal emails to other attorneys, he has it write the first version and then changes it around, "because I still don't like the way that it talks."

I suggested one more step on the call. Once he's found the cases himself, he can hand them back and say, "draft me a section... that uses this case law that I'm providing you." The model fits his cases to his facts, he reviews it and adds his own flair, and every citation in that section came from him.

How do you check a citation ChatGPT gives you?

Look up every single one in a real source, including the ones you think you know, and confirm it's the right rule from the right court.

He checks "every single case law that it tells me or that it refers me to." Even the black letter law he could recite. His example was a motion to dismiss. It gets decided within the four corners of the complaint, no outside evidence. He knows that standard cold. When ChatGPT hands it back with a rule number attached, he still puts the rule into Google to confirm it.

The reason is a subtle one. The principle can be right while the citation is wrong. "I don't know for sure that it isn't giving me... a federal rule of civil procedure instead of a state rule of civil procedure." That kind of mistake is easy to miss on a read-through.

That's how he thinks about the whole thing. The AI is the paralegal. His name is on the filing. He said the lawyers he spends time with all use it the same way, and that there are "probably some more lazy lawyers" doing it the trusting way.

Can you stop AI from making up cases in the first place?

Mostly, by having it pull cases from a real case-law database instead of from memory, and keeping a lawyer reviewing everything before it's filed.

This is where I came in on the call. I asked him if he'd heard of the Free Law Project. He hadn't. It's a nonprofit working to make case law open to anybody, and it has an API you can hook into. So instead of asking ChatGPT to remember a case, you build an agent where the language model queries a database that's actually designed to give you accurate information.

The example I gave him was a motion to dismiss. Somebody files one against your lawsuit. You hand the agent their arguments and ask it to find case law against them, and every case it hands back came out of the database. The Free Law Project runs CourtListener, and it also offers a citation lookup API that checks the citations in a block of text against its database. They describe it as a guardrail against hallucinated citations.

I've built this. On a system I put together for a law firm's litigation team, a research agent searches case law through CourtListener and Google Scholar, and what it brings back gets validated before the next step can use it. When I'd built prototypes that drafted responses automatically, there was always going to be a human proofreading at the end. "I would always recommend that."

His reaction was that it "sounds like something that would feel much more safe when drafting."

There's a limit to what a database check gives you, and I'll say it plainly. A lookup tells you the case exists. It doesn't tell you the case says what the draft claims it says, or that it's still good law. And CourtListener doesn't have every case, so a citation it can't find isn't automatically fake. Reading the case is still the lawyer's job, every time.

What this means for your business

You don't have to be a lawyer for this to apply. Any time AI writes something with a fact in it, like a client's renewal date, a candidate's last role or a number from a contract, you've got the same problem he has with case law. The model will fill in something that looks right.

So split it the way he does. Let the AI brainstorm and write the first draft. Make the facts come from a real source the AI looks up. Put a person at the end whose name is on it. I wrote more about where AI actually pays off and where it fails, and it comes down to the same thing: AI is only as accurate as the place it pulls from.

That's the reason Skaiward centralizes a business's data in one place, so the agents and the people doing the work are looking at the same records. The agent pulls from your actual client file instead of guessing. And if your files are the kind you'd never paste into public ChatGPT, that's a different setup, and I walked through what to settle before pricing a private one.