AI & Data Privacy
General AI vs. Legal-Specific AI: Why Structure and Consistency Matter for Case Analysis
Ask a general model to summarise a submission twice and you get two different documents. That is fine for a briefing note and disqualifying for a case report.
General-purpose models are extraordinarily good at producing text that reads like a competent summary. That is precisely what makes them risky for case work, because a case report is not judged on how it reads. It is judged on whether every statement in it can be traced back to a document.
Key takeaways
- Free-form generation optimises for plausibility. Extraction optimises for coverage and traceability. They are different jobs.
- Non-determinism is the quiet problem: the same input twice gives different output, so nothing is reproducible.
- Summarisation silently drops what it judges unimportant. In a dispute, that judgement is the reviewer's to make.
- The only output you can use without re-reading the file is output that cites where it came from.
Four failure modes, in order of how much they cost
Fabrication
A general model asked for authority will produce authority. Where none exists that fits, it can generate citations, clause numbers, or exhibit references that look correct and are not. Courts in several jurisdictions have now sanctioned filings containing invented citations. The defence is not a better prompt; it is a system that can only return content traceable to a supplied document.
Silent omission
More damaging than fabrication, and much harder to notice. Ask for a summary of a 300-page bundle and you get a summary — you do not get a list of what was left out. If the omitted paragraph is the notice provision, nothing in the output tells you it is missing.
Non-determinism
Run the same document through twice and the two outputs differ: different emphasis, different items, sometimes different dates. For a research assistant that is tolerable. For a case file it means two reviewers working from the same source reach different chronologies, and neither can reconstruct why.
Unverifiable output
A fluent paragraph with no references transfers the whole verification burden back to you. If you have to re-read the source to confirm each assertion, the tool has reorganised your work rather than reduced it.
What structure actually changes
A legal-specific system constrains the problem before the model sees it. Rather than “summarise this”, the task becomes “for this document type, populate these fields, and attach a source reference to each value”. Three things follow from that constraint:
- Comparability. Every dispute submission yields the same categories, so two documents can be set side by side, and a gap in one is visible as an empty field rather than as an absence you have to notice.
- Reviewability. A field with a page reference can be checked in seconds. A paragraph of prose cannot be checked at all without returning to the bundle.
- Accumulation. Structured output can be merged across a hundred documents into one chronology and one evidence index. Prose summaries cannot be merged; they can only be stacked.
This is the same discipline that makes a case report usable, applied one step earlier. The eight-part report framework is, in effect, a description of what the extraction layer should be producing.
How to test a tool before you trust it
Vendor demos run on clean documents. Run your own pilot on a real bundle, and design it to catch the four failure modes above:
- 01Run the same document three times. Diff the outputs. Any variance in dates, parties, or figures tells you what to expect at volume.
- 02Plant a fact. Take a document you know well and check whether a material detail buried deep in it appears in the output. This is the omission test, and most tools fail it.
- 03Check every reference. Pick ten extracted facts at random and follow each to its cited page or timestamp. If references are absent or approximate, verification cost stays with you.
- 04Feed it something adversarial. A poor scan, a bilingual document, an unsigned draft, a document that contradicts another in the same bundle. Case files are not clean.
- 05Ask what it does when it does not know. A tool that returns an empty field and a reviewer note is more useful than one that always produces something.
The tool that scores well on those five is doing a different job from a chat window, whatever the underlying model. And the confidentiality question sits underneath all of it — what the rules require before any case material goes near an AI tool.
Keep reading
Workflow
How to Structure a Legal Case Report: A Step-by-Step Framework
Most case reports are rewritten not because the analysis was wrong, but because the structure was decided while writing. Here is a framework that settles it beforehand.
Workflow
5 Ways Paralegals Can Cut Case File Review Time Without Cutting Corners
Speed in file review comes from doing fewer passes over the same pages, not from reading faster. Five techniques that hold up when the file is 4,000 pages.