# Build a legal document conflict detector > Source: > Published: 2026-11-12 01:55:05+00:00 *This tutorial is aimed at intermediate Python or JavaScript users who understand LLMs. Try our [Build a legal RAG app that won’t be held in contempt](https://isaacus.com/blog/build-a-legal-rag-app-that-wont-be-held-in-contempt) guide for something simpler.* ## tl;dr This guide walks through building an app that finds conflicts between two legal documents: provisions that conflict with each other, and provisions one document has that the other doesn’t. Every finding is grounded in the text of both documents. The crux of the work is handled by Isaacus’ [enrichment](https://isaacus.com/docs/models/introduction#enrichment), [embedding](https://isaacus.com/docs/models/introduction#embedding) and [universal classification](https://isaacus.com/docs/models/introduction#universal-classification) models. A generative model is used to summarize **findings**. While we focus on the backend, the entire demo app below is available to run in our [open-source cookbook GitHub repo](https://github.com/isaacus-dev/cookbooks/tree/main/cookbooks/conflict-detector). The cookbook is available only in Python. ## The problem: finding conflict between provisions Comparisons have been a part of legal work for a long time: manually reading line by line, while tracking a checklist or a playbook, all to create issue lists for review. Extraction and redlining grew easier with tools like Microsoft Word’s change tracking, Kira, and contract lifecycle management platforms… but that was still fairly manual. And slow. Then gen AI emerged, with all sorts of models being integrated into legal review tooling. They provided some speed on the document processing side – great – but came with an Achilles’ heel: recall. Point a generative model at two documents and ask for conflicts: it’ll find some, but it won’t do a comprehensive pairwise comparison, it doesn’t scale to long documents, and it can miss obvious conflicts. Hallucinations are still a problem on top of that. When introducing automation, there are a few things to take into consideration, which include: - What constitutes a difference between two provisions in separate documents? - How large is the volume of differences? - Which of the differences found are actually significant? - How can human understanding of the differences become more efficient? We can take these questions into consideration while improving recall and precision by relegating gen AI models to what they’re actually good at: summarizing and editing, rather than reviewing. For reviewing, we’re going to replace the void they left behind with models that don’t hallucinate: [Isaacus’ models](https://isaacus.com/docs/models/introduction). The generative model only comes in at the very end, so any hallucinations are confined to the summaries, recommendations, and rewrites. *Side note: during this article’s drafting, I was fresh out of an Odyssey IMAX screening. Every time I mention ‘provisions’, know I’m thinking of the provisions Odysseus would’ve had packed for a voyage. And probably hungry.* ## Process Fundamentally, we have an incoming **input document** we want to compare to an existing **reference document** we already have. We divide them up into provisions, and work through each pair making decisions about them. We call all the results **findings**, of which there are two types: - **Deviation** : two provisions that cover the same topic but don’t have the same legal effect. - **Gap** : a provision in one document with no agreement or difference in the other document. For the purposes of this demo, provisions that agree with each other are ignored; this means we’re only flagging items that may provide a source of conflict. The general process is as follows: 1. Documents are prepared beforehand, i.e. cleaned and then normalized. 2. The documents are split into segments with [Kanon 2 Enricher](https://isaacus.com/docs/models/introduction#enrichment) . 3. The segments are [embedded](https://isaacus.com/blog/what-are-vector-embeddings) with[Kanon 2 Embedder](https://isaacus.com/docs/models/introduction#embedding) . 4. The segments get filtered with [Kanon Universal Classifier](https://isaacus.com/docs/models/introduction#universal-classification) to leave only relevant provisions (dropping things like footnotes). 5. Every provision in an input document gets matched against every provision in a reference document, and the closest pairs are shortlisted. 6. Each shortlisted pair is scored for conflict with the universal classifier and [Kanon 2 Reranker](https://isaacus.com/docs/models/introduction#reranking) . 7. The pairs are then sorted into deviations and gaps. Our app will be a bit more involved. The code is split into three parts: - Document indexing - Document analysis - Generation (optional) We’ll focus more on the first two as they cover finding deviations and gaps. *Document indexing* runs once per document: 1. **Normalize** the documents. Documents are converted into markdown and cleaned (encoding, stray characters, and whitespace). It’s basic pre-processing. 2. **Enrich** the text into provisions (clauses). 3. **Embed** every clause. We also **cache** the results to test values for future steps instead of making more API calls. *Document analysis* runs once per pair of documents (an input document and a reference document): 1. **Read** both documents through*document indexing* . 2. **Filter** out what isn’t a provision, such as footnotes, citations and signature blocks, using the enricher’s labels and the classifier. Definition clauses skip the classifier. 3. **Match** every input clause against every reference clause and shortlist the closest. 4. **Score** each shortlisted pair for conflict with the classifier and the reranker. 5. **Compare** provision pairs and decide which are deviations and which clauses are gaps, in document order. Definition clauses are ignored. 6. **Assemble** the report, then run the entire backend. A pair that hasn’t changed since its last run reuses the saved report. The third part, *generation*, is optional. It writes a summary of each finding using a gen AI model: a sentence or two on the differences between the two provisions. It also writes a recommendation for each finding about whether to keep, remove, or rewrite it. I briefly cover how it works, but we’ll just be dropping in the whole file to see how it works with everything else. ## Prerequisites - Python 3.11+ and [uv](https://docs.astral.sh/uv/getting-started/installation/) . - An [Isaacus account](https://isaacus.com/docs/quickstart#1-set-up-your-account) and[API key](https://isaacus.com/docs/quickstart#1-set-up-your-account) . - Sample documents. I’ve prepared a set in the repo’s [`samples`](https://github.com/isaacus-dev/cookbooks/tree/main/cookbooks/conflict-detector/samples) folder, already converted to markdown. Your own documents can be plain text: the normalizer converts them. - Any of them can be a **reference document** (the document to check against) or an**input document** (the document to check). - Any of them can be a This tutorial compares two different agreements: the Common Paper Cloud Service Agreement as the reference document, and Mistral AI’s Terms of Use as the input document. ## Setup 1. Clone the cookbooks repo. 1. Create a `.env` file with your API key in that folder. ## A. Index a document Before we can compare documents for conflicts, we first have to identify the provisions in them. Every document goes through the same stages: load it, enrich it into clauses, embed those, and then cache everything. ### Stage 1: clean the documents Legal documents can be messy with all sorts of different patterns and layouts that need particular attention, such as tables. The sample files for this demo were fairly straightforward so general ‘cleaning’ fixes the rest: broken encoding, curly quotes, non-breaking spaces, stray control characters, and trailing whitespace. Cleaning them up helps us get better results from the models and makes them look prettier (💅) when we present them. Isaacus’ models were trained on markdown, so typical layout markers, such as headings and text formatting, are digested appropriately. Our sample files have already been converted to markdown. Any other text file goes through [`text2markdown`](https://github.com/isaacus-dev/text2markdown), our library that turns plain text into markdown. We’ll focus on the *normalize* step first. It takes a document’s text and returns it as clean markdown. This section uses the following dependencies: - [`ftfy`](https://pypi.org/project/ftfy/) : repairs text that was decoded with the wrong encoding, and straightens curly quotes (its`uncurl_quotes` fix, which is on by default) so`"Agreement"` and`“Agreement”` read the same. - [`text2markdown`](https://github.com/isaacus-dev/text2markdown) : turns a plain-text document into markdown, for any file that isn’t markdown already. 1. Add the cleaner and the converter. 1. Test the cleaner by printing the first few lines: - Curly quotes are straightened, and trailing whitespace is gone. - The markdown is still there: the `#` headings and the section numbers are untouched. ### Stage 2: enrich into clauses We compare documents provision by provision, so first we need to find where each provision starts and ends. Kanon 2 Enricher turns a wall of text into structured segments: headings, containers, clauses, definition clauses, and list items. Segments can nest, so this stage removes nested text from each one and gives it a heading. This section uses the following dependencies: - [`isaacus`](https://docs.isaacus.com/) : the official SDK. - [`python-dotenv`](https://pypi.org/project/python-dotenv/) : loads` ISAACUS_API_KEY` out of`.env` , so the key never lands in your code. - [`msgspec`](https://jcristharif.com/msgspec/) : typed records for clauses, and later for indexes and pair scores. 1. Start *document indexing* with the imports, the paths, the clause record and the loader. - `text` is what we embed and classify. - `start` and`end` are character offsets in the document’s text, so a report can point at the exact words rather than printing them again. - `type` and`category` are the enricher’s labels for the segment.`category` is how we distinguish body text from front matter, back matter and annotations. - `load_text()` reads a document from disk. It hands the text to`normalize_text()` . 1. Create the API client. Every function that calls a model takes it as its first argument, so importing a module never connects to anything on its own. 1. Add the enricher call, the section list and the heading resolver. - `resolve_headings()` gives each clause the title of its nearest titled ancestor, or its own. Where the enricher finds no title at all, the fallback is the nearest markdown heading above the clause, without its number or bold markers. Headings only label findings. In the interactive demo, they label the cards. 1. Add the clause extractor. - The enricher returns segments nested inside each other: a section, the clauses beneath it, and the list items beneath those. Each segment’s span covers its children’s text too, so the same words would be counted more than once. `extract_clauses()` cuts the children’s spans out, so each segment keeps only its own text. - The enricher also marks each segment’s own label: its type name (“Section”), its code (“4.2”) and its title. `has_body()` drops a piece with nothing left outside those, such as a bare heading. - The enricher doesn’t mark every heading’s title or number, so a markdown heading line (one starting with `#` ) counts as a label too. Without it, ”## Definitions” would be a clause of its own, which we don’t want. - `defines_term()` marks a clause as a definition when it opens with a term the enricher found, such as in`"Fees" means…` . 2. The enricher returns segments nested inside each other: a section, the clauses beneath it, and the list items beneath those. Each segment’s span covers its children’s text too, so the same words would be counted more than once. 1. This is the first call that spends credits, so it runs on one document. Test the enricher by printing the first few clauses: - There should be fewer clauses than segments: sections that only hold other clauses, and bare headings, are dropped. If the numbers match, nothing was dropped. - Each printed clause should show its heading in brackets. If every heading is `None` ,`resolve_headings()` isn’t finding them. ### Stage 3: embed and cache the clauses Time to convert text into embeddings to be sorted, searched and compared. We’re using a semantic embedding model, which lets us measure how close two clauses are in meaning and not just wording. This section uses one dependency: [`numpy`](https://numpy.org/), which holds the embeddings as one matrix, compares them with a dot product, and saves them to disk as an `.npy` file. 1. Add the model, the batch size and the normalizer. - The embedder turns each clause into a vector. Clauses about the same thing end up ‘close’ to each other, and cosine similarity scores that closeness. - `unit_normalize()` scales every vector to length 1. Cosine similarity is the**dot product** (multiply matching numbers and add them up) divided by both vectors’ lengths, so when every length is 1, it’s just the dot product. Because the vectors are scaled before they’re saved, comparing every clause in one document with every clause in another takes a single matrix multiplication. 1. Test the embeddings by printing the clauses closest to the payment clause: - Every clause should get one vector, so the two counts match. - The closest clauses should be about the same thing: here, fees, taxes and invoicing. - The clause itself comes first with a score of `1.00` . Anything else and`unit_normalize()` isn’t being applied. #### Cache the index Unlike Penelope’s loom situation, we don’t want to have to repeat anything more than necessary. Enriching and embedding a document only depends on its text, so the result is saved under a hash of that text, not under the file’s name or path. Rename or move a document and it’s still found in the cache; only a change to its text sends it back to the models. 1. Add the cache: the index record, a hash function, a reader, a writer, the document records, and the function that ties the stages together. - Bump `PIPELINE_VERSION` by hand when you change how clauses are extracted. The text hasn’t changed, so nothing else would. - An index is five files: `embeddings.npy` ,`propositions.json` , the cleaned text, the sections and a manifest. Row`j` of the embeddings belongs to proposition`j` , so`read_index()` refuses the pair if their counts differ. 2. Bump 1. The earlier tests called the enricher directly, so they spent credits on every run; this goes through the cache. Test the cache by indexing the same document twice and timing each run: - The warm run should take a fraction of a second and report the same count. If both runs take the same time, the cache is never being read: check `results/cache/` has a folder in it. 1. Index the input document. It needs to run through the same process as the reference in order to be compared to it. Test the index by printing how many propositions each document has: Neither count should be zero. ## B. Analyze documents Everything below is part of *document analysis*. We’re essentially going to build a search engine, but for conflicts. We’ve already embedded our provisions, so this is where we compare them and classify conflicts. Penelope gets suitors; we’d probably just get a suit. At least we can also get rid of it in minutes instead of two decades – much like how it feels to work on some documents (or blog posts 👀). ### Stage 1: filter and match This is the bulk of the conflict search engine. Scoring every clause against every other clause would take thousands of model calls, so we narrow them down first. Footnotes, citations and signature blocks aren’t provisions, so they’re dropped. The rest are matched by embedding similarity, with no extra model calls. The closest pairs go on to the classifier and reranker in batches. Think of it as cutting Penelope’s suitors before the feast. 1. Add the classifier call, the filter, the matrix and the shortlist. - The filter has two parts. The enricher already labels front matter, back matter and annotations, so those are filtered immediately. The classifier scores every other clause against one [IQL](https://docs.isaacus.com/iql/introduction) statement, roughly “this is a footnote, citation or signature, not a provision”, and sets aside anything at or above`FILTER_THRESHOLD` . - Definitions are never filtered. The classifier scores them inconsistently, and a changed definition can matter as much as any obligation. Stage 2 weighs them down instead. - `shortlist_pairs()` keeps a pair if it scores at least`SHORTLIST_MIN_RELEVANCE` and is one of the top`SHORTLIST_TOP_K` matches for either clause. Ranking from both sides means every reference clause gets its own best matches, so one missing from the input can still be reported. 2. The filter has two parts. The enricher already labels front matter, back matter and annotations, so those are filtered immediately. The classifier scores every other clause against one 1. Both documents come out of the cache; the filter is the part that calls the classifier. Test the filter by counting how many clauses it drops and how many candidate pairs are left: - About a tenth of the input is filtered: headings, cross-references and other text that isn’t a provision. The reference is almost all provisions, so only one clause goes. If nearly everything is filtered, `FILTER_THRESHOLD` is too low. - Candidate pairs are what go on to the classifier and reranker, about 200 here. If there are thousands, raise `SHORTLIST_MIN_RELEVANCE` or lower`SHORTLIST_TOP_K` . - Your counts may differ by one or two between runs, because the embedder’s output varies slightly. ### Stage 2: score One of the few times it’s good to be judgy: the universal classifier and the reranker earning their keep. In this stage we produce a score that quantifies how much two provisions conflict. Every shortlisted pair gets two numbers: - Combined and calibrated into 1. Add the questions, the weights and the scorer. - `match_matrix()` turns the embedding similarity between two clauses into . A similarity of 0.6 can be the best match in one pair of documents and an ordinary one in another, so the raw score isn’t enough. Instead, a pair scores high only when each clause is clearly the other’s closest match. A clause with no real counterpart has no clear closest match, so all its pairs score low. - Four questions decide whether a difference matters. `empty` (is it standard wording, a heading, a signature or a definition?) and`money` (does it state an amount, a percentage or a cap?) are asked of each clause.`care` and`agree` compare the two clauses directly. - `materiality` combines the four answers into one score. A clause`empty` flags, such as standard wording found in most contracts or a signature block, pulls it down hard; clauses that agree, or that a party wouldn’t care about, pull it down a little; an amount pushes it up.`differs` turns that score into a probability between 0 and 1: . - `conflict` combines and into . Multiplying them is too cautious, so the combination is fitted: 0.73 means about a 73% chance of a real difference. - The `SAME_*` ,`DIFFERS_*` ,`CONFLICT_*` and`*_WEIGHT` constants are fitted on labeled clause pairs. With the questions, they’re what you’d retune. - `CLASSIFY_QUERY_MAX_CHARS` trims the clause sent with`care` . Each distinct clause is sent once, however many pairs it’s in. Scoring is the stage that costs money; you’ll see its results in the Stage 4 run. ### Stage 3: compare Stage 2 scored every shortlisted pair. This stage decides which of them are deviations and which clauses are gaps to complete our actual findings. 1. Add the thresholds, the row helpers, the partition and the two roll-ups. - `partition()` first drops any pair that isn’t about the same topic (below`SAME_THRESHOLD` ). Then every clause, on both sides, picks the one counterpart it matches best. If that pair’s reaches`CONFLICT_THRESHOLD` , it’s a deviation. - Only the best match counts. Otherwise a clause would get flagged against every vaguely similar clause nearby. No drowning in findings, thanks. - Picking from both sides matters because documents don’t split things up the same way. If the reference spreads one input clause across four, all four pick it, so it ends up as one row with four matches. - By the end, every clause has one of three statuses: - `deviation` : its best match conflicts with it. - `covered` : it has a match, and they don’t conflict. - `gap` : nothing in the other document covers the same thing. - `roll_up_gaps()` turns gaps into findings for each side: what the input document dropped, and what it added. Filtered clauses never become gaps, and`covered` clauses are ignored; they agree, so there’s nothing to flag. - A definition with nothing to match isn’t a gap. Documents define different terms, or define them in different places: the input defines “Effective Date”, while the reference just uses it because its cover page sets the date. If a term actually matters, the clauses that use it will turn up as findings anyway. ### Stage 4: report and run The last stage puts every finding into one report and saves it under the pair it belongs to. Then we run the whole pipeline. 1. Add the report, the writer, and the check that reuses a saved run. - `report.json` holds the findings.`pairs.json` holds every shortlisted pair behind them, with all four answers and the three probabilities. Want to try a different threshold? You can see what would change without paying for the models again. - `params.json` is the paper trail: the models, the thresholds, the wording of every question, and the pipeline version. If two runs disagree, look here first. - `saved_report_if_current()` reuses a saved run as long as the settings and both documents’ text haven’t changed, so running the same pair twice doesn’t hit the models again. The exception is a plain-text document: it still gets converted to markdown on every run, since that’s how we get the text to compare. 1. Add the pipeline. `run_comparison()` runs every stage in order for one pair, and`use_cache=False` runs everything from scratch. The`STAGE:` lines follow the overview, with Report split into assembling and saving. 1. Run it against the sample pair. This is the first time the whole pipeline runs end to end. The report’s `summaries` is`None` for now: part C fills it in. - We get a line per stage, then a new folder under `results/reports/` holding`report.json` ,`pairs.json` and`params.json` , and a record of each document under`results/documents/` . - 56 deviations in 54 rows (two input clauses have two counterparts each), and many more gaps, is about right for two different agreements that overlap only in part. - Hundreds of deviations on a pair that should mostly agree means `CONFLICT_THRESHOLD` is too low; none on a pair you know disagrees means it’s too high. - Read the pairs, not the count. Each deviation’s `matches` in`report.json` should say different things about the same point; expect a few that only rename a defined term. - Each row’s `heading` should be a clause a reviewer could find in the document. - Your counts may differ by one or two, and scores by a hundredth or so, between runs. Swap in any other document from `samples/` to check a new pair. Running the same pair again with nothing changed reuses the saved run and costs nothing; pass `use_cache=False` to start over. ## C. Generation (optional) We can make our lives easier by having a generative model summarize each finding and make recommendations for us. *Document analysis* creates a saved analysis report. This optional step adds summaries and recommendations to that same report file. Here’s a deviation from the sample pair: - **Cloud Service Agreement (reference):** “Except as provided in Section 8.4 (Exceptions), each party’s total cumulative liability for all claims arising out of or relating to this Agreement will not be more than the General Cap Amount.” - **Mistral AI Terms of Use (input):** “To the extent permitted by law, the total aggregate liability of Mistral AI in respect of any Losses incurred by the Customer under or in relation to this Agreement will not exceed, in the aggregate, the lower of (i) the amount of the Fees paid or payable by the Customer in the twelve (12) calendar months preceding the date on which the first such event or events occurred or (ii) 10.000 euros.” of 0.68 puts them on the same matter; with of 0.99, is 0.81, over `CONFLICT_THRESHOLD`. Our optional third part, *generation*, turns those results into something more user-friendly. It’s easier to read a summary of what the values mean than to decipher the meaning after reading two pairs of provisions. These models are generative, so a watchful eye is certainly needed in case of hallucination and other errors. I’ve found generative models giving me different results even with similar prompts, so it’s worth trying a few for preferences’ sake. I was working with Opus and Sonnet; you should customize the prompts for models you prefer to use. The code for *generation* is long because it supports several LLM providers and adds to the report. I’ll summarize the quirks of the prompts instead. There are two prompts. Their placeholders are filled with the documents, parties, and findings at run time. `PARTIES_PROMPT` names the parties once per pair, from the input document only. Legislation or a policy has no signing parties, so it lists the groups the document binds or protects instead, such as “employers” and “employees”. `RECOMMEND_PROMPT` takes the findings in batches. For each one, it writes the summaries, a comment, an action and reason for each party, and a rewrite if appropriate. Some of the quirks: - It writes a finding’s summaries, comment, actions and rewrite in one answer, so they don’t contradict each other. - Every call includes both full documents. When summaries were written from the two passages alone, they often said the other document lacked something that was in a different clause. - The model can’t recommend `remove` for a missing clause, or for a passage that another clause refers to. Before this rule, it deleted subclauses that other clauses cite by number. - A rewrite is only a suggestion until the reviewer accepts it, so the model judges every finding against the input document as written. A rewrite goes no further than the reference position, and keeps the input document’s numbering, list markers and line breaks so the redline shows only the real change. An added passage can get no rewrite (`null` ) when the reference document has no such passage, so the model doesn’t invent a narrower clause. 1. You can have a look at the [generation file](https://github.com/isaacus-dev/cookbooks/blob/main/cookbooks/conflict-detector/generation.py) containing the prompts above if you like. Import it and run it on the pair from*document analysis* . Set`model` to the model you want, and`api_key` to its key. The default,`claude-cli-opus` , runs the Claude CLI on your Claude Code sign-in, so it needs no key. For an API, use a`claude-` ,`gpt-` or`gemini-` model ID.`generate_run()` reads the pair’s`report.json` , fills in`summaries` and`recommendations` and saves it again. It keeps findings already saved for the same model; pass`use_cache=False` to write them again. ### The results `report.json` has a `summaries` block. Here’s an example, trimmed to the liability cap deviation and the first gap on the reference side: You can find this in the demo above. ### Recommendations Here is the `recommendations` block from the same run, trimmed to the liability cap finding: ### Run the demo UI to view results ## Further development You’ve built a conflict checker containing normalization, indexing, and analysis that can check one document against another and write down every deviation and gap. No vector database, and no generative model deciding what counts as a conflict. Generation optionally adds a written summary and recommendation to each finding. There are plenty of ways to extend it: - **Tune it.** The questions, the weights and the thresholds can be refitted to your own reference documents;`pairs.json` keeps every answer, so you can try new weights or a new threshold against a saved run for free. A lower`CONFLICT_THRESHOLD` finds more real differences at the cost of more false alarms. - **Mind your inputs.** The pipeline expects clean markdown. Plain text gets converted for you, but if you’ve got tables or scanned documents, fix the files rather than adding rules for them. - **Run your reference documents against each other.** Find the conflicts in your own templates (not some meta conflictception, I promise). - **Make it faster.** Cache classifier results, skip repeated clause pairs, and find the sweet spot for parallel requests. Originally I’d kept a full history of results plus a dashboard, but, y’know… Brevity. - **Build on it.** Redlining, review queues, pre-signing checklists, template and playbook checks, and comparisons when a reference document is updated. It’s a demo so I didn’t do any of that, but I needed you to know I thought of it. 🩵 If you somehow managed to scroll this far without actually doing any coding, the notebook is available in our [cookbook](https://github.com/isaacus-dev/cookbooks/tree/main/cookbooks/conflict-detector) and the demo files are available separately on GitHub. Run it on your own files – good luck!