Legal AI, not a coding agent with scaffolding

Legal AI tools built on coding-agent scaffolds fail to meet the evidentiary and procedural rigor required for legal work, according to Alan Yahya. He argues that legal AI must provide granular evidence grounding, explicit source hierarchies, and version-controlled redline edits rather than automated changes. Current coding-agent approaches treat evidence as optional, risking sanctions for lawyers who rely on them.

Legal AI, not a coding agent with scaffolding. Alan Yahya https://www.linkedin.com/in/alan-yahya/ 12 min read The majority of legal AI use is transacted through agents built for coding. Let’s focus on what a system built for a legal purpose might tangibly achieve. Generally, legal AI should find the strongest supportable arguments, identify the correct evidence, and make that evidence easy to verify. An agent workspace is the correct interface. The user sees an audit trail including the tools and sources used by the model. They are able to control the context by creating side chats, and define specific sources and skills to manipulate model behaviour. Agent grounding must happen at the level of individual claims and edits. It is simply not good enough to retrieve an external document and cite it as a whole. This level of granularity depends on the argument- which may call for an individual sentence, a paragraph, or a set of pages. The hierarchy, status, and intended purpose of each source should be explicit. That information can be stored via an ontology or similar graph network. For example, legal authority vs client facts, current document language vs previous drafting language. When the AI suggests an edit, the lawyer should be able to see why it made that change. That includes the original instruction, the clauses it reviewed, the facts it considered, the legal sources it relied on, and any warnings or uncertainties that can be calculated deterministically and packaged neatly for the user. AI should not make changes automatically. It should show the proposed edit as a redline or tracked change that can be processed individually. This workflow goes some way to deterrings users from blindly accepting changes. Needless to say, changes are logged and revocable at any time using an underlying version control system. Letting a probabilistic process edit a document without a version control system would simply be reckless. Skills, memory and compaction should preserve access to evidence. They should not replace evidence with summaries. That is not good enough in a legal use-case. Rather, it should leave behind a lookup artifact that allows the exact original content to be viewed for factual information. AI, even evidence grounded, must never subvert legal judgment. Sources may be outdated, narrow, incomplete, or limited to a particular jurisdiction. The system should make these limitations more visible, not less, and enable a lawyer to make fully informed decisions with up-to-date information. Coding agents only enforce mechanical correctness, with interpretation as a guideline. Lawyers are being sanctioned because evidence grounding is being treated as an optional guidance layer, rather than a core product or workflow requirement. And unfortunately, that is the current story for legal AI tools. Deleting a clause: did the patch match, and were its dependencies checked? Take the instruction: delete Section 12.7 because its obligation appears to be duplicated elsewhere. There are two different ways this can go wrong. The text may have changed since the edit was prepared, or another clause may still refer to Section 12.7. Codex's native patch path addresses the first failure directly. It looks for the old lines in the current file: codex/codex-rs/apply-patch/src/lib.rs:765 Code example js let mut pattern: & String = &chunk.old lines; let mut found = seek sequence::seek sequence original lines, pattern, line index, chunk.is end of file ; If the expected text is no longer present, the patch fails rather than applying against a stale location: codex/codex-rs/apply-patch/src/lib.rs:787 Code example js if let Some start idx = found { replacements.push start idx, pattern.len , new slice.to vec ; line index = start idx + pattern.len ; } else { return Err ApplyPatchError::ComputeReplacements format "Failed to find expected lines in {}:\n{}", path, chunk.old lines.join "\n" , ; } That is a strong textual invariant, but it says nothing about legal-document dependencies. Codex can search for cross-references, but the native patch application does not require that search to have happened. Lexifina records whether a structured cross-reference lookup for the section completed successfully: Code example php def completed cross reference terms ctx: ToolUseContext - set str : terms: set str = set for row in getattr ctx.document state, "tool call log", None or : if str row.get "tool name" or "" = "get cross references": continue if str row.get "status" or "" .lower = "ok" or row.get "is error" : continue raw input = row.get "raw input" if not isinstance raw input, dict : raw input = row.get "input preview" if not isinstance raw input, dict : continue term = normalized reference term raw input.get "term" if term: terms.add term return terms For a whole-section deletion, absence of that recorded review becomes an edit-level warning: Code example python def cross reference advisories , ctx: ToolUseContext, original text: str, new text: str, - List Dict str, Any : if not original text.strip or new text.strip : return label = section reference label original text if not label or normalized reference term label in completed cross reference terms ctx : return return { "code": "cross reference review not recorded", "severity": "warning", "message": f"No completed structured cross-reference review for {label} " "was recorded before this deletion. The deletion remains staged " "for your decision." , "source": "deterministic legal review", "details": {"section label": label}, } Each system protects against different failures. Codex prevents a stale hunk from being applied to text it no longer matches. Lexifina carries the absence of a structural-dependency check into the review object. Compaction: can the model recover the exact tool output? Suppose a research tool returned a long opinion extract, the model used two paragraphs from it, and the conversation was compacted before drafting finished. A narrative summary that says “the case supports the argument” is not the evidence. The relevant question is whether the active agent can get the exact tool output back. In Codex's local compaction path, the replacement history is built from retained user messages and a generated summary: codex/codex-rs/core/src/compact.rs:323 Code example js let history snapshot = sess.clone history .await; let history items = history snapshot.raw items ; let summary suffix = get last assistant message from turn history items .unwrap or default ; let summary text = format "{SUMMARY PREFIX}\n{summary suffix}" ; let user messages = collect user messages history items ; let mut new history = build compacted history Vec::new , &user messages, &summary text ; That is the model's active replacement history. It does not add a per-tool-result lookup to the summary. This does not mean Codex deletes its durable rollout, and an agent may be able to reread a source file or rerun a tool. Lexifina's compaction marker leaves the model an executable address for a cleared result: Code example Marker template — note the {tool use id} placeholder is filled per tool result so the model can call read tool result with the right argument without searching for the id. Format string, not f-string, so callers that import this module can introspect it. TOOL RESULT CLEARED MARKER TEMPLATE = " Tool result cleared during compaction — full content available " 'via read tool result tool use id="{tool use id}" if needed. ' read tool result resolves that id to the persisted full output and returns the requested range without asking another model to reconstruct it: Code example path = Path RESULT PERSIST DIR / process id / f"{input data.tool use id}.txt" if not path.exists : return ToolResult is error=True, assistant text= f"No persisted result found for tool use id=" f"{input data.tool use id r}. Either the id is wrong " "or the result was small enough to ship inline no " "truncation notice = no persisted file . Check the " "tool result you copied the id from." , summary="not found", Code example bounded len = min int input data.length , MAX FETCH CHARS try: with path.open "r", encoding="utf-8" as f: f.seek int input data.offset chunk = f.read bounded len This recovery is limited to the active run: the model-facing file is temporary and deleted when the run ends: Code example Result-size cap: oversized rendered text is persisted to disk and the model-facing block becomes a head/tail preview with the tool use id. Skipped when the tool itself opts out via max result size chars=math.inf read tool result, where persisting a fetched chunk would create a circular loop or when the result is an error errors are usually small and already structured for the model — persisting them just adds a layer of indirection . The persisted file is cleaned up at end-of-run by routers/agentic.py. In short, compaction can replace bulk text without forcing the next model to trust a paraphrase; the tool use id remains a route back to the exact persisted tool text. Codex's local replacement history retains a summary but does not itself expose an equivalent call-id lookup. A redline shows what changed. Can it prove which drafting decision changed it? Suppose version 3 capped indemnity at the fees paid under the agreement. The team then reverted to an earlier draft and created version 6, where the cap is gone. The lawyer's question is not merely “what text differs?” It is: did the current drafting branch remove the cap, which accepted edit did it, what instruction produced that edit, and is the answer based on the saved change record or a reconstruction made later? Codex maintains an exact net diff for file changes made by committed patches during the current request: codex/codex-rs/core/src/turn diff tracker.rs:48 Code example /// Tracks the net text diff for the current turn from committed apply patch /// mutations, without rereading the workspace filesystem. pub struct TurnDiffTracker { valid: bool, display roots by environment: HashMap<String, PathBuf , baseline by path: HashMap<TrackedPath, TrackedContent , current by path: HashMap<TrackedPath, TrackedContent , origin by current path: HashMap<TrackedPath, TrackedPath , next revision: u64, rendered diffs: HashMap<DiffCacheKey, Option<String , unified diff: Option<String , Only exact patch deltas are admitted to that tracker. An inexact delta invalidates the result rather than allowing a potentially misleading diff to survive: codex/codex-rs/core/src/turn diff tracker.rs:93 Code example pub fn track delta &mut self, environment id: &str, delta: &AppliedPatchDelta { if self.valid { return; } if delta.is exact { self.invalidate ; return; } for change in delta.changes { self.apply change environment id, change ; } self.refresh unified diff ; } The event emitted to the client contains that unified diff: codex/codex-rs/core/src/session/turn.rs:2495 Code example js if should emit turn diff { let unified diff = { let tracker = turn diff tracker.lock .await; tracker.get unified diff }; if let Some unified diff = unified diff { let msg = EventMsg::TurnDiff TurnDiffEvent { unified diff } ; sess.clone .send event &turn context, msg .await; } } The event's entire payload is one textual field: codex/codex-rs/protocol/src/protocol.rs:3669 Code example derive Debug, Clone, Deserialize, Serialize, JsonSchema, TS pub struct TurnDiffEvent { pub unified diff: String, } That is a strong answer to “what did this Codex request change?” It does not make the diff event itself a saved-document lineage record: the hunk has no parent version, accepted edit id, triggering instruction, reviewer, application time, or confidence field. Codex can retain surrounding rollout events, and a version-control system can add commit history. The narrower point is that those relationships are not invariants of this native diff object. Lexifina first follows the document's actual parent chain. It does not assume that version 6 descended from version 5, which would be wrong after a revert or branch: Code example Walk the parent chain rather than blindly sorting by version number. Branched versions e.g. user reverted to v3 then applied changes to produce v6 with parent=3 describe their delta against their explicit parent, not against v N-1 . Following the parent field gives the true evolution graph of the current document. chain = walk parent chain versions all registered: List Dict str, Any = for i in range len chain - 1 : registered = register pair diffcites ctx.document state, version a=chain i , version b=chain i + 1 , When the child version contains its apply-time change records, Lexifina uses the exact before-and-after strings captured when the user accepted the edit, before later parsing could renumber sections: Code example python def extract apply time hunks version a: Dict str, Any , version b: Dict str, Any , - List Dict str, Any : """Tier 0: authoritative hunks from versions N .changes . The apply route routers/changes.py:3158 writes one entry per accepted edit on the resulting version, capturing the exact pre-edit and post-edit text strings BEFORE the post-apply re-parse that renumbers section IDs. Reading that array gives us position-blind, intent-bearing hunks for free — no LCS, no section-id-equality join, no positional drift. The resulting hunk carries the version pair and text alongside the drafting provenance available for that accepted edit: Code example hunks.append { "from version": from v, "to version": to v, "from date": from date, "to date": to date, "section id": sid, "section heading": heading, "change kind": change kind, "removed text": original, "added text": new text, "explanation": str entry.get "explanation" or "" .strip or None , "prompt text": str source prompt text or "" .strip or None , "request user": str request user or "" .strip or None , "applied at": applied at, "run id": str run id or "" .strip or None, "edit id": str edit id .strip if edit id else None, "confidence": "high", "source": "apply time", } Lexifina also states when it cannot make that stronger claim. If a legacy or manually edited version has no apply-time record, it reconstructs a line diff and labels the source and confidence differently: Code example class RiskDiffcite BaseModel : """One atomic hunk between two persisted document versions. Two source paths populate this: - apply time : built from versions N .changes — the exact original / new strings captured at the moment the user clicked Accept. Position-blind by construction no re-derivation from post-apply structure . Carries apply-time provenance explanation, triggering request, who applied, when . confidence is "high". - line diff : a fallback hunk computed via difflib.unified diff over the version's flat text content. Position-blind via LCS same approach as the version-history panel's Monaco diff . Used when versions N .changes is missing legacy versions, manual editor edits that bypass the apply route, the v1 baseline . confidence is "medium". Finally, the model cannot create a convincing change receipt merely by inventing a diff n marker. Markers absent from the registered version-diff evidence are removed before the response is returned: Code example php def sanitize diff markers text: Optional str , valid diff ids: Iterable str - str: """Drop every diff X whose id is not in the registry. Collapse whitespace introduced by the removals so the prose still reads cleanly. Identical surface to sanitize pin markers .""" if not text: return "" valid = set valid diff ids def replace match: re.Match str - str: diff id = match.group 1 return match.group 0 if diff id in valid else "" cleaned = DIFF ID PATTERN.sub replace, text What this proves: both systems can produce an exact textual delta, but the native objects answer different questions. Codex proves the net file change made by committed patches during the current request. Lexifina can make a citable hunk prove which parent-and-child document versions it came from and, when an apply-time record exists, attach the exact accepted text, instruction, explanation, user, time, run, and edit id; when that stronger record is absent, it labels the reconstruction as lower-confidence evidence. An agent harness boils down to a multi-layered series of defences for an intended use-case. There is a lot of work that can still be done in this direction for legal AI, and we have highlighted some here.