{"slug": "legal-ai-not-a-coding-agent-with-scaffolding", "title": "Legal AI, not a coding agent with scaffolding", "summary": "Legal AI tools built on coding-agent scaffolds fail to meet the evidentiary and procedural rigor required for legal work, according to Alan Yahya. He argues that legal AI must provide granular evidence grounding, explicit source hierarchies, and version-controlled redline edits rather than automated changes. Current coding-agent approaches treat evidence as optional, risking sanctions for lawyers who rely on them.", "body_md": "# Legal AI, not a coding agent with scaffolding.\n\n[Alan Yahya](https://www.linkedin.com/in/alan-yahya/)12 min read\n\nThe majority of legal AI use is transacted through agents built for coding. Let’s focus on what a system built for a legal purpose might tangibly achieve.\n\nGenerally, legal AI should find the strongest supportable arguments, identify the correct evidence, and make that evidence easy to verify.\n\nAn agent workspace is the correct interface. The user sees an audit trail including the tools and sources used by the model. They are able to control the context by creating side chats, and define specific sources and skills to manipulate model behaviour.\n\nAgent grounding must happen at the level of individual claims and edits. It is simply not good enough to retrieve an external document and cite it as a whole. This level of granularity depends on the argument- which may call for an individual sentence, a paragraph, or a set of pages. The hierarchy, status, and intended purpose of each source should be explicit. That information can be stored via an ontology or similar graph network. For example, legal authority vs client facts, current document language vs previous drafting language.\n\nWhen the AI suggests an edit, the lawyer should be able to see why it made that change. That includes the original instruction, the clauses it reviewed, the facts it considered, the legal sources it relied on, and any warnings or uncertainties that can be calculated deterministically and packaged neatly for the user.\n\nAI should not make changes automatically. It should show the proposed edit as a redline (or tracked change!) that can be processed individually. This workflow goes some way to deterrings users from blindly accepting changes. Needless to say, changes are logged and revocable at any time using an underlying version control system. Letting a probabilistic process edit a document without a version control system would simply be reckless.\n\nSkills, memory and compaction should preserve access to evidence. They should not replace evidence with summaries. That is not good enough in a legal use-case. Rather, it should leave behind a lookup artifact that allows the exact original content to be viewed for factual information.\n\nAI, even evidence grounded, must never subvert legal judgment. Sources may be outdated, narrow, incomplete, or limited to a particular jurisdiction. The system should make these limitations more visible, not less, and enable a lawyer to make fully informed decisions with up-to-date information.\n\nCoding agents only enforce mechanical correctness, with interpretation as a guideline. Lawyers are being sanctioned because evidence grounding is being treated as an optional guidance layer, rather than a core product or workflow requirement. And unfortunately, that is the current story for legal AI tools.\n\n## Deleting a clause: did the patch match, and were its dependencies checked?\n\nTake the instruction: delete Section 12.7 because its obligation appears to be duplicated elsewhere. There are two different ways this can go wrong. The text may have changed since the edit was prepared, or another clause may still refer to Section 12.7.\n\nCodex's native patch path addresses the first failure directly. It looks for the old lines in the current file:\n\n`codex/codex-rs/apply-patch/src/lib.rs:765`\n\n## Code example\n\n``` js\nlet mut pattern: &[String] = &chunk.old_lines;\n\nlet mut found =\n    seek_sequence::seek_sequence(original_lines, pattern, line_index, chunk.is_end_of_file);\n```\n\nIf the expected text is no longer present, the patch fails rather than applying against a stale location:\n\n`codex/codex-rs/apply-patch/src/lib.rs:787`\n\n## Code example\n\n``` js\nif let Some(start_idx) = found {\n    replacements.push((start_idx, pattern.len(), new_slice.to_vec()));\n    line_index = start_idx + pattern.len();\n} else {\n    return Err(ApplyPatchError::ComputeReplacements(format!(\n        \"Failed to find expected lines in {}:\\n{}\",\n        path,\n        chunk.old_lines.join(\"\\n\"),\n    )));\n}\n```\n\nThat is a strong textual invariant, but it says nothing about legal-document dependencies. Codex can search for cross-references, but the native patch application does not require that search to have happened.\n\nLexifina records whether a structured cross-reference lookup for the section completed successfully:\n\n## Code example\n\n``` php\ndef _completed_cross_reference_terms(ctx: ToolUseContext) -> set[str]:\n    terms: set[str] = set()\n    for row in getattr(ctx.document_state, \"tool_call_log\", None) or []:\n        if str(row.get(\"tool_name\") or \"\") != \"get_cross_references\":\n            continue\n        if str(row.get(\"status\") or \"\").lower() != \"ok\" or row.get(\"is_error\"):\n            continue\n        raw_input = row.get(\"raw_input\")\n        if not isinstance(raw_input, dict):\n            raw_input = row.get(\"input_preview\")\n        if not isinstance(raw_input, dict):\n            continue\n        term = _normalized_reference_term(raw_input.get(\"term\"))\n        if term:\n            terms.add(term)\n    return terms\n```\n\nFor a whole-section deletion, absence of that recorded review becomes an edit-level warning:\n\n## Code example\n\n``` python\ndef _cross_reference_advisories(\n    *,\n    ctx: ToolUseContext,\n    original_text: str,\n    new_text: str,\n) -> List[Dict[str, Any]]:\n    if not original_text.strip() or new_text.strip():\n        return []\n\n    label = _section_reference_label(original_text)\n    if not label or _normalized_reference_term(label) in _completed_cross_reference_terms(ctx):\n        return []\n\n    return [\n        {\n            \"code\": \"cross_reference_review_not_recorded\",\n            \"severity\": \"warning\",\n            \"message\": (\n                f\"No completed structured cross-reference review for {label} \"\n                \"was recorded before this deletion. The deletion remains staged \"\n                \"for your decision.\"\n            ),\n            \"source\": \"deterministic_legal_review\",\n            \"details\": {\"section_label\": label},\n        }\n    ]\n```\n\nEach system protects against different failures. Codex prevents a stale hunk from being applied to text it no longer matches. Lexifina carries the absence of a structural-dependency check into the review object.\n\n## Compaction: can the model recover the exact tool output?\n\nSuppose a research tool returned a long opinion extract, the model used two paragraphs from it, and the conversation was compacted before drafting finished. A narrative summary that says “the case supports the argument” is not the evidence. The relevant question is whether the active agent can get the exact tool output back.\n\nIn Codex's local compaction path, the replacement history is built from retained user messages and a generated summary:\n\n`codex/codex-rs/core/src/compact.rs:323`\n\n## Code example\n\n``` js\nlet history_snapshot = sess.clone_history().await;\nlet history_items = history_snapshot.raw_items();\nlet summary_suffix = get_last_assistant_message_from_turn(history_items).unwrap_or_default();\nlet summary_text = format!(\"{SUMMARY_PREFIX}\\n{summary_suffix}\");\nlet user_messages = collect_user_messages(history_items);\n\nlet mut new_history = build_compacted_history(Vec::new(), &user_messages, &summary_text);\n```\n\nThat is the model's active replacement history. It does not add a per-tool-result lookup to the summary. This does not mean Codex deletes its durable rollout, and an agent may be able to reread a source file or rerun a tool.\n\nLexifina's compaction marker leaves the model an executable address for a cleared result:\n\n## Code example\n\n```\n# Marker template — note the {tool_use_id} placeholder is filled per\n# tool result so the model can call read_tool_result with the right\n# argument without searching for the id. Format string, not f-string,\n# so callers that import this module can introspect it.\nTOOL_RESULT_CLEARED_MARKER_TEMPLATE = (\n    \"[Tool result cleared during compaction — full content available \"\n    'via read_tool_result(tool_use_id=\"{tool_use_id}\") if needed.]'\n)\n```\n\n`read_tool_result`\n\nresolves that id to the persisted full output and returns the requested range without asking another model to reconstruct it:\n\n## Code example\n\n```\npath = Path(RESULT_PERSIST_DIR) / process_id / f\"{input_data.tool_use_id}.txt\"\nif not path.exists():\n    return ToolResult(\n        is_error=True,\n        assistant_text=(\n            f\"No persisted result found for tool_use_id=\"\n            f\"{input_data.tool_use_id!r}. Either the id is wrong \"\n            \"or the result was small enough to ship inline (no \"\n            \"truncation notice = no persisted file). Check the \"\n            \"tool result you copied the id from.\"\n        ),\n        summary=\"not found\",\n    )\n```\n\n## Code example\n\n```\nbounded_len = min(int(input_data.length), MAX_FETCH_CHARS)\ntry:\n    with path.open(\"r\", encoding=\"utf-8\") as f:\n        f.seek(int(input_data.offset))\n        chunk = f.read(bounded_len)\n```\n\nThis recovery is limited to the active run: the model-facing file is temporary and deleted when the run ends:\n\n## Code example\n\n```\n# Result-size cap: oversized rendered text is persisted to\n# disk and the model-facing block becomes a head/tail preview\n# with the tool_use_id. Skipped when the tool itself opts out\n# via max_result_size_chars=math.inf (read_tool_result, where\n# persisting a fetched chunk would create a circular loop) or\n# when the result is an error (errors are usually small and\n# already structured for the model — persisting them just adds\n# a layer of indirection). The persisted file is cleaned up\n# at end-of-run by routers/agentic.py.\n```\n\nIn short, compaction can replace bulk text without forcing the next model to trust a paraphrase; the `tool_use_id`\n\nremains a route back to the exact persisted tool text. Codex's local replacement history retains a summary but does not itself expose an equivalent call-id lookup.\n\n## A redline shows what changed. Can it prove which drafting decision changed it?\n\nSuppose version 3 capped indemnity at the fees paid under the agreement. The team then reverted to an earlier draft and created version 6, where the cap is gone. The lawyer's question is not merely “what text differs?” It is: did the current drafting branch remove the cap, which accepted edit did it, what instruction produced that edit, and is the answer based on the saved change record or a reconstruction made later?\n\nCodex maintains an exact net diff for file changes made by committed patches during the current request:\n\n`codex/codex-rs/core/src/turn_diff_tracker.rs:48`\n\n## Code example\n\n```\n/// Tracks the net text diff for the current turn from committed apply_patch\n/// mutations, without rereading the workspace filesystem.\npub struct TurnDiffTracker {\n    valid: bool,\n    display_roots_by_environment: HashMap<String, PathBuf>,\n    baseline_by_path: HashMap<TrackedPath, TrackedContent>,\n    current_by_path: HashMap<TrackedPath, TrackedContent>,\n    origin_by_current_path: HashMap<TrackedPath, TrackedPath>,\n    next_revision: u64,\n    rendered_diffs: HashMap<DiffCacheKey, Option<String>>,\n    unified_diff: Option<String>,\n```\n\nOnly exact patch deltas are admitted to that tracker. An inexact delta invalidates the result rather than allowing a potentially misleading diff to survive:\n\n`codex/codex-rs/core/src/turn_diff_tracker.rs:93`\n\n## Code example\n\n```\npub fn track_delta(&mut self, environment_id: &str, delta: &AppliedPatchDelta) {\n    if !self.valid {\n        return;\n    }\n\n    if !delta.is_exact() {\n        self.invalidate();\n        return;\n    }\n\n    for change in delta.changes() {\n        self.apply_change(environment_id, change);\n    }\n    self.refresh_unified_diff();\n}\n```\n\nThe event emitted to the client contains that unified diff:\n\n`codex/codex-rs/core/src/session/turn.rs:2495`\n\n## Code example\n\n``` js\nif should_emit_turn_diff {\n    let unified_diff = {\n        let tracker = turn_diff_tracker.lock().await;\n        tracker.get_unified_diff()\n    };\n    if let Some(unified_diff) = unified_diff {\n        let msg = EventMsg::TurnDiff(TurnDiffEvent { unified_diff });\n        sess.clone().send_event(&turn_context, msg).await;\n    }\n}\n```\n\nThe event's entire payload is one textual field:\n\n`codex/codex-rs/protocol/src/protocol.rs:3669`\n\n## Code example\n\n```\n#[derive(Debug, Clone, Deserialize, Serialize, JsonSchema, TS)]\npub struct TurnDiffEvent {\n    pub unified_diff: String,\n}\n```\n\nThat is a strong answer to “what did this Codex request change?” It does not make the diff event itself a saved-document lineage record: the hunk has no parent version, accepted edit id, triggering instruction, reviewer, application time, or confidence field. Codex can retain surrounding rollout events, and a version-control system can add commit history. The narrower point is that those relationships are not invariants of this native diff object.\n\nLexifina first follows the document's actual parent chain. It does not assume that version 6 descended from version 5, which would be wrong after a revert or branch:\n\n## Code example\n\n```\n# Walk the parent chain rather than blindly sorting by version\n# number. Branched versions (e.g. user reverted to v3 then applied\n# changes to produce v6 with parent=3) describe their delta against\n# their explicit parent, not against v(N-1). Following the parent\n# field gives the true evolution graph of the current document.\nchain = _walk_parent_chain(versions)\nall_registered: List[Dict[str, Any]] = []\nfor i in range(len(chain) - 1):\n    registered = register_pair_diffcites(\n        ctx.document_state,\n        version_a=chain[i],\n        version_b=chain[i + 1],\n    )\n```\n\nWhen the child version contains its apply-time change records, Lexifina uses the exact before-and-after strings captured when the user accepted the edit, before later parsing could renumber sections:\n\n## Code example\n\n``` python\ndef _extract_apply_time_hunks(\n    version_a: Dict[str, Any],\n    version_b: Dict[str, Any],\n) -> List[Dict[str, Any]]:\n    \"\"\"Tier 0: authoritative hunks from ``versions[N].changes``.\n\n    The apply route (`routers/changes.py:3158`) writes one entry per\n    accepted edit on the resulting version, capturing the exact\n    pre-edit and post-edit text strings BEFORE the post-apply re-parse\n    that renumbers section IDs. Reading that array gives us\n    position-blind, intent-bearing hunks for free — no LCS, no\n    section-id-equality join, no positional drift.\n```\n\nThe resulting hunk carries the version pair and text alongside the drafting provenance available for that accepted edit:\n\n## Code example\n\n```\nhunks.append(\n    {\n        \"from_version\": from_v,\n        \"to_version\": to_v,\n        \"from_date\": from_date,\n        \"to_date\": to_date,\n        \"section_id\": sid,\n        \"section_heading\": heading,\n        \"change_kind\": change_kind,\n        \"removed_text\": original,\n        \"added_text\": new_text,\n        \"explanation\": (\n            str(entry.get(\"explanation\") or \"\").strip() or None\n        ),\n        \"prompt_text\": (\n            str(source_prompt_text or \"\").strip() or None\n        ),\n        \"request_user\": (\n            str(request_user or \"\").strip() or None\n        ),\n        \"applied_at\": applied_at,\n        \"run_id\": str(run_id or \"\").strip() or None,\n        \"edit_id\": str(edit_id).strip() if edit_id else None,\n        \"confidence\": \"high\",\n        \"source\": \"apply_time\",\n    }\n)\n```\n\nLexifina also states when it cannot make that stronger claim. If a legacy or manually edited version has no apply-time record, it reconstructs a line diff and labels the source and confidence differently:\n\n## Code example\n\n```\nclass RiskDiffcite(BaseModel):\n    \"\"\"One atomic hunk between two persisted document versions.\n\n    Two `source` paths populate this:\n\n    - ``apply_time``: built from `` versions[N].changes`` — the exact\n      ``original``/`` new`` strings captured at the moment the user\n      clicked Accept. Position-blind by construction (no re-derivation\n      from post-apply structure). Carries apply-time provenance\n      (explanation, triggering request, who applied, when).\n      ``confidence`` is \"high\".\n    - ``line_diff``: a fallback hunk computed via\n      ``difflib.unified_diff`` over the version's flat text content.\n      Position-blind via LCS (same approach as the version-history\n      panel's Monaco diff). Used when ``versions[N].changes`` is\n      missing (legacy versions, manual editor edits that bypass the\n      apply route, the v1 baseline). ``confidence`` is \"medium\".\n```\n\nFinally, the model cannot create a convincing change receipt merely by inventing a `[[diff_n]]`\n\nmarker. Markers absent from the registered version-diff evidence are removed before the response is returned:\n\n## Code example\n\n``` php\ndef sanitize_diff_markers(text: Optional[str], valid_diff_ids: Iterable[str]) -> str:\n    \"\"\"Drop every ``[[diff_X]]`` whose id is not in the registry. Collapse\n    whitespace introduced by the removals so the prose still reads\n    cleanly. Identical surface to `sanitize_pin_markers`.\"\"\"\n    if not text:\n        return \"\"\n    valid = set(valid_diff_ids)\n\n    def _replace(match: re.Match[str]) -> str:\n        diff_id = match.group(1)\n        return match.group(0) if diff_id in valid else \"\"\n\n    cleaned = DIFF_ID_PATTERN.sub(_replace, text)\n```\n\nWhat this proves: both systems can produce an exact textual delta, but the native objects answer different questions. Codex proves the net file change made by committed patches during the current request. Lexifina can make a citable hunk prove which parent-and-child document versions it came from and, when an apply-time record exists, attach the exact accepted text, instruction, explanation, user, time, run, and edit id; when that stronger record is absent, it labels the reconstruction as lower-confidence evidence.\n\nAn agent harness boils down to a multi-layered series of defences for an intended use-case. There is a lot of work that can still be done in this direction for legal AI, and we have highlighted some here.", "url": "https://wpnews.pro/news/legal-ai-not-a-coding-agent-with-scaffolding", "canonical_source": "https://lexifina.com/blog/legal-ai-not-a-coding-agent-with-scaffolding", "published_at": "2026-07-14 23:51:57+00:00", "updated_at": "2026-07-15 00:17:54.179842+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-ethics", "ai-safety", "ai-products"], "entities": ["Alan Yahya", "Codex"], "alternates": {"html": "https://wpnews.pro/news/legal-ai-not-a-coding-agent-with-scaffolding", "markdown": "https://wpnews.pro/news/legal-ai-not-a-coding-agent-with-scaffolding.md", "text": "https://wpnews.pro/news/legal-ai-not-a-coding-agent-with-scaffolding.txt", "jsonld": "https://wpnews.pro/news/legal-ai-not-a-coding-agent-with-scaffolding.jsonld"}}