{"slug": "clausewatch-an-agent-that-refuses-to-give-you-one-number", "title": "\"ClauseWatch: an agent that refuses to give you one number\"", "summary": "A developer built ClauseWatch, an open-source agent that surfaces conflicts between legal instruments rather than returning a single answer, such as the EU AI Act's six-month log retention floor colliding with the GDPR's unnamed ceiling. The tool pairs a structured Sanity dataset of provisions, requirements, and claims with an LLM agent and a deterministic no-model mode, marking unresolved conflicts as \"unmade\" with empty decidedBy and decidedAt fields. It runs without credentials against a public query API, with the knowledge base and agent requiring a Context token.", "body_md": "*This is a submission for the [Sanity Challenge, Path One: Ship an Agent That Queries Real Content](https://dev.to/challenges/sanity-2026-09-16)*\n\nAsk any model how long you must keep AI system logs under the EU AI Act. It will say six\n\nmonths. That is true, and it is not the answer.\n\nOne instrument sets a floor. Another sets a ceiling. Neither says where the other one sits.\n\nThe honest answer is a period you can justify against both — plus a record of who decided it.\n\nClauseWatch gives that answer. It reads a structured model of obligations and the prose of the\n\ninstruments behind them, and when two sources pull in opposite directions it shows both with\n\ntheir citations and reports that the decision is **unmade**:\n\nBoth clauses stand until someone signs. The dataset keeps a `decidedBy` and a `decidedAt`\n\nfield for exactly that, and they are empty.\n\nThis is the part that needed structured content. A keyword search over the same texts returns\n\nArt. 19 and stops. It cannot know that a clause in a *different regulation* constrains the\n\nsame artifact from the opposite direction, because that fact is not inside either document.\n\nIt is in the relationship between them — which is to say, in the schema.\n\n```\ngit clone https://github.com/oleg-vdv/clausewatch && cd clausewatch/agent\nnpm install && npm run web       # http://localhost:4173\n```\n\n**No credentials.** No login, no token, no API key. The dataset is public, so a fresh clone\n\nreads it over the public query API and renders the full report. The knowledge base and the\n\nLLM agent do need a Context token, which you cannot have — so `--check` and the page footer\n\nsay which of the two sources the answer came from, rather than crediting one they never\n\ntouched.\n\nIt also runs with no model at all:\n\n```\nnpm run ask -- --profile biometric-access --no-llm   # provider, high-risk, KZ → EU\nnpm run ask -- --profile support-agent --no-llm      # deployer, risk class unset\n```\n\nTwo elements of the viewer do the arguing.\n\n**The pressure bar** — a solid edge on the left where Art. 19 states six months, and a\n\nhatched, edgeless right where the GDPR caps the period without naming one. Drawing a tidy\n\nrange there would be a lie about the law.\n\n**The signature block** — every conflict prints a *decided by* and a *date* rule. Resolved,\n\nthey carry a name and a date. Open, they are two empty lines in a compliance report. The\n\nthesis as a form field rather than a paragraph.\n\nThe Studio is live at **[https://clausewatch.sanity.studio/](https://clausewatch.sanity.studio/)**, with open conflicts on the first\n\nscreen, because that is the one thing an editor of this dataset actually does.\n\n[https://github.com/oleg-vdv/clausewatch](https://github.com/oleg-vdv/clausewatch)\n\n```\nstudio/   schema, desk structure, seed dataset\nagent/    MCP client, domain layer, LLM agent, deterministic report, viewer\ndemo/     saved runs with their tool calls attached\nsource        an instrument: binding status, official URL, version label\nprovision     one citable unit: \"Art. 19\", verbatim text, effective date, roles bound\nrequirement   a normalised obligation: \"retain automatically generated logs\"\n  └ claim[]   what each instrument asserts — direction (floor/ceiling/none), period if\n              stated, condition, and a reference to its provision\nconflict      two clauses that cannot both be satisfied, plus resolution, rationale,\n              decidedBy, decidedAt\nsystemProfile roles, jurisdictions, risk class, agentic or not\n```\n\nThree decisions carry the weight.\n\n**Claims sit apart from requirements.** One obligation, many instruments, and disagreement\n\nbetween them is content — not a data-quality problem to be cleaned up.\n\n**`direction` on a claim.** An earlier version bucketed claims by *does it state a number*,\n\nwhich filed GDPR Art. 5(1)(e) under silence. It is not silent: it caps the period without\n\nnaming one. Floor, ceiling and duty-only are three different answers.\n\n**A conflict holds a decision, not a resolution rule.** An agent that silently picks the\n\nstricter number is guessing on your behalf. One that shows both sides and cites a named,\n\ndated decision produces something an auditor can accept.\n\nTwo website sources, both crawled to exact paths rather than wildcards:\n\n| Source | Paths | Documents | \n|---|---|---|\n| artificialintelligenceact.eu | Arts. 3, 6, 9, 11–14, 16–21, 26, 27, 72, 73, 99 and Annexes I, III, IV | 21 | \n| gdpr-info.eu | Arts. 5, 17, 25, 30, 32 | 5 | \n\n26 documents, indexed into **10 entries**. Curated on purpose — and also capped, because the\n\nContext beta allows 150 indexed documents on this plan and I hit that twice before narrowing\n\nproperly.\n\nThe indexing surprised me. Entries are topic nodes rather than chunks: each carries an article\n\nrange, a topic list, cross-references (`excludes: … see <other entry>`) and a numbered Sources\n\nblock with a URL behind every statement.\n\nTwo endpoints, because **one endpoint serves one mode**. Attaching a Knowledge Base to an\n\nendpoint that already served a dataset replaced the GROQ tools entirely.\n\n| Endpoint | Tools | Job | \n|---|---|---|\n| `clausewatch-data` | `initial_context` ,`groq_query` ,`schema_explorer` ,`array_field_reader` | *which* obligations bind this system | \n| `clausewatch-docs` | `initial_context` ,`knowledge_base_read` | *what the clause says* , with its source link | \n\nBoth carry the same endpoint Instructions — the citation discipline the agent must follow,\n\ndelivered before it sees any data. The rule that matters most:\n\nNever fill a gap from your own legal knowledge. If it is not in the dataset or the knowledge\n\nbase, say that it is not there.\n\nBeyond the tools I leaned on two Context features: **Issues**, which surfaced a contradiction\n\nin the corpus, and **Instructions**, which corrected a fact at the source instead of in my\n\ncode. Both stories are below.\n\nA real tool-use loop. Nothing is pre-fetched: the model gets the two endpoints and has to go\n\nand look. A typical run:\n\n```\n1. initial_context         (dataset — schema and the citation rules)\n2. kb_initial_context      (knowledge base — the outline of 10 entries)\n3. groq_query              (the logging requirement, its claims, its conflicts)\n4. kb_knowledge_base_read  (design_documentation, post_market_obligations)\n5. groq_query              (jurisdictions, extraterritoriality)\n6. kb_knowledge_base_read  (high_risk_categories)\n7. groq_query              (system profiles)\n```\n\nSeven calls across both endpoints. The answer states it cannot give one number, shows the\n\nfloor and the ceiling with citations and effective dates, reports the conflict as open with\n\nnobody named, and notes the obligation does not bite until 2 December 2027. In a second run\n\nthe agent found the matching system profile unprompted, treated `riskClass: unknown` as\n\nload-bearing, separated provider duties from deployer ones, and closed with a section titled\n\n*What I can't answer from these sources*.\n\nSaved runs with their tool calls are in\n\n[`demo/`](https://github.com/oleg-vdv/clausewatch/tree/main/demo). The answer alone cannot\n\nshow that a number came from the endpoints rather than from the model, so the tool calls\n\ntravel with it.\n\nThe dataset side does the reasoning that a prose search cannot: claims are filtered by the\n\njurisdictions a system touches **and** the roles it holds, so a deployer is shown Art. 26(6)\n\nand a provider Art. 19. Telling a deployer that Art. 19 is their duty is not a rounding error,\n\nit is the wrong party.\n\nWhile indexing, Context flagged that an entry claimed Annex I lists 21 harmonisation\n\ninstruments while the source shows 20 — item 1 deleted by amendment, item 21 added. I checked\n\nagainst the source: true. It also proposed a Section A/B split that I could **not** verify,\n\nbecause EUR-Lex blocks automated retrieval.\n\nSo the decision recorded in the dataset says the detection was right, states only the verified\n\npart, explicitly excludes the unverified arithmetic, and carries a name and a date.\n\nThat is the whole product in one incident. Automated detection is good at *finding*\n\ndisagreement and is not authoritative about *resolving* it.\n\nIn one run the agent cited **Art. 26(5)** for the deployer's log-retention duty, from the\n\nknowledge base. The source says **Art. 26(6)**; 26(5) is the monitoring duty. A one-digit\n\nerror that reads as correct.\n\nChasing it was the most useful hour of the build. The knowledge-base entry for Article 26\n\nnumbers its own sections, and the fifth one is headed `### 5. Log retention`. Section five of\n\nan entry about Article 26 — and a model reading it produced \"Art. 26(5)\". The entry states the\n\nrule itself correctly, and **gives no paragraph number at all**:\n\nDeployers shall keep automatically generated logs for a period appropriate to the intended\n\npurpose, with a minimum of six months, unless applicable Union or national law … provides\n\notherwise.\n\nThat is the argument for two layers rather than one. Prose is right about the *rule* and\n\nsilent on the *address*; the dataset carries the address, verified by a human against the\n\nsource. Art. 26(6) is now a provision with the checked citation, its own claim on the\n\nretention requirement, and the role it binds — which is why a deployer profile is shown\n\nArt. 26(6) and a provider Art. 19.\n\nI also added a Context **Instruction**, which is *honored on every build, over the raw sources*, so a future rebuild cannot reintroduce the wrong number. Being precise about what\n\nThe layers check each other in both directions. In another run the agent found a duty present\n\nin the prose that my dataset had not modelled at all, and said so.\n\nHere is the check itself, thirty seconds of it: a script pulls Article 26 from the source and\n\nprints its numbered paragraphs. Paragraph 5 is the duty to monitor. Paragraph 6 is the duty to\n\nkeep the logs. That is the whole difference between citing the right clause and the wrong one,\n\nand nothing short of going to the text would have shown it.\n\n*Embedded session cards open collapsed — click the ▸ to see the command and everything it printed. Same for the three further down.*\n\n`count(*)` said 31, an anonymous one\nsaid 0. Sanity treats `_id` as a path and public read covers the root path only. An\nauthenticated count is not evidence that your dataset is public.`/article/*`\npulled 200 pages including Polish and French translations of the same articles. For a legal\nagent a translation is a correctness hazard, not noise — and the sitemap is the tell: if a\ncrawl returns more pages than the sitemap lists, it followed in-page language links.`_source no longer available` — and the entry is still listed in `initial_context`\nfor the agent to read and cite. Dismissing the issue keeps the entry; only a rebuild cleared\nthem.`roles[]->name` comes back as\n`[{name, _id}]`. It also injects `_type != \"sanity.agentContext\"` and pins\n`perspective: published`. Worth knowing when a query behaves differently in Vision.\nAnd one that is nobody's fault but mine: `count()` of a missing field is **null, not 0**, so a\n\nrole filter written as `count(appliesToRoles) == 0` for *binds everyone* silently dropped every\n\nGDPR claim. It looked fine in the provider's report and only broke for the deployer.\n\nThe numbers are the whole story, so here is the terminal output: an import that reports\n\nsuccess, a document fetch that comes back `{\"documents\":[],\"omitted\":[{\"reason\":\"permission\"}]}`,\n\na one-document probe with a plain id that proves the ids are the cause, and a final anonymous\n\n`count(*)` of 18 once everything was renamed to hyphens.\n\n`4yzoidsq`\n`production` (public)\nPublic dataset, no token — every conflict, both sides, with its decision:\n\n```\nhttps://4yzoidsq.api.sanity.io/v2026-09-19/data/query/production?query=*[_type==\"conflict\"]{summary,nature,resolution,decidedBy,\"sides\":sides[]->{\"cite\":source->shortName+\" \"+citation}}\n```\n\nThe whole build is on record — 485 messages in one Claude Code session. A note before you\n\nopen it: I work in Russian, so the conversation is in Russian. Everything that matters here\n\nis not. The tool calls, the GROQ queries, the MCP traces, the terminal output and the agent's\n\nown answers are all English, and that is what each slice below is made of. I have said what\n\nto look for in front of each one — and each card opens collapsed, so click the ▸ to see the\n\noutput rather than just the command line.\n\nThe question is how long a provider must keep logs for a high-risk biometric system. Watch\n\nthe trace: `initial_context` on the dataset, then on the knowledge base, then a GROQ query it\n\nwrites itself, then two knowledge-base entries, then jurisdictions, then profiles. Seven\n\ncalls across both Context endpoints before it says anything.\n\nThen read the answer. It gives a floor with a citation, a ceiling with a citation, states\n\nthat the conflict between them is unresolved and that nobody is recorded as having decided\n\nit, and points out that the obligation does not apply until December 2027. It never produces\n\nthe single number the question was fishing for.\n\nThis one starts with a Sanity Context issue: an entry says Annex I lists 21 harmonisation\n\ninstruments, the source shows 20. You will see me go to the source rather than take it on\n\ntrust — the `curl` that pulls the annex straight from the source, and the amendment markers\n\nin its output: item 1 struck, item 21 added.\n\nThe end of the slice is the part I would point a judge at. Context also proposed a Section\n\nA / Section B split, and I could not verify it, because EUR-Lex blocks automated retrieval.\n\nSo the decision recorded in the dataset states the verified part and explicitly excludes the\n\nrest. Finding disagreement and settling it are different jobs.\n\nThe knowledge base is built from authoritative reproductions, not the Official Journal —\n\nEUR-Lex blocks automated retrieval. Every `source` carries an `officialUrl` and the agent is\n\ninstructed to give that one for anything the reader will act on.\n\nThe dataset is small and deliberately so: 2 instruments, 9 provisions, 2 requirements,\n\n3 conflicts, 2 system profiles. A working core that demonstrates the model, not a compliance\n\nproduct. Nothing here is legal advice.", "url": "https://wpnews.pro/news/clausewatch-an-agent-that-refuses-to-give-you-one-number", "canonical_source": "https://dev.to/olegvdv/clausewatch-an-agent-that-refuses-to-give-you-one-number-19ke", "published_at": "2026-09-19 13:21:05+00:00", "updated_at": "2026-09-19 13:24:18.545244+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "structured-data", "agent-protocols", "ai-policy"], "entities": ["ClauseWatch", "Sanity", "EU AI Act", "GDPR", "GitHub", "oleg-vdv", "Context"], "alternates": {"html": "https://wpnews.pro/news/clausewatch-an-agent-that-refuses-to-give-you-one-number", "markdown": "https://wpnews.pro/news/clausewatch-an-agent-that-refuses-to-give-you-one-number.md", "text": "https://wpnews.pro/news/clausewatch-an-agent-that-refuses-to-give-you-one-number.txt", "jsonld": "https://wpnews.pro/news/clausewatch-an-agent-that-refuses-to-give-you-one-number.jsonld"}}