Check source spans in AI-extracted task lists

A developer published a small standard-library Python script, source_span_gate.py, that validates AI-extracted task lists by checking each cited quote against its claimed source line and character span. In a six-line fictional test set, the gate passed two correctly cited commitments and rejected fixtures with a changed word, a shifted offset, and an invented line ID, even though all were valid JSON.

An extracted task can cite a real line ID while quoting words that are not in that line. A small source-span check makes that error visible before you use the row as evidence. This exercise has six fictional lines. Maya promises to send an agenda, Noah commits to checking two totals, and Friday remains a proposal. All candidate outputs below are deliberate fixtures, not captured responses from a model. Ask your extractor for a narrow output: Extract explicit personal commitments. Return a JSON array with line id, start, end and quote. Quote the source verbatim. Use zero-based Python string indexes into the line text, including the speaker prefix; end is exclusive. Keep proposals and unassigned requests out of the commitment list. The runnable source and README https://gist.github.com/ilookatshow-gif/9d57d34196fc5e63a1754c09601b9f4e contain the same example. Save the code below as source span gate.py ; it uses only Python's standard library. python import json Fictional source lines. Candidate outputs below are also synthetic. SOURCES = { "L1": "Maya: I will send the agenda.", "L2": "Noah: Could we review it on Friday?", "L3": "Maya: Friday is still a proposal.", "L4": "Leo: Please check both totals.", "L5": "Noah: I will check both totals. No deadline yet.", "L6": "Maya: Nobody is assigned to send the final file.", } GOOD = """ {"line id":"L1","start":6,"end":29,"quote":"I will send the agenda."}, {"line id":"L5","start":6,"end":31,"quote":"I will check both totals."} """ def check raw : rows = json.loads raw if not isinstance rows, list : raise ValueError "expected an array" seen = set for number, row in enumerate rows, 1 : if not isinstance row, dict or set row = { "line id", "start", "end", "quote" }: raise ValueError f"row {number}: unexpected fields" ref, start, end, quote = row "line id" , row "start" , row "end" , row "quote" if not isinstance ref, str or ref not in SOURCES: raise ValueError f"row {number}: unknown source line" if type start is not int or type end is not int or not 0 <= start < end <= len SOURCES ref : raise ValueError f"row {number}: invalid span bounds" if not isinstance quote, str or SOURCES ref start:end = quote: raise ValueError f"row {number}: quote does not match source" span = ref, start, end if span in seen: raise ValueError f"row {number}: duplicate span" seen.add span return len rows if name == " main ": examples = { "good": GOOD, "changed word": GOOD.replace "send the agenda", "approve the agenda" , "wrong offset": GOOD.replace '"start":6', '"start":7', 1 , "unknown line": GOOD.replace '"L1"', '"L99"', 1 , } for name, raw in examples.items : try: print f"{name}: PASS {check raw } spans " except ValueError as error: print f"{name}: FAIL {error} " Run: python3 -B source span gate.py The locally executed output was: good: PASS 2 spans changed word: FAIL row 1: quote does not match source wrong offset: FAIL row 1: quote does not match source unknown line: FAIL row 1: unknown source line The good fixture selects two exact pieces of evidence: | Source | Expected interpretation | Agreed deadline | |---|---|---| | L1, indexes 6:29 | Maya will send the agenda. | Unknown | | L5, indexes 6:31 | Noah will check both totals. | Unknown | The changed-word fixture replaces “send” with “approve” in the candidate quote. The source stays unchanged, so the check rejects it. The other two fixtures move an offset or invent a source ID. They are rejected even though the JSON is parseable. The offsets refer to Python string positions, not UTF-8 byte positions. The JSON documentation https://docs.python.org/3/library/json.html describes parsing; the sequence documentation https://docs.python.org/3/library/stdtypes.html common-sequence-operations defines the slicing convention used here. A passing row establishes only that its quote matches the selected source span. It does not establish that the quote supports a proposed interpretation, that the list is complete, or that the source itself is true. An extractor could omit Noah's commitment and still produce valid spans for Maya. Keep a short manual review beside this check: L2 asks about Friday and L3 still calls it a proposal, so Friday is not an agreed deadline. L6 leaves the final file without a sender. Neither detail should turn into an assigned, dated task just because a nearby quote passes. For a real integration, first decide how to handle those semantic and completeness checks. This teaching function is not a complete production-input or JSON-hardening layer. It connects to no model, CRM or network. Disclosure: AI agents prepared this article and code. The source lines and outputs are synthetic; the exact code shown above was run locally. No model API was called. Dmitry Lukashov Дмитрий Лукашов — founder of Lookatshow, instructor and expert in applied AI at the Russian School of Management. Public author reference https://uprav.ru/news/424182/ .