{"slug": "show-hn-booth-ambiguity-detection-and-acceptance-checks-for-llm-outputs", "title": "Show HN: Booth – ambiguity detection and acceptance checks for LLM outputs", "summary": "Booth v0.4.2, a lightweight checkpoint library for LLM outputs, detects ambiguity and enforces acceptance conditions such as confidence thresholds and custom validation rules. It provides structured statuses (VERIFIED, REPAIRED, AMBIGUOUS, UNCERTAIN, BLOCKED) and supports evidence-agreement checks, with a default confidence threshold of 0.7. The library is provider-agnostic and designed to sit between applications and LLM calls to decide whether outputs should pass.", "body_md": "**A lightweight checkpoint library for LLM outputs.**\n\nBOOTH sits between your application and an LLM call and provides structured checkpoints for deciding whether an output should pass through, be reconsidered, be flagged as ambiguous, be checked against a custom validation rule, or be checked against evidence supplied by your application.\n\nBOOTH does not claim to know the truth. It checks whether an output meets a defined acceptance condition.\n\nThe name comes from the idea of a **ticket booth, toll booth, or parking/payment booth**: a booth doesn't need to know everything about what is happening beyond it. It checks whether the required condition has been met before allowing something to pass.\n\n**v0.4.2**\n\nBOOTH currently provides:\n\n- ambiguity detection\n- self-reported confidence checking\n- reconsideration retries for low-confidence answers\n- separate retry handling for unparseable model responses\n- an optional caller-supplied\n`validator`\n\nfor custom pass/fail rules on`check()`\n\n/`acheck()`\n\n, with its own distinct retry prompt - synchronous and asynchronous LLM checkpoint functions\n- evidence-agreement checking against evidence supplied by the caller\n- configurable confidence and evidence thresholds\n- structured result objects, including\n`result.method`\n\nto identify which mechanism produced a result - attempt history\n- explicit\n`VERIFIED`\n\n,`REPAIRED`\n\n,`AMBIGUOUS`\n\n,`UNCERTAIN`\n\n, and`BLOCKED`\n\nstatuses\n\nBOOTH is provider-agnostic. It does not require a particular LLM provider, retrieval system, vector database, or framework.\n\nA normal LLM call might look like:\n\n```\nanswer = call_llm(prompt)\n```\n\nBOOTH adds a checkpoint around the model call:\n\n``` python\nimport booth\n\nresult = booth.check(\n    call_llm,\n    \"What is the capital of France?\"\n)\n\nif result.ok:\n    print(result.answer)\nelse:\n    print(f\"BOOTH returned {result.status}\")\n```\n\nBOOTH asks the model to provide structured information about its response, including:\n\n```\n{\n  \"ambiguous\": false,\n  \"interpretations\": [],\n  \"chosen_interpretation\": null,\n  \"answer\": \"Paris\",\n  \"confidence\": 0.95\n}\n```\n\nBOOTH then applies its configured acceptance rules to that response, in this order:\n\n- If the question is identified as ambiguous, BOOTH returns\n`AMBIGUOUS`\n\nimmediately. - If a\n`validator`\n\nwas supplied and the answer fails it, BOOTH asks the model to reconsider, showing it the specific validation failure. - If the reported confidence is below the configured threshold, BOOTH can ask the model to reconsider its previous answer.\n\nIf the model reaches the threshold (and passes validation, if supplied) after reconsideration, the result is `REPAIRED`\n\n.\n\nIf BOOTH cannot obtain an acceptable result, it returns `UNCERTAIN`\n\n.\n\nBOOTH also provides `check_with_evidence()`\n\nfor applications that already have evidence from their own RAG, search, database, or tool pipeline.\n\nBOOTH asks the model to identify whether the question has multiple valid interpretations before accepting the answer.\n\nFor example:\n\n```\nWhat is the capital of Georgia?\n```\n\ncould refer to:\n\n``` php\nGeorgia (the country) -> Tbilisi\nGeorgia (the US state) -> Atlanta\n```\n\nBOOTH can return:\n\n```\nAMBIGUOUS\n```\n\nwith the detected interpretations available through:\n\n```\nresult.interpretations\n```\n\nAmbiguity takes priority over everything else BOOTH checks. A highly confident, validator-passing answer can still be returned as `AMBIGUOUS`\n\nif the model identifies multiple valid readings — and a `validator`\n\nis never even invoked on an ambiguous attempt.\n\nBOOTH uses the model's reported confidence as an acceptance signal.\n\nThe default threshold is:\n\n```\n0.7\n```\n\nYou can configure it:\n\n```\nresult = booth.check(\n    call_llm,\n    prompt,\n    threshold=0.8\n)\n```\n\nThe confidence value is **self-reported by the model**. BOOTH does not calibrate or independently validate that probability.\n\nWhen an answer is not ambiguous but its confidence is below the configured threshold, BOOTH can ask the model to reconsider its previous answer.\n\nFor example:\n\n```\nPrevious answer: \"Lyon\"\nPrevious confidence: 0.3\n\nReconsider carefully. If that answer is correct, restate it.\nIf it is wrong, give the corrected answer.\n```\n\nIf the reconsidered answer reaches the threshold, BOOTH returns:\n\n```\nREPAIRED\n```\n\nYou can control the number of retries:\n\n```\nresult = booth.check(\n    call_llm,\n    prompt,\n    max_retries=2\n)\n```\n\n`max_retries=0`\n\nmeans only the initial model call is made.\n\nLLM responses do not always follow the requested format.\n\nBOOTH handles parse failures separately from low-confidence answers.\n\nWhen a response cannot be parsed, the retry prompt tells the model that its previous response failed to meet the required format rather than simply repeating the original request.\n\nYou can determine whether an `UNCERTAIN`\n\nresult occurred because no response could ever be parsed:\n\n```\nresult.all_parse_failed\n```\n\nA value of `True`\n\nmeans that every attempt failed to produce a valid BOOTH response.\n\nBOOTH can run a caller-supplied validation rule against each attempt's answer, in addition to (and checked separately from) ambiguity and confidence.\n\n``` php\ndef is_valid_order_id(answer: str) -> bool:\n    return answer.strip().upper().startswith(\"ORD-\")\n\nresult = booth.check(\n    call_llm,\n    \"What is the order ID for this request?\",\n    validator=is_valid_order_id,\n)\n```\n\n`validator`\n\nreceives the attempt's answer and returns either:\n\n`True`\n\n/`False`\n\n— a plain pass/fail.`False`\n\nproduces a generic failure message.`(bool, str)`\n\n— pass/fail plus a specific reason, shown to the model verbatim on the retry prompt:\n\n``` python\ndef validate_amount(answer: str):\n    if not answer.replace(\".\", \"\", 1).isdigit():\n        return False, \"The answer must be a plain numeric amount, e.g. 42.50\"\n    return True, \"\"\n\nresult = booth.check(call_llm, prompt, validator=validate_amount)\n```\n\n**Ordering:** an attempt is only run through `validator`\n\nif it parsed successfully **and** was not flagged ambiguous — a question that's ambiguous as asked isn't something a validator should be judging, and there is nothing to validate if the response never parsed. A validation failure is checked *before* the confidence gate: an answer that fails your validator does not get a chance to pass purely on high self-reported confidence.\n\nAn exception raised inside `validator`\n\n, or a return value that isn't `bool`\n\nor `(bool, str)`\n\n, is treated as a failed validation — it never propagates out of `check()`\n\n/`acheck()`\n\n.\n\n`validator=None`\n\n(the default) is a true no-op: every code path this parameter introduces is unreachable if you never pass it, so existing calls are unaffected.\n\n`validator`\n\nmust be **synchronous**, for both `check()`\n\nand `acheck()`\n\n. If your validation logic needs to await something (an API call, a DB lookup), resolve it yourself first and pass a plain sync closure in.\n\nTells you which of BOOTH's mechanisms actually determined a result, derived entirely from existing fields:\n\n```\nresult.method\n# \"ambiguity\"     — status is AMBIGUOUS\n# \"evidence\"      — result came from check_with_evidence()\n# \"parse_failure\" — UNCERTAIN because every attempt failed to parse\n# \"validation\"    — UNCERTAIN because the last attempt parsed fine\n#                    and was confident enough, but failed your validator\n# \"confidence\"    — the ordinary case: VERIFIED / REPAIRED, or\n#                    UNCERTAIN from persistent low confidence on an\n#                    attempt that did parse and did pass validation\n```\n\nThis is most useful for `UNCERTAIN`\n\nresults, where it distinguishes three genuinely different problems that call for different fixes:\n\n```\nif result.status == booth.UNCERTAIN:\n    if result.method == \"parse_failure\":\n        print(\"Model never produced a parseable response — check call_fn / prompt formatting.\")\n    elif result.method == \"validation\":\n        print(\"Model was confident, but never satisfied the custom validator.\")\n    else:\n        print(\"Model tried, but confidence never reached the threshold.\")\n```\n\n`method`\n\nreflects the **last** attempt's determining factor for a mixed history (e.g. a parse failure followed by a validation failure reports `\"validation\"`\n\n), the same rule `all_parse_failed`\n\nalready follows — it is not a full history of every attempt's outcome.\n\nBOOTH provides both:\n\n```\nbooth.check()\n```\n\nand:\n\n```\nawait booth.acheck()\n```\n\nThe synchronous version accepts:\n\n```\nCallable[[str], str]\n```\n\nThe asynchronous version accepts:\n\n```\nCallable[[str], Awaitable[str]]\n```\n\nExample:\n\n``` php\nimport asyncio\nimport booth\n\nasync def call_llm(prompt: str) -> str:\n    response = await async_client(...)\n    return response\n\nasync def main():\n    result = await booth.acheck(\n        call_llm,\n        \"What is the capital of France?\"\n    )\n\n    if result.ok:\n        print(result.answer)\n\nasyncio.run(main())\n```\n\nBoth APIs use the same decision logic, including `validator`\n\n. The difference is how the supplied LLM function is called.\n\nBOOTH also provides:\n\n```\nbooth.check_with_evidence()\n```\n\nThis checks whether an answer agrees with evidence that **your application has already retrieved**.\n\nExample:\n\n```\nresult = booth.check_with_evidence(\n    answer=\"Paris is the capital of France.\",\n    evidence=[\n        \"France's capital city is Paris.\"\n    ],\n    compare_fn=compare_answer_to_evidence,\n)\n```\n\nThe comparison function belongs to the caller:\n\n``` python\ndef compare_answer_to_evidence(answer, evidence):\n    ...\n```\n\nBOOTH does not choose a retrieval system or comparison algorithm for you.\n\nThe comparison function can return either `True`\n\n/`False`\n\nfor a simple pass/fail comparison, or a float between `0.0`\n\nand `1.0`\n\n:\n\n```\n0.87\n```\n\nWhen a float is returned, BOOTH compares it with `evidence_threshold`\n\n:\n\n```\nresult = booth.check_with_evidence(\n    answer=answer,\n    evidence=evidence,\n    compare_fn=compare_answer_to_evidence,\n    evidence_threshold=0.8,\n)\n```\n\nA score of `0.87`\n\npasses. A score of `0.62`\n\ndoes not.\n\nBoolean comparison results are treated as strict pass/fail values. `evidence_threshold`\n\nis not applied to boolean results.\n\n`check_with_evidence()`\n\nhas no `validator`\n\nconcept — it is a standalone, single-purpose comparison gate, untouched by the `validator`\n\naddition in this release.\n\n`check_with_evidence()`\n\nchecks **agreement with the evidence supplied to it**.\n\nIt does not establish that the evidence itself is true.\n\nFor example, if your application retrieves an incorrect document:\n\n```\nDigital downloads are never eligible for refunds.\n```\n\nand your comparison function determines that the answer agrees with that document, BOOTH can return:\n\n```\nVERIFIED\n```\n\nThat means the answer passed the supplied evidence comparison. It does **not** mean BOOTH independently established that the evidence is correct.\n\nThe quality, relevance, completeness, freshness, and correctness of retrieved evidence remain the responsibility of the application. This applies with equal force when evidence is baked into a prompt as RAG context and then separately checked — the model can produce a highly confident, unambiguous, evidence-agreeing answer that is still simply wrong, if the retrieved evidence itself was wrong. Neither `check()`\n\n's confidence check nor `check_with_evidence()`\n\n's agreement check can catch that; only the quality of retrieval can.\n\n```\nbooth.check(\n    call_fn,\n    prompt,\n    threshold=0.7,\n    max_retries=1,\n    on_attempt=None,\n    *,\n    validator=None,\n)\n```\n\nChecks an LLM response using ambiguity detection, confidence checking, reconsideration, and (if supplied) a custom validator.\n\nA synchronous function, `Callable[[str], str]`\n\n, that receives a prompt and returns the model's raw response.\n\nThe original application or user prompt.\n\nMinimum self-reported confidence required to accept an unambiguous, validator-passing answer. Default `0.7`\n\n. Must be between `0.0`\n\nand `1.0`\n\n.\n\nNumber of retries after the initial attempt. Default `1`\n\n.\n\nOptional callback invoked after each attempt.\n\nOptional `Callable[[str], bool | tuple[bool, str]]`\n\n. Runs on an attempt's answer only if that attempt parsed successfully and was not ambiguous. See [Custom validation with validator](#custom-validation-with-validator) above for the full contract. Must be synchronous. Default\n\n`None`\n\n— a true no-op.\n\n```\nawait booth.acheck(\n    call_fn,\n    prompt,\n    threshold=0.7,\n    max_retries=1,\n    on_attempt=None,\n    *,\n    validator=None,\n)\n```\n\nAsynchronous equivalent of `check()`\n\n, including full `validator`\n\nsupport (still required to be synchronous itself). The supplied `call_fn`\n\nmust be asynchronous:\n\n``` php\nasync def call_llm(prompt: str) -> str:\n    ...\nbooth.check_with_evidence(\n    answer,\n    evidence,\n    compare_fn,\n    evidence_threshold=0.7,\n)\n```\n\nChecks an answer against caller-supplied evidence. It:\n\n- makes no LLM calls\n- makes no network calls\n- performs no retrieval\n- performs no retries\n- does not modify a previous\n`BoothResult`\n\n- has no\n`validator`\n\nparameter — it is a standalone comparison gate - uses the caller's\n`compare_fn`\n\nThe answer being checked.\n\nA sequence of evidence strings already retrieved by the application.\n\nA caller-supplied comparison function, `Callable[[str, Sequence[str]], bool | float]`\n\n. Receives `answer`\n\nand `evidence`\n\n, returns either a boolean or a score from `0.0`\n\nto `1.0`\n\n.\n\nMinimum score required when `compare_fn`\n\nreturns a float. Default `0.7`\n\n. Separate from `check()`\n\n's `threshold`\n\nbecause the two values represent different things.\n\nBOOTH returns a `BoothResult`\n\n.\n\nImportant fields include:\n\n```\nresult.answer\nresult.status\nresult.confidence\nresult.evidence_agreement\nresult.attempts\nresult.n_attempts\nresult.ok\nresult.ambiguous\nresult.interpretations\nresult.all_parse_failed\nresult.method\n```\n\nThe answer produced by the model or supplied to the evidence checker. May be `None`\n\nwhen no usable answer exists.\n\nOne of `VERIFIED`\n\n, `REPAIRED`\n\n, `AMBIGUOUS`\n\n, `UNCERTAIN`\n\n, `BLOCKED`\n\n.\n\nFor normal LLM checks, the model's self-reported confidence. For evidence checks, the comparison score when available.\n\nThe comparison score produced by `check_with_evidence()`\n\n. `None`\n\nfor normal `check()`\n\n/ `acheck()`\n\nresults.\n\nThe full history of LLM attempts made by `check()`\n\nor `acheck()`\n\n, each including per-attempt `passed_validation`\n\n/ `validation_error`\n\n(always `True`\n\n/ `None`\n\nif no `validator`\n\nwas supplied). Evidence checks do not make attempts, so their attempt list is empty.\n\nNumber of recorded attempts.\n\n`True`\n\nonly for `VERIFIED`\n\n/ `REPAIRED`\n\n. `False`\n\nfor `AMBIGUOUS`\n\n, `UNCERTAIN`\n\n, `BLOCKED`\n\n.\n\nWhether the model marked the question as ambiguous.\n\nThe interpretations reported when the model marks a question as ambiguous.\n\n`True`\n\nif every LLM attempt failed to produce a parseable BOOTH response. Useful for distinguishing a formatting/integration problem from persistent model uncertainty or validation failure.\n\nWhich mechanism produced the result — `\"ambiguity\"`\n\n, `\"evidence\"`\n\n, `\"parse_failure\"`\n\n, `\"validation\"`\n\n, or `\"confidence\"`\n\n. See [ result.method](#resultmethod) above.\n\nThe result passed BOOTH's acceptance condition on the relevant check. For normal LLM checking, the answer was not ambiguous, passed validation (if supplied), and met the confidence threshold on the initial attempt. For evidence checking, the supplied comparison passed. `VERIFIED`\n\ndoes **not** mean independently proven true.\n\nThe initial LLM answer did not meet the confidence or validation requirement, but a reconsideration attempt produced an acceptable result.\n\nThe model identified multiple valid interpretations of the question. BOOTH returns this immediately rather than using a confidence retry or a validator to resolve it.\n\nBOOTH could not obtain an acceptable result. This can occur because:\n\n- the model remained below the confidence threshold\n- every response failed to parse\n- an answer that did parse and was confident enough still failed the supplied\n`validator`\n\non every attempt - the answer or evidence supplied to\n`check_with_evidence()`\n\nwas empty - the evidence comparison function raised an exception\n- the evidence comparison function returned an invalid score\n\nCheck `result.method`\n\nto tell these apart.\n\nThe supplied evidence comparison did not pass — a float score below `evidence_threshold`\n\n, or a boolean `False`\n\nfrom `compare_fn`\n\n.\n\nBOOTH currently does **not**:\n\n- guarantee factual correctness\n- independently establish truth\n- automatically browse the web\n- automatically perform RAG\n- automatically retrieve evidence\n- automatically choose a vector database\n- automatically choose an evidence-comparison method\n- retry evidence retrieval\n- manage a tool-calling loop\n- compare multiple independent LLMs\n- provide calibrated confidence probabilities\n- guarantee that retrieved evidence is correct, complete, relevant, or current\n- guarantee that a custom\n`validator`\n\nis itself correct — a validator can pass a wrong answer or reject a correct one, same as any other application-supplied rule - replace application-specific validation or safety systems (though\n`validator`\n\ngives you a documented hook to plug your own logic into BOOTH's retry loop rather than reimplementing that loop yourself)\n\nBOOTH is a **checkpoint library**, not an LLM framework, search engine, RAG framework, or autonomous verification system.\n\nA model can report `{\"confidence\": 0.99}`\n\nand still be wrong. BOOTH does not independently calibrate that number.\n\nAmbiguity detection depends on the model recognizing the ambiguity. BOOTH can detect useful structural ambiguities, but it cannot guarantee every possible interpretation is identified. A model can also mistake its own uncertainty for ambiguity.\n\n`validator`\n\nis exactly as reliable as the logic you give it. BOOTH enforces that a validator's decision is respected consistently in the retry loop — it does not, and cannot, check whether the validator's own logic is actually correct for your use case.\n\nEvidence checking is only as useful as the evidence and comparison function supplied by the application. If the evidence is wrong, incomplete, outdated, or unrelated, BOOTH does not independently detect that. Likewise, a weak `compare_fn`\n\ncan produce a misleading result. This includes the case where retrieved evidence is baked into the model's own prompt as RAG context — a wrong document can make the model's answer both more *confident* and more evidence-*consistent*, without becoming more correct.\n\n`check_with_evidence()`\n\ndeliberately does not retrieve documents. The application owns retrieval:\n\n```\nApplication\n    ↓\nRetrieve evidence\n    ↓\nBOOTH.check_with_evidence()\n    ↓\nVERIFIED / BLOCKED / UNCERTAIN\n```\n\nThis keeps BOOTH small and provider-agnostic.\n\n`check_with_evidence()`\n\nis a standalone evidence checkpoint. It does not automatically consume or modify the result of `check()`\n\nor `acheck()`\n\n. If an application wants to use multiple BOOTH checks together — including building a reconsideration loop that runs `check()`\n\nagain after a `BLOCKED`\n\nevidence result — the application decides how those results should be combined and how many extra attempts that composition is allowed to cost. BOOTH's own `max_retries`\n\nonly bounds a single `check()`\n\n/`acheck()`\n\ncall; it has no visibility into, or control over, retries you build on top across multiple calls.\n\nFor example:\n\n```\nb_result = booth.check(call_llm, prompt)\n\nif b_result.ok:\n    a_result = booth.check_with_evidence(\n        b_result.answer,\n        evidence,\n        compare_fn,\n    )\n\n    if a_result.ok:\n        print(a_result.answer)\n```\n\nThe composition logic remains under application control.\n\nFuture BOOTH development may explore:\n\n- stronger evidence adequacy checks\n- better handling of evidence completeness\n- improved detection of convention-based ambiguity\n- methods for distinguishing genuine ambiguity from model uncertainty\n- additional evidence-comparison strategies\n- richer composition of multiple checkpoint results\n- better evaluation and calibration tooling\n- additional integrations with retrieval and tool systems\n\nThese are future directions, not capabilities currently guaranteed by the library.\n\n```\npip install boothpy\n```\n\nBOOTH is also installable directly from GitHub:\n\n```\npip install git+https://github.com/Vedantgitbot/booth.git\n```\n\nClone the repository and install the development dependencies:\n\n```\npip install -e \".[dev]\"\n```\n\nRun the test suite:\n\n```\npytest\n```\n\nThe test suite covers the core checkpoint behavior, asynchronous API, parsing behavior, ambiguity handling, reconsideration, custom validation, and evidence checking. CI runs the full suite on push/PR across Python 3.9–3.12.\n\nThe validator tests include cases for: `validator=None`\n\nfull-regression parity, boolean and `(bool, str)`\n\nreturns, exceptions raised inside a validator, invalid return types, validator never running on ambiguous or unparseable attempts, fail-then-pass producing `REPAIRED`\n\n, the distinct validation-failure retry prompt, mixed multi-attempt histories and how `result.method`\n\nresolves them, and `check()`\n\n/`acheck()`\n\nparity under `validator`\n\n.\n\n**Keep the checkpoint small.** BOOTH should provide a reusable decision layer rather than become another full LLM framework.**Make uncertainty explicit.** When an output does not meet the configured acceptance condition, return a structured status instead of silently passing it through.**Treat ambiguity, validation, and confidence as separate, ordered checks.** A confident, validator-passing answer can still be ambiguous; a confident answer can still fail a caller's own validation rule before confidence is ever consulted.**Reconsider instead of blindly resampling.** Retries give the model an opportunity to examine its previous response — and the reason it failed (parse failure, validation failure, or low confidence) determines what the model is actually shown.**Keep evidence retrieval outside BOOTH.** Applications remain free to use their own RAG, search, database, or tool infrastructure.**Do not pretend agreement is truth.** Agreement with an answer, confidence value, custom validator, or retrieved evidence is not the same as independently proving the claim.**Stay provider-agnostic.** BOOTH works with different LLM providers because the application supplies the model-calling function.\n\nThis is the official BOOTH repository — Vedant Brahmbhatt\n\nBOOTH is released under the MIT License.\n\nSee [ LICENSE](/Vedantgitbot/booth/blob/main/LICENSE) for the full license text.", "url": "https://wpnews.pro/news/show-hn-booth-ambiguity-detection-and-acceptance-checks-for-llm-outputs", "canonical_source": "https://github.com/Vedantgitbot/booth", "published_at": "2026-08-28 13:37:38+00:00", "updated_at": "2026-08-28 13:48:33.664875+00:00", "lang": "en", "topics": ["ai-tools", "ai-safety", "artificial-intelligence"], "entities": ["Booth"], "alternates": {"html": "https://wpnews.pro/news/show-hn-booth-ambiguity-detection-and-acceptance-checks-for-llm-outputs", "markdown": "https://wpnews.pro/news/show-hn-booth-ambiguity-detection-and-acceptance-checks-for-llm-outputs.md", "text": "https://wpnews.pro/news/show-hn-booth-ambiguity-detection-and-acceptance-checks-for-llm-outputs.txt", "jsonld": "https://wpnews.pro/news/show-hn-booth-ambiguity-detection-and-acceptance-checks-for-llm-outputs.jsonld"}}