Show HN: Booth – ambiguity detection and acceptance checks for LLM outputs

Booth v0.4.2, a lightweight checkpoint library for LLM outputs, detects ambiguity and enforces acceptance conditions such as confidence thresholds and custom validation rules. It provides structured statuses (VERIFIED, REPAIRED, AMBIGUOUS, UNCERTAIN, BLOCKED) and supports evidence-agreement checks, with a default confidence threshold of 0.7. The library is provider-agnostic and designed to sit between applications and LLM calls to decide whether outputs should pass.

A lightweight checkpoint library for LLM outputs. BOOTH sits between your application and an LLM call and provides structured checkpoints for deciding whether an output should pass through, be reconsidered, be flagged as ambiguous, be checked against a custom validation rule, or be checked against evidence supplied by your application. BOOTH does not claim to know the truth. It checks whether an output meets a defined acceptance condition. The name comes from the idea of a ticket booth, toll booth, or parking/payment booth : a booth doesn't need to know everything about what is happening beyond it. It checks whether the required condition has been met before allowing something to pass. v0.4.2 BOOTH currently provides: - ambiguity detection - self-reported confidence checking - reconsideration retries for low-confidence answers - separate retry handling for unparseable model responses - an optional caller-supplied validator for custom pass/fail rules on check / acheck , with its own distinct retry prompt - synchronous and asynchronous LLM checkpoint functions - evidence-agreement checking against evidence supplied by the caller - configurable confidence and evidence thresholds - structured result objects, including result.method to identify which mechanism produced a result - attempt history - explicit VERIFIED , REPAIRED , AMBIGUOUS , UNCERTAIN , and BLOCKED statuses BOOTH is provider-agnostic. It does not require a particular LLM provider, retrieval system, vector database, or framework. A normal LLM call might look like: answer = call llm prompt BOOTH adds a checkpoint around the model call: python import booth result = booth.check call llm, "What is the capital of France?" if result.ok: print result.answer else: print f"BOOTH returned {result.status}" BOOTH asks the model to provide structured information about its response, including: { "ambiguous": false, "interpretations": , "chosen interpretation": null, "answer": "Paris", "confidence": 0.95 } BOOTH then applies its configured acceptance rules to that response, in this order: - If the question is identified as ambiguous, BOOTH returns AMBIGUOUS immediately. - If a validator was supplied and the answer fails it, BOOTH asks the model to reconsider, showing it the specific validation failure. - If the reported confidence is below the configured threshold, BOOTH can ask the model to reconsider its previous answer. If the model reaches the threshold and passes validation, if supplied after reconsideration, the result is REPAIRED . If BOOTH cannot obtain an acceptable result, it returns UNCERTAIN . BOOTH also provides check with evidence for applications that already have evidence from their own RAG, search, database, or tool pipeline. BOOTH asks the model to identify whether the question has multiple valid interpretations before accepting the answer. For example: What is the capital of Georgia? could refer to: php Georgia the country - Tbilisi Georgia the US state - Atlanta BOOTH can return: AMBIGUOUS with the detected interpretations available through: result.interpretations Ambiguity takes priority over everything else BOOTH checks. A highly confident, validator-passing answer can still be returned as AMBIGUOUS if the model identifies multiple valid readings — and a validator is never even invoked on an ambiguous attempt. BOOTH uses the model's reported confidence as an acceptance signal. The default threshold is: 0.7 You can configure it: result = booth.check call llm, prompt, threshold=0.8 The confidence value is self-reported by the model . BOOTH does not calibrate or independently validate that probability. When an answer is not ambiguous but its confidence is below the configured threshold, BOOTH can ask the model to reconsider its previous answer. For example: Previous answer: "Lyon" Previous confidence: 0.3 Reconsider carefully. If that answer is correct, restate it. If it is wrong, give the corrected answer. If the reconsidered answer reaches the threshold, BOOTH returns: REPAIRED You can control the number of retries: result = booth.check call llm, prompt, max retries=2 max retries=0 means only the initial model call is made. LLM responses do not always follow the requested format. BOOTH handles parse failures separately from low-confidence answers. When a response cannot be parsed, the retry prompt tells the model that its previous response failed to meet the required format rather than simply repeating the original request. You can determine whether an UNCERTAIN result occurred because no response could ever be parsed: result.all parse failed A value of True means that every attempt failed to produce a valid BOOTH response. BOOTH can run a caller-supplied validation rule against each attempt's answer, in addition to and checked separately from ambiguity and confidence. php def is valid order id answer: str - bool: return answer.strip .upper .startswith "ORD-" result = booth.check call llm, "What is the order ID for this request?", validator=is valid order id, validator receives the attempt's answer and returns either: True / False — a plain pass/fail. False produces a generic failure message. bool, str — pass/fail plus a specific reason, shown to the model verbatim on the retry prompt: python def validate amount answer: str : if not answer.replace ".", "", 1 .isdigit : return False, "The answer must be a plain numeric amount, e.g. 42.50" return True, "" result = booth.check call llm, prompt, validator=validate amount Ordering: an attempt is only run through validator if it parsed successfully and was not flagged ambiguous — a question that's ambiguous as asked isn't something a validator should be judging, and there is nothing to validate if the response never parsed. A validation failure is checked before the confidence gate: an answer that fails your validator does not get a chance to pass purely on high self-reported confidence. An exception raised inside validator , or a return value that isn't bool or bool, str , is treated as a failed validation — it never propagates out of check / acheck . validator=None the default is a true no-op: every code path this parameter introduces is unreachable if you never pass it, so existing calls are unaffected. validator must be synchronous , for both check and acheck . If your validation logic needs to await something an API call, a DB lookup , resolve it yourself first and pass a plain sync closure in. Tells you which of BOOTH's mechanisms actually determined a result, derived entirely from existing fields: result.method "ambiguity" — status is AMBIGUOUS "evidence" — result came from check with evidence "parse failure" — UNCERTAIN because every attempt failed to parse "validation" — UNCERTAIN because the last attempt parsed fine and was confident enough, but failed your validator "confidence" — the ordinary case: VERIFIED / REPAIRED, or UNCERTAIN from persistent low confidence on an attempt that did parse and did pass validation This is most useful for UNCERTAIN results, where it distinguishes three genuinely different problems that call for different fixes: if result.status == booth.UNCERTAIN: if result.method == "parse failure": print "Model never produced a parseable response — check call fn / prompt formatting." elif result.method == "validation": print "Model was confident, but never satisfied the custom validator." else: print "Model tried, but confidence never reached the threshold." method reflects the last attempt's determining factor for a mixed history e.g. a parse failure followed by a validation failure reports "validation" , the same rule all parse failed already follows — it is not a full history of every attempt's outcome. BOOTH provides both: booth.check and: await booth.acheck The synchronous version accepts: Callable str , str The asynchronous version accepts: Callable str , Awaitable str Example: php import asyncio import booth async def call llm prompt: str - str: response = await async client ... return response async def main : result = await booth.acheck call llm, "What is the capital of France?" if result.ok: print result.answer asyncio.run main Both APIs use the same decision logic, including validator . The difference is how the supplied LLM function is called. BOOTH also provides: booth.check with evidence This checks whether an answer agrees with evidence that your application has already retrieved . Example: result = booth.check with evidence answer="Paris is the capital of France.", evidence= "France's capital city is Paris." , compare fn=compare answer to evidence, The comparison function belongs to the caller: python def compare answer to evidence answer, evidence : ... BOOTH does not choose a retrieval system or comparison algorithm for you. The comparison function can return either True / False for a simple pass/fail comparison, or a float between 0.0 and 1.0 : 0.87 When a float is returned, BOOTH compares it with evidence threshold : result = booth.check with evidence answer=answer, evidence=evidence, compare fn=compare answer to evidence, evidence threshold=0.8, A score of 0.87 passes. A score of 0.62 does not. Boolean comparison results are treated as strict pass/fail values. evidence threshold is not applied to boolean results. check with evidence has no validator concept — it is a standalone, single-purpose comparison gate, untouched by the validator addition in this release. check with evidence checks agreement with the evidence supplied to it . It does not establish that the evidence itself is true. For example, if your application retrieves an incorrect document: Digital downloads are never eligible for refunds. and your comparison function determines that the answer agrees with that document, BOOTH can return: VERIFIED That means the answer passed the supplied evidence comparison. It does not mean BOOTH independently established that the evidence is correct. The quality, relevance, completeness, freshness, and correctness of retrieved evidence remain the responsibility of the application. This applies with equal force when evidence is baked into a prompt as RAG context and then separately checked — the model can produce a highly confident, unambiguous, evidence-agreeing answer that is still simply wrong, if the retrieved evidence itself was wrong. Neither check 's confidence check nor check with evidence 's agreement check can catch that; only the quality of retrieval can. booth.check call fn, prompt, threshold=0.7, max retries=1, on attempt=None, , validator=None, Checks an LLM response using ambiguity detection, confidence checking, reconsideration, and if supplied a custom validator. A synchronous function, Callable str , str , that receives a prompt and returns the model's raw response. The original application or user prompt. Minimum self-reported confidence required to accept an unambiguous, validator-passing answer. Default 0.7 . Must be between 0.0 and 1.0 . Number of retries after the initial attempt. Default 1 . Optional callback invoked after each attempt. Optional Callable str , bool | tuple bool, str . Runs on an attempt's answer only if that attempt parsed successfully and was not ambiguous. See Custom validation with validator custom-validation-with-validator above for the full contract. Must be synchronous. Default None — a true no-op. await booth.acheck call fn, prompt, threshold=0.7, max retries=1, on attempt=None, , validator=None, Asynchronous equivalent of check , including full validator support still required to be synchronous itself . The supplied call fn must be asynchronous: php async def call llm prompt: str - str: ... booth.check with evidence answer, evidence, compare fn, evidence threshold=0.7, Checks an answer against caller-supplied evidence. It: - makes no LLM calls - makes no network calls - performs no retrieval - performs no retries - does not modify a previous BoothResult - has no validator parameter — it is a standalone comparison gate - uses the caller's compare fn The answer being checked. A sequence of evidence strings already retrieved by the application. A caller-supplied comparison function, Callable str, Sequence str , bool | float . Receives answer and evidence , returns either a boolean or a score from 0.0 to 1.0 . Minimum score required when compare fn returns a float. Default 0.7 . Separate from check 's threshold because the two values represent different things. BOOTH returns a BoothResult . Important fields include: result.answer result.status result.confidence result.evidence agreement result.attempts result.n attempts result.ok result.ambiguous result.interpretations result.all parse failed result.method The answer produced by the model or supplied to the evidence checker. May be None when no usable answer exists. One of VERIFIED , REPAIRED , AMBIGUOUS , UNCERTAIN , BLOCKED . For normal LLM checks, the model's self-reported confidence. For evidence checks, the comparison score when available. The comparison score produced by check with evidence . None for normal check / acheck results. The full history of LLM attempts made by check or acheck , each including per-attempt passed validation / validation error always True / None if no validator was supplied . Evidence checks do not make attempts, so their attempt list is empty. Number of recorded attempts. True only for VERIFIED / REPAIRED . False for AMBIGUOUS , UNCERTAIN , BLOCKED . Whether the model marked the question as ambiguous. The interpretations reported when the model marks a question as ambiguous. True if every LLM attempt failed to produce a parseable BOOTH response. Useful for distinguishing a formatting/integration problem from persistent model uncertainty or validation failure. Which mechanism produced the result — "ambiguity" , "evidence" , "parse failure" , "validation" , or "confidence" . See result.method resultmethod above. The result passed BOOTH's acceptance condition on the relevant check. For normal LLM checking, the answer was not ambiguous, passed validation if supplied , and met the confidence threshold on the initial attempt. For evidence checking, the supplied comparison passed. VERIFIED does not mean independently proven true. The initial LLM answer did not meet the confidence or validation requirement, but a reconsideration attempt produced an acceptable result. The model identified multiple valid interpretations of the question. BOOTH returns this immediately rather than using a confidence retry or a validator to resolve it. BOOTH could not obtain an acceptable result. This can occur because: - the model remained below the confidence threshold - every response failed to parse - an answer that did parse and was confident enough still failed the supplied validator on every attempt - the answer or evidence supplied to check with evidence was empty - the evidence comparison function raised an exception - the evidence comparison function returned an invalid score Check result.method to tell these apart. The supplied evidence comparison did not pass — a float score below evidence threshold , or a boolean False from compare fn . BOOTH currently does not : - guarantee factual correctness - independently establish truth - automatically browse the web - automatically perform RAG - automatically retrieve evidence - automatically choose a vector database - automatically choose an evidence-comparison method - retry evidence retrieval - manage a tool-calling loop - compare multiple independent LLMs - provide calibrated confidence probabilities - guarantee that retrieved evidence is correct, complete, relevant, or current - guarantee that a custom validator is itself correct — a validator can pass a wrong answer or reject a correct one, same as any other application-supplied rule - replace application-specific validation or safety systems though validator gives you a documented hook to plug your own logic into BOOTH's retry loop rather than reimplementing that loop yourself BOOTH is a checkpoint library , not an LLM framework, search engine, RAG framework, or autonomous verification system. A model can report {"confidence": 0.99} and still be wrong. BOOTH does not independently calibrate that number. Ambiguity detection depends on the model recognizing the ambiguity. BOOTH can detect useful structural ambiguities, but it cannot guarantee every possible interpretation is identified. A model can also mistake its own uncertainty for ambiguity. validator is exactly as reliable as the logic you give it. BOOTH enforces that a validator's decision is respected consistently in the retry loop — it does not, and cannot, check whether the validator's own logic is actually correct for your use case. Evidence checking is only as useful as the evidence and comparison function supplied by the application. If the evidence is wrong, incomplete, outdated, or unrelated, BOOTH does not independently detect that. Likewise, a weak compare fn can produce a misleading result. This includes the case where retrieved evidence is baked into the model's own prompt as RAG context — a wrong document can make the model's answer both more confident and more evidence- consistent , without becoming more correct. check with evidence deliberately does not retrieve documents. The application owns retrieval: Application ↓ Retrieve evidence ↓ BOOTH.check with evidence ↓ VERIFIED / BLOCKED / UNCERTAIN This keeps BOOTH small and provider-agnostic. check with evidence is a standalone evidence checkpoint. It does not automatically consume or modify the result of check or acheck . If an application wants to use multiple BOOTH checks together — including building a reconsideration loop that runs check again after a BLOCKED evidence result — the application decides how those results should be combined and how many extra attempts that composition is allowed to cost. BOOTH's own max retries only bounds a single check / acheck call; it has no visibility into, or control over, retries you build on top across multiple calls. For example: b result = booth.check call llm, prompt if b result.ok: a result = booth.check with evidence b result.answer, evidence, compare fn, if a result.ok: print a result.answer The composition logic remains under application control. Future BOOTH development may explore: - stronger evidence adequacy checks - better handling of evidence completeness - improved detection of convention-based ambiguity - methods for distinguishing genuine ambiguity from model uncertainty - additional evidence-comparison strategies - richer composition of multiple checkpoint results - better evaluation and calibration tooling - additional integrations with retrieval and tool systems These are future directions, not capabilities currently guaranteed by the library. pip install boothpy BOOTH is also installable directly from GitHub: pip install git+https://github.com/Vedantgitbot/booth.git Clone the repository and install the development dependencies: pip install -e ". dev " Run the test suite: pytest The test suite covers the core checkpoint behavior, asynchronous API, parsing behavior, ambiguity handling, reconsideration, custom validation, and evidence checking. CI runs the full suite on push/PR across Python 3.9–3.12. The validator tests include cases for: validator=None full-regression parity, boolean and bool, str returns, exceptions raised inside a validator, invalid return types, validator never running on ambiguous or unparseable attempts, fail-then-pass producing REPAIRED , the distinct validation-failure retry prompt, mixed multi-attempt histories and how result.method resolves them, and check / acheck parity under validator . Keep the checkpoint small. BOOTH should provide a reusable decision layer rather than become another full LLM framework. Make uncertainty explicit. When an output does not meet the configured acceptance condition, return a structured status instead of silently passing it through. Treat ambiguity, validation, and confidence as separate, ordered checks. A confident, validator-passing answer can still be ambiguous; a confident answer can still fail a caller's own validation rule before confidence is ever consulted. Reconsider instead of blindly resampling. Retries give the model an opportunity to examine its previous response — and the reason it failed parse failure, validation failure, or low confidence determines what the model is actually shown. Keep evidence retrieval outside BOOTH. Applications remain free to use their own RAG, search, database, or tool infrastructure. Do not pretend agreement is truth. Agreement with an answer, confidence value, custom validator, or retrieved evidence is not the same as independently proving the claim. Stay provider-agnostic. BOOTH works with different LLM providers because the application supplies the model-calling function. This is the official BOOTH repository — Vedant Brahmbhatt BOOTH is released under the MIT License. See LICENSE /Vedantgitbot/booth/blob/main/LICENSE for the full license text.