My Paper Reader Answered Questions for Weeks. I Never Checked If It Was Right.

Developer Talkit's paper-reading app, which reads research papers aloud and answers spoken interruptions, had no real answer-quality evaluation for weeks: its two test scripts only checked that an answer returned, so an answer inventing a BLEU score would have passed. The developer moved the answer logic out of the /api/ask handler into app/qa.py, wrote ten questions with known answers and four metrics, and added span attributes for backend scoring, with the first run surfacing a failure mode that had been present all along. Talkit sends the entire paper (capped at 60,000 characters; the Attention paper is 30,138 after reference-list removal) into the model context with no vector store, chunking, or embeddings on the answer path.

TL;DR I’ve been chipping away at a small project called Talkit. It reads research papers aloud and lets you interrupt it. You say “wait, why do they scale that?”, the narration stops, it answers in two or three spoken sentences, and then it picks up where it left off. That part works. I use it. What I could not have told you, for weeks, was whether the answers were any good. The evidence I had was two scripts. One uploads the Attention paper, asks a single question, and checks that an answer comes back. The other does the same in Hinglish. An answer that invented a BLEU score would have passed both. So would an answer that confidently explained the wrong paragraph. The endpoint returned 200 and the suite stayed green, and a green suite quietly becomes the thing you trust. The other reason I put it off is that the RAG advice I kept reading didn’t fit. Every guide opens with chunking and embeddings and retrieval metrics, and Talkit has no vector store at all: the paper fits in the model’s context, so it sends the whole thing, which leaves half the standard eval playbook describing a component I don’t have. So I finally sat down and measured the thing I actually built. This post is what that took: pulling the answer path into one function, writing ten questions with known answers, four metrics that each catch a different failure, and span attributes that let a backend score answers as they happen. None of it is elaborate. It’s the smallest setup that would tell me when an answer is wrong, and the first run turned up a failure mode that had been sitting there the whole time. Here’s the path a question takes: Steps 2 and 3 are the RAG part, and the answer that comes out of them is what this post evaluates. Talkit has no chunking and no embeddings on the answer path, and that was a deliberate choice. Uploads are capped at 60,000 characters. The Attention paper, after the reference list is cut, comes to 30,138. That fits in a model’s context with plenty of room, so the model sees all of it. The bigger reason is the kind of question I actually ask it. Halfway through a paper I’ll say “why does that matter” or “what’s this number”. Those questions retrieve badly by similarity, because the words that matter “that”, “this” point at what was just read out, not at anything an embedding can match. So the current passage goes in as its own labelled block. That passage is the retrieval signal. Every question sends the whole paper, so input tokens scale with paper length, not question complexity. The 60k cap also rules out books and long theses. If Talkit ever answers questions across a whole library, or takes documents past the cap, that’s when I’d go add a vector store, and neither of those is true today. The answer logic used to live inline in the /api/ask handler. An eval can't call a FastAPI route without a database, a session, and a stored key. And an eval that reimplements the prompt assembly measures a copy, which drifts the first time I edit the original. So it moved into app/qa.py, and the route calls it: python async def answer question provider: TextProvider, lang: str, paper: str, passage: str, question: str, history: list dict | None = None, user id: str | None = None - str: with tracer.start as current span "ask.answer" as span: messages = build messages paper, passage, history or , question raw = await provider.converse answer system lang , messages, max tokens=ANSWER MAX TOKENS answer = speakable raw, question describe llm span span, name="ask", model=provider.model, provider=provider.label, question=question.strip , answer=answer, grounding= passage, paper , user id=user id, metadata={"lang": lang, "history turns": len history or }, return answer build messages puts the paper and passage first, then up to six recent turns, then the question: python def build messages paper: str, passage: str, history: list dict , question: str - list dict : messages = {"role": "user", "content": f"FULL PAPER:\n{paper}\n\nCURRENT PASSAGE:\n{passage}"}, {"role": "assistant", "content": "Understood. I have the paper and I know where the listener is."}, for turn in history -HISTORY TURNS: : role = "user" if turn.get "role" == "user" else "assistant" messages.append {"role": role, "content": str turn.get "content", "" :HISTORY TURN CHARS } messages.append {"role": "user", "content": question.strip } return messages And speakable removes markdown plus a restated question. The system prompt already forbids markdown and the model mostly complies, which isn't good enough for text that goes straight into a text-to-speech voice. The refactor was pure motion: behavior didn’t change, and the existing 85 tests still pass. I added six more for this module, which brings the offline suite to 91. Talkit already sends OpenTelemetry traces. Spans are created unconditionally, and with no exporter configured the global provider is a no-op. Picking a backend is environment variables, with no vendor SDK in the code. I picked Confident AI because DeepEval, which I was already using for the evals, is theirs, so the eval runs and the traces land in the same project. Langfuse, Phoenix and others accept OpenTelemetry too, and everything up to this point would work with any of them. Pointing it at Confident AI: OTEL EXPORTER OTLP ENDPOINT=https://otel.confident-ai.comOTEL EXPORTER OTLP HEADERS=x-confident-api-key=<key OTEL LOGS EXPORTER=noneOTEL RESOURCE ATTRIBUTES=confident.trace.environment=production The HTTP exporter appends /v1/traces to the endpoint, so the base URL is enough. The environment label rides on the standard resource attribute variable, which the SDK already reads. The third line needed a code change. Talkit ships logs through the same exporter so each log record carries its trace and span id. Confident AI’s OpenTelemetry docs say the endpoint takes traces and not logs, so the logs have to be turned off separately. OTEL LOGS EXPORTER is the standard variable for this, so the setup now respects it: if ENDPOINT and OTEL LOGS EXPORTER = "none": logs = LoggerProvider resource=resource logs.add log record processor BatchLogRecordProcessor OTLPLogExporter set logger provider logs logging.getLogger .addHandler LoggingHandler logger provider=logs At this point traces arrive, but they’re only timings. A trace backend can show that ask.answer took four seconds. It can't tell whether the answer was grounded in the paper. To score an answer, Confident AI needs to know which span is the model call and what went in and came out. It reads that from span attributes in its own namespace. Talkit writes them in one function in app/telemetry.py: python def describe llm span span, , name: str, model: str | None, provider: str, question: str, answer: str, grounding: list str , user id: str | None = None, metadata: dict | None = None, content: bool | None = None, metric collection: str | None = None - None: content = TRACE CONTENT if content is None else content collection = TRACE METRIC COLLECTION if metric collection is None else metric collection span.set attributes { "confident.span.type": "llm", "confident.trace.name": name, "gen ai.request.model": model or "unknown", "gen ai.provider.name": provider.lower , } if user id: span.set attribute "confident.trace.user id", user id if metadata: span.set attribute "confident.span.metadata", json.dumps metadata if not content: return span.set attributes { "confident.span.input": json.dumps question , "confident.span.output": json.dumps answer , "confident.span.retrieval context": json.dumps grounding , "confident.trace.input": json.dumps question , "confident.trace.output": json.dumps answer , } if collection: span.set attribute "confident.span.metric collection", collection A few things here cost me more time than the code suggests. Text values are JSON-encoded. The convention expects "\"Why scale it?\"", not "Why scale it?", and a list of passages becomes a JSON array string. Before writing this I ran Confident AI's own confident-trace package against an in-memory exporter and read what it set, so the shapes match what their SDK emits. The collection name is the exception: it's a plain string. A backend accepts a wrongly encoded attribute without complaint and just shows an empty panel, so the tests assert the encoding directly. The grounding is the whole paper. retrieval context is passage, paper , because that's what the model saw. If I sent only the passage, the faithfulness metric would flag every true claim drawn from elsewhere in the paper as unsupported. Content is off unless you turn it on. Scoring needs the question, the answer, and the paper attached to the span and exported. That’s my reading and my questions leaving for a third party. So TRACE CONTENT=true is an explicit switch. Without it, the span carries the span type, model name, and metadata, and nothing a person wrote. There is an SDK for this, confident-trace, built on OpenTelemetry. It auto-instruments frameworks like LangChain, LlamaIndex and Pydantic AI, which would be the argument for using it. Talkit calls Sarvam over httpx and Claude through the Anthropic SDK directly, so there is nothing for it to auto-instrument. It would add a dependency and I would still set the input, output and grounding by hand. Writing the attributes myself keeps "no vendor SDK" true, and the ten confident. names above are the entire vendor-specific surface: swap backends and the spans, timings and gen ai. attributes go with you. The cost is that a vendor's attribute names now sit in the code, and if they change, that function changes with them. The live suites in Talkit already use arXiv 1706.03762v7, Attention Is All You Need , so the evals do too. The goldens live in evals/goldens.json. Each one has a question, a phrase that locates the current passage, a language, and an expected answer: { "question": "Why do they divide by the square root of d k here?", "passage contains": "particular attention", "lang": "en", "expected output": "For large values of d k the dot products grow large in magnitude, which pushes the softmax into regions with extremely small gradients. Scaling by one over the square root of d k counteracts that."} That one does double duty. It tests the decision to send the whole paper: the current passage is the one that introduces scaled dot-product attention, while the reason for the scaling comes later on, so an answer that only used that passage would miss it. The passage is located by searching the chunks for that phrase: php def locate chunks: list str , needle: str - str: flat = lambda s: re.sub r"\s+", " ", s .lower for chunk in chunks: if needle.lower in flat chunk : return chunk raise SystemExit f"no passage contains {needle r}; did chunking change?" An index would silently point at a different passage the first time chunking changes. A missing phrase fails loudly. I checked every phrase against the extracted text before writing the expected answers, and each one matches a passage among the paper’s 46. What the ten cover: Ten questions doesn’t cover much of anything. I’ll add a golden every time a real answer comes back wrong. php def metrics for model - list: return FaithfulnessMetric model=model, threshold=0.8 , AnswerRelevancyMetric model=model, threshold=0.7 , GEval name="Correct", model=model, threshold=0.7, evaluation params= SingleTurnParams.INPUT, SingleTurnParams.ACTUAL OUTPUT, SingleTurnParams.EXPECTED OUTPUT , evaluation steps= "Check that every fact in the expected output appears in the actual output. Wording may differ.", "Penalise any number, name or claim in the actual output that contradicts the expected output.", "If the expected output says the paper does not address the question, the actual output must say so and must not supply an answer from general knowledge.", "The actual output must be in the same language and script as the expected output.", , , GEval name="Speakable", model=model, threshold=0.7, evaluation params= SingleTurnParams.ACTUAL OUTPUT , evaluation steps= "The output will be read aloud by a text-to-speech voice to someone with no screen.", "It should be two to four sentences of plain prose.", "Penalise lists, headings, markdown, LaTeX, and symbols a voice would read literally, such as square root signs, subscripts or underscores.", "Numbers and variable names should be written the way a person would say them.", , , Faithfulness extracts the claims in the answer and checks each against the grounding, which is how it catches invented numbers. Answer relevancy catches an accurate answer to a different question, a real risk when the question is “what does this mean” and “this” could be several things. Correct compares against the expected answer, and it’s the metric that scores the refusals. Faithfulness can’t: “the paper doesn’t say” makes no claims to check. Speakable is the one no generic metric covers. A Talkit answer is heard, not read. A correct answer containing √dk hands the text-to-speech voice a symbol where it needs words, which is why the prompt asks for spoken forms and this metric checks that it got them. Anything code can check, code checks first, and prints before any judge weighs in: php def mechanical failures answer: str, lang: str - list str : failures = sentences = s for s in re.split r" ?<= . ? \s+", answer if s.strip if not 1 <= len sentences <= 5: failures.append f"{len sentences } sentences" if lang == "hinglish" and DEVANAGARI.search answer : failures.append "Devanagari in a Hinglish answer" if re.search r" √∈ \\$ ", answer : failures.append "unspeakable symbol" return failures Hinglish is romanised Hindi, and both Hinglish prompts forbid Devanagari outright because a model asked for it can drift into Hindi script, which a regex catches without any judge weighing in. Every metric above is a model grading a model. Zheng et al. 2023 https://arxiv.org/abs/2306.05685 documented that LLM judges show self-enhancement bias: they favor answers written by the same model. So the judge defaults to a different family from the one answering. Talkit’s default text model is sarvam-105b, and the default judge is claude-sonnet-5. Evaluating the Claude path instead trips a warning that suggests an OpenAI judge: if args.provider == "claude" and args.judge.startswith "claude" : print "warning: Claude is judging Claude. Pass --judge gpt-5.4 for a " "judge from a different family." The Claude judge broke on its first run. DeepEval’s Anthropic wrapper defaults to 1,024 output tokens. Faithfulness starts by listing every factual statement in the grounding, and the grounding here is a whole paper. That list ran past 1,024 tokens, came back as truncated JSON, and DeepEval reported “Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.” The fix was the budget: python def judge name: str : if name.startswith "claude" : return AnthropicModel model=name, generation kwargs={"max tokens": 16000} return OpenAIModel model=name Next time a judge hands back broken JSON over long context I’ll check the token limit before I blame the model. Four lines, run by hand when I’ve changed something worth checking: pip install -r evals/requirements.txtcurl -L -o attention.pdf https://arxiv.org/pdf/1706.03762v7export SARVAM API KEY=... OPENAI API KEY=...export CONFIDENT API KEY=...python evals/run.py --pdf attention.pdf --judge gpt-5.4 The script loads the paper exactly as an upload does extract, cut the back matter, cap at 60k , answers all ten goldens through answer question, runs the mechanical checks, then hands the test cases to DeepEval. With CONFIDENT API KEY set, the run uploads to Confident AI tagged with the provider, the model, the judge, and a short hash of the answer prompt. That hash is what makes prompt changes comparable, since two runs with different hashes can be put side by side against the same goldens. Evals have their own requirements file, which includes the app’s requirements plus deepeval==4.2.3. The Docker image never installs DeepEval. One side effect: DeepEval needs python-dotenv 1.1.1 or later, so the app's pin moved up from 1.0.1. Passages in the eval come from the extracted text. The app narrates a cleaned read-aloud script instead, and cleaning a paper takes minutes of paid API calls, which isn’t what this suite measures. Wording differs slightly from what I actually hear, and I’ve accepted that. I ran it with sarvam-105b answering and gpt-5.4 judging: Nine of nine, because one question never got an answer. Sometimes there’s no answer at all. sarvam-105b is a reasoning model, and on one question it spent the entire 3,000-token budget reasoning and never wrote a word. The provider raises ChatBudgetExceeded with finish reason=length. I ran the suite four times. Three runs had exactly one unanswered question the BLEU question twice, the attention-heads question once and one run answered all ten. The failure is intermittent and doesn't track any particular question. The same thing happened through the app itself, where "How many layers are in the encoder?" came back as a 502. Before evals, the only live test asked one question, so whether it passed depended on the run. The first version of the eval script crashed on it and threw away nine paid answers, so it now records an unanswered question as a failed case with the reason attached. Faithfulness didn’t discriminate. Every answer scored 1.00. The answers stayed inside the paper, including the refusals. Asked how much GPU memory the big model needed, it said the paper doesn’t say, and added that the model has 213 million parameters, which is in Table 3. That’s a good result, but it means faithfulness alone would have told me nothing, and the metric that actually moved was Correct. The one Correct failure was my golden’s fault. The Hinglish question asks “Encoder mein kitne layers hain?”, how many layers are in the encoder. The answer said six. My expected output also listed the two sub-layers, which the question never asked for, and the judge docked it to 0.65 for leaving them out. The judge’s reason spelled it out. The golden needs fixing, and the prompt can stay as it is. Speakable let one thing through that I’d call a miss: the answer to the scaling question writes “dk” for d k, which a voice reads as a two-letter word. The judge gave it 0.83, comfortably above threshold, and the mechanical check doesn’t look for it either. Later answers to the same question through the app did it again. That’s the next check to add in code, since a regex settles it better than a judge. Talkit’s offline checks are pyflakes, 91 tests, and a Node script that checks the voice command matcher against the shipped HTML, and none of them cost anything. This suite costs credits on every run: ten answers, then four metrics per answer, and each metric makes several judge calls. And as those runs showed, the model under test sometimes doesn’t answer at all. A hard gate would turn that into a flaky build I’d just rerun until it went green. So it runs by hand, like the two live suites that exercise Sarvam’s speech APIs. The script exits non-zero if any case fails a metric, a mechanical check, or gets no answer, so it could run in a scheduled job later. For now it’s a tool I run before changing a prompt or a model. Offline evals only know the questions I thought of. Actually using the thing turns up the ones I didn’t. Confident AI can score incoming traces with a metric collection: a named set of metrics defined in the project. Talkit reads the name from the environment: TRACE CONTENT=trueTRACE METRIC COLLECTION=talkit-answers With both set, every ask.answer span carries the question, the answer, the grounding, and the collection name, and Confident AI runs the collection's metrics against it. Faithfulness and answer relevancy both work here. Correctness doesn't, because a question asked in the moment has no expected answer written for it. Correctness still comes from goldens. When a live answer scores badly, it becomes a golden. One trap when creating the collection: Confident AI offers multi-turn metrics like Turn Faithfulness and Turn Relevancy alongside the single-turn ones. A collection of turn metrics has nothing to score on these spans. Talkit’s answer span is a single turn, one question and one answer with its grounding, so the collection needs the single-turn Faithfulness and Answer Relevancy. The other thing that cost me an evening: online evaluation needs a judge model of your own. Confident AI includes 100 traces of online evaluation, and after that the trace still arrives, still shows the collection name on the span, and simply is not scored until you configure an evaluation model in the project. The span looked correctly wired for a while before I noticed the banner saying so. With a judge configured, the first scored answer came back at 1.00 on both faithfulness and answer relevancy, with a written reason on each, against the same passage and paper the model had been given. The scores in the collection default to a threshold of 0.5, while the offline suite uses 0.8 for faithfulness and 0.7 for relevancy, so the same answer can pass in one place and fail in the other. I haven’t reconciled them yet. When one does score badly, the span has what I need to start: the model and provider, the language, how many history turns were in play, and the exact grounding. The usual first question is whether the problem is in the current passage or the answer. Unlike a vector-search RAG system, there’s no “wrong chunks retrieved” branch to rule out, since the model had the whole paper. Running the whole thing locally, upload to answer, surfaced two problems outside the answer path. Cleaning falls back more than I knew. Uploading the Attention paper through Sarvam took 837 seconds, and 14 of its 28 cleaning batches fell back to raw text. The log shows the same cause 52 times: a batch overran the token budget and was halved, until it hit the split limit and kept the raw text. That’s the same reasoning-budget problem as the missing answers, on the cleaning side. A fallback is recorded and never surfaced, so I just hear citation markers read out with no explanation. It was already on Talkit’s list of known gaps. Now it has a number. The title is wrong. Talkit’s title extractor reads the Attention paper’s title as “Provided proper attribution is provided, Google hereby grants permission to”, which is the permission banner at the top of the arXiv version. Neither is fixed. I only ran into them because I was reading logs and traces for a different reason. My Paper Reader Answered Questions for Weeks. I Never Checked If It Was Right. https://pub.towardsai.net/my-paper-reader-answered-questions-for-weeks-i-never-checked-if-it-was-right-f01996a69205 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.