SledTrace – A local debugger that shows why your RAG app answered wrong

SledTrace, a local Python debugger for RAG pipelines, has been released and is installable via `pip install sledtrace`, with the dashboard served at http://127.0.0.1:4319. The tool records retrieval chunks, LLM prompts and responses, tool calls, timing and token usage to a local SQLite database at ~/.sledtrace/sledtrace.db, then runs seven deterministic local heuristic warnings — including no_retrieved_chunks, conflicting_chunks, numeric_mismatch and answer_not_grounded — to show why an answer diverged from retrieved context. SledTrace is Python-only with explicit trace, retrieval and llm calls, has no automatic LangChain or LlamaIndex integrations yet, and ships prebuilt binaries for Windows, macOS and Linux on x86-64 and ARM64.

A local debugger for RAG pipelines. When your app gives a wrong answer, SledTrace shows you why: what the retriever returned, what the model was given, what it said, and where those disagree. Everything runs on your machine. No account, no API key, nothing uploaded. pip install sledtrace sledtrace serve Your browser opens the dashboard at http://127.0.0.1:4319 . Now send it a trace. Save this as first trace.py and run it in another terminal: python from sledtrace import trace question = "How many days do customers have to return items after delivery?" with trace name="refund-question", query=question as t: t.retrieval query=question, chunks= {"id": "policy-2024", "text": "Customers can return items within 30 days of delivery.", "score": 0.82, "metadata": {"source": "refund policy.md"}}, {"id": "policy-2021", "text": "Customers can return items within 14 days of delivery.", "score": 0.79, "metadata": {"source": "legacy refund policy.md"}}, , t.llm model="demo-model", prompt=question, response="Customers have 45 days to return items after delivery." print t.flush Refresh the dashboard and open refund-question . SledTrace points out that the two retrieved policies contradict each other, and that the answer's "45 days" isn't supported by either of them: Ready to trace your own app? Follow the 5-minute quickstart https://github.com/Schromeo/SledTrace/blob/main/docs/QUICKSTART.md . | Warning | Meaning | |---|---| | no retrieved chunks | The retriever returned nothing usable | | low retrieval score | Even the best chunk scored low | | duplicate chunks | The same text was retrieved more than once | | weak query chunk overlap | Top chunks barely mention the question's key terms | | conflicting chunks | Retrieved chunks disagree with each other | | numeric mismatch | A number in the answer contradicts the retrieved context | | answer not grounded | A claim in the answer is weakly supported by the context | Every warning shows the evidence behind it and what to check next. The rules are deterministic heuristics that run locally; no LLM judges your data. See warning rules https://github.com/Schromeo/SledTrace/blob/main/docs/demo/WARNING RULES.md for how each one works and where it falls short. SledTrace also records tool calls, the final task result, timing, and LLM token usage. Values it doesn't know are shown as unknown, never as zero. your Python app ── sledtrace SDK ──▶ local collector ──▶ SQLite │ └──▶ dashboard in your browser You add a few calls to your request path trace , retrieval , llm . The SDK sends each finished trace to the collector that sledtrace serve starts; the collector runs the warning rules and stores everything in ~/.sledtrace/sledtrace.db . - Python only, with explicit calls: there are no automatic LangChain or LlamaIndex integrations yet. - Warnings are heuristics built on English text patterns, not a correctness verdict. - Token usage is recorded only when you pass it for example with sledtrace.openai.record response ; cost estimates are indicative. - Local, single-user tool: no hosting, authentication or team features. - Prebuilt sledtrace serve for Windows, macOS and Linux x86-64 and ARM64 . On other platforms, run from source https://github.com/Schromeo/SledTrace/blob/main/docs/DEVELOPMENT.md . - Quickstart https://github.com/Schromeo/SledTrace/blob/main/docs/QUICKSTART.md : instrument your own RAG app. - Python SDK guide https://github.com/Schromeo/SledTrace/blob/main/docs/integrations/PYTHON SDK GUIDE.md : full API reference. - Warning rules https://github.com/Schromeo/SledTrace/blob/main/docs/demo/WARNING RULES.md : what each warning checks. - Development setup https://github.com/Schromeo/SledTrace/blob/main/docs/DEVELOPMENT.md : run from source, Docker, demos, configuration. - Contributing https://github.com/Schromeo/SledTrace/blob/main/CONTRIBUTING.md and release notes https://github.com/Schromeo/SledTrace/blob/main/docs/releases/V0 8 1.md . - Renaming from RAGLens https://github.com/Schromeo/SledTrace/blob/main/docs/REBRANDING.md : raglens imports still work. Named after my husky. A RAG pipeline is like a sled team: retrievers, rerankers and LLMs all pulling together. When the sled goes off course, you read the tracks in the snow to find out which dog stumbled. SledTrace shows you the tracks.