I asked my docs chatbot how to change the repository password. It answered with confidence, and the answer was about installing restic and choosing a strong password. It never said how to change one. That bothered me because a confident wrong answer is worse than "I don't know", especially when a mistake with a password can mean losing a backup.
text
$ python -m docpilot ask "How do I change the repository password?"
Sources: tasks/create-repository.md#before-you-begin, concepts/repositories-and-snapshots.md#why-the-password-matters, tasks/check-repository.md#steps
(confidence 0.62)
DocPilot is a small docs-as-code toolkit I built to document restic, an open-source backup tool. The chatbot splits each docs page into sections, finds the sections closest to your question, and answers only from that text, with a link to the source.
My first version ranked sections by TF-IDF cosine similarity and accepted anything above a low score of 0.12. The four questions the docs can't answer scored between 0.32 and 0.48, so the bot tried to answer all of them. Not one was rejected.
The problem was that words like "repository" and "backup" appear on almost every page. They pushed every score up, even when the real subject of the question was missing.
So I switched to scoring coverage: how much of the question is actually found in the best-matching sections. Rare words count for more. If the docs never use a word in the question, like "Azure" or "cron", the score drops sharply.
text
$ python -m docpilot ask "How do I schedule automatic backups with cron?"
I could not find this in the docs. This question is logged as a content gap.
(confidence 0.15) text
$ python -m docpilot ask "How do I delete old snapshots?"
The forget command removes snapshot records. The prune command deletes stored data that no snapshot uses anymore. The --prune flag runs both in one step.
Sources: tasks/prune-old-snapshots.md#forget-and-prune, tasks/prune-old-snapshots.md, tasks/prune-old-snapshots.md#before-you-begin
(confidence 1.00)
The threshold is 0.5. Below it, the bot says it can't find the answer and writes the question to a gap log, so I know what to document next.
To test this, I wrote 14 questions: 10 the docs answer and 4 they don't. One more fix came out of testing. "What is a snapshot?" pulled up a how-to page instead of the concept page, so I made "what" and "why" questions favor concept pages and "how" questions favor task pages.
text
Answerable questions: 10
Right topic ranked first: 100%
Unanswerable questions correctly rejected: 75%
The password question still scores about 0.6, so the bot still answers it. I think this happens because every word in the question ("repository", "password", "change") appears somewhere in the docs. My score measures word overlap, not meaning, so it can't tell that no page explains how to change a password. At the time of writing I haven't written that topic. The evaluation reports the failure instead of hiding it, and the real fix is a new page. A later fix would be embeddings, which compare meaning instead of words.
Three things I'd tell other docs teams:
DocPilot is on GitHub: [https://github.com/shravanigalande/docpilot](https://github.com/shravanigalande/docpilot)
Docs site: [https://shravanigalande.github.io/docpilot/](https://shravanigalande.github.io/docpilot/)