Track what AI models recommend—and see when the answer changes.
Your brand, product, or category can look different depending on which model someone asks. Answerwatch makes that visible: run the same prompt across models, compare their recommendations, and keep a local audit trail as answers drift.
Everything stays local in SQLite. The only external call is the model request you choose to make.
┌─ OpenAI
prompts ───────────├─ Anthropic ──► ranked entities ──► agreement + drift
└─ Google │
▼
static HTML report
Requires Python 3.10 or newer.
python -m pip install -e .
answerwatch init
answerwatch run --demo
answerwatch compare
answerwatch report
Open the HTML path printed by the last command. The demo makes no network requests and spends no API credits. It produces a polished dashboard with agreement, consensus, rank gaps, citation domains, and raw answers.
| You need to know | Answerwatch shows |
|---|---|
| Do models agree? | Pairwise recommendation-agreement matrix |
| What does only one model mention? | Unique recommendations per model |
| Is ranking materially different? | Largest rank gaps for shared entities |
| Did the answer change this week? | Run-to-run drift with additions and removals |
| Can I trust the analysis? | Raw-response audit trail and extraction evidence |
Answerwatch uses OpenRouter so one API key can query models from different providers.
export OPENROUTER_API_KEY="..."
answerwatch run
answerwatch compare
answerwatch report
answerwatch init
creates:
models:
- openai/gpt-5.2
- anthropic/claude-sonnet-4.5
- google/gemini-3-flash-preview
prompts: sample-prompts.yml
database: .answerwatch/answerwatch.db
settings:
temperature: 0.2
max_tokens: 1200
retries: 2
prompts:
- id: best-running-shoes
text: What are the best running shoes for beginners?
entity: product
Use model identifiers currently available in OpenRouter.
answerwatch init
answerwatch run [--demo]
answerwatch compare [RUN_ID] [--baseline RUN_ID|previous]
answerwatch report [RUN_ID] [--baseline RUN_ID|previous] [--output report.html]
Without a run ID, compare
and report
use the latest completed run.
To see recommendation drift, run once, change time or model settings, then:
answerwatch compare --baseline previous
answerwatch report --baseline previous
Version 0.1 asks models to put named recommendations first in a numbered list. A deterministic local parser extracts those list items. Both the raw answer and the exact extracted evidence are retained in SQLite.
This keeps the first release fast, private, and auditable. It does not claim that generated URLs are verified citations.
The default database and reports live under .answerwatch/
. Delete that directory to remove local run history.
PYTHONPATH=src python -m unittest discover -s tests -v
Please read CONTRIBUTING.md before opening a pull request. We use Conventional Commit-style messages, for example:
feat(report): surface consensus recommendations
fix(extract): preserve linked entity names
docs: clarify live model setup
Now — local runs, recommendation comparison, static reports, and OpenRouter support.
Next — stronger entity normalization, structured provider citations, and scheduled local runs.
Later — optional hosted history, alerts, teams, and branded reports.
See ROADMAP.md for the intended boundaries.
MIT