Answerwatch – Track changes in what AI models recommend

Answerwatch, a new open-source tool, lets users track changes in what AI models recommend by running the same prompt across models like OpenAI, Anthropic, and Google, storing results locally in SQLite, and generating static HTML reports showing agreement, rank gaps, and drift. Version 0.1 requires Python 3.10+, uses OpenRouter for multi-provider access, and keeps all data private with no external calls except model requests.

Track what AI models recommend—and see when the answer changes. Your brand, product, or category can look different depending on which model someone asks. Answerwatch makes that visible: run the same prompt across models, compare their recommendations, and keep a local audit trail as answers drift. Everything stays local in SQLite. The only external call is the model request you choose to make. ┌─ OpenAI prompts ───────────├─ Anthropic ──► ranked entities ──► agreement + drift └─ Google │ ▼ static HTML report Requires Python 3.10 or newer. python -m pip install -e . answerwatch init answerwatch run --demo answerwatch compare answerwatch report Open the HTML path printed by the last command. The demo makes no network requests and spends no API credits. It produces a polished dashboard with agreement, consensus, rank gaps, citation domains, and raw answers. | You need to know | Answerwatch shows | |---|---| | Do models agree? | Pairwise recommendation-agreement matrix | | What does only one model mention? | Unique recommendations per model | | Is ranking materially different? | Largest rank gaps for shared entities | | Did the answer change this week? | Run-to-run drift with additions and removals | | Can I trust the analysis? | Raw-response audit trail and extraction evidence | Answerwatch uses OpenRouter https://openrouter.ai/ so one API key can query models from different providers. export OPENROUTER API KEY="..." answerwatch run answerwatch compare answerwatch report answerwatch init creates: answerwatch.yml models: - openai/gpt-5.2 - anthropic/claude-sonnet-4.5 - google/gemini-3-flash-preview prompts: sample-prompts.yml database: .answerwatch/answerwatch.db settings: temperature: 0.2 max tokens: 1200 retries: 2 sample-prompts.yml prompts: - id: best-running-shoes text: What are the best running shoes for beginners? entity: product Use model identifiers currently available in OpenRouter. answerwatch init answerwatch run --demo answerwatch compare RUN ID --baseline RUN ID|previous answerwatch report RUN ID --baseline RUN ID|previous --output report.html Without a run ID, compare and report use the latest completed run. To see recommendation drift, run once, change time or model settings, then: answerwatch compare --baseline previous answerwatch report --baseline previous Version 0.1 asks models to put named recommendations first in a numbered list. A deterministic local parser extracts those list items. Both the raw answer and the exact extracted evidence are retained in SQLite. This keeps the first release fast, private, and auditable. It does not claim that generated URLs are verified citations. The default database and reports live under .answerwatch/ . Delete that directory to remove local run history. PYTHONPATH=src python -m unittest discover -s tests -v Please read CONTRIBUTING.md /haror1/answerwatch/blob/main/CONTRIBUTING.md before opening a pull request. We use Conventional Commit-style messages, for example: feat report : surface consensus recommendations fix extract : preserve linked entity names docs: clarify live model setup Now — local runs, recommendation comparison, static reports, and OpenRouter support. Next — stronger entity normalization, structured provider citations, and scheduled local runs. Later — optional hosted history, alerts, teams, and branded reports. See ROADMAP.md /haror1/answerwatch/blob/main/ROADMAP.md for the intended boundaries. MIT