Promptfoo 0.123.0 can be wired into GitHub Actions via promptfoo/promptfoo-action@v1 to score prompts against fixed test cases on every pull request, failing the PR when an edit breaks tone, policy ru…
Promptfoo 0.122.2 can be wired into GitHub Actions to run an eval suite against a golden dataset on every pull request that touches prompts, failing the CI check when outputs regress and posting a bef…
A tutorial by Rachel Goldstein demonstrates how to regression-test LLM prompts using promptfoo 0.122.2 in GitHub Actions, catching prompt and model regressions on every pull request. The setup include…
A developer's guide to AI red teaming in 2026 distinguishes between risk management frameworks like NIST AI 100-1 and practical tools such as garak, PyRIT, and promptfoo. The article emphasizes that f…
Microsoft's Python Risk Identification Tool (PyRIT) was archived on GitHub on March 27, 2026, making it read-only with no further updates. The developer behind the comparison guide recommends alternat…
A developer compared five open-source tools for testing LLM applications against prompt injection: promptfoo, garak, Giskard, PyRIT, and sentinel-scan-cli. The review found that promptfoo, Giskard, an…
A developer has identified a critical flaw in AI agents that report task completion without verifying the actual state of the world, calling it the 'nastiest species of hallucination.' They propose a …
A developer built a tool to measure whether multi-agent scaffolding improves coding task performance, only to find that adding a planner, two drafters, and a judge made results worse (80% vs. 95%) at …
A developer found that an LLM judge gave opposite results for the same eval run on consecutive days due to position bias, one of three systematic biases documented in the 2023 paper "Judging LLM-as-a-…
Probelock, a new open-source tool, provides a lockfile for LLM tool calling that records model capabilities and fails CI when a model or quant swap lowers scores. It derives probes from existing tool …
A developer maintaining bulwark-mcp, an open-source proxy that scans MCP tool results for indirect prompt injection, discovered that a class of attacks bypasses the detector by impersonating the secur…