A developer built a tool to measure whether multi-agent scaffolding improves coding task performance, only to find that adding a planner, two drafters, and a judge made results worse (80% vs. 95%) at …
A developer found that an LLM judge gave opposite results for the same eval run on consecutive days due to position bias, one of three systematic biases documented in the 2023 paper "Judging LLM-as-a-…
Probelock, a new open-source tool, provides a lockfile for LLM tool calling that records model capabilities and fails CI when a model or quant swap lowers scores. It derives probes from existing tool …
A developer maintaining bulwark-mcp, an open-source proxy that scans MCP tool results for indirect prompt injection, discovered that a class of attacks bypasses the detector by impersonating the secur…