Your LLM red-team report should be reproducible, not a screenshot

A developer has released a red-teaming kit that makes LLM safety evaluations reproducible by pinning a versioned probe corpus, raw prompts and replies, and a sha256 hash of the corpus, scoring results 0-100 for use as a CI gate. The kit ships with 35 probes spanning 17 attack classes, including role-play, encoding hops, indirect injection, system-prompt extraction, and tool abuse, each with a per-class remediation note. A free 8-probe scan is offered with no account required, aimed at endpoints the tester is authorised to use.

Most "we red-teamed our model" posts I see are a screenshot of a bad output and a shrug. That is not a test you can run again, a test another engineer can verify, or a test that fails your CI. It is a story. The difference is small but it changes everything: pin the exact probe, pin the exact prompt, and keep the raw model reply. Then the report is a function of probes, model, prompt and anyone can re-run it. Here is the minimum bar for a red-team report I will actually trust: A fixed, versioned probe corpus. Not "we tried some jailbreaks." A numbered list of probes, each with a name and an intent. When you swap models, the corpus stays the same so the delta is the model, not your testing. Raw prompt + raw reply per probe. The moment you paraphrase, the evidence is gone. Keep the exact bytes the model saw and the exact bytes it returned. That is the difference between "I think it jailbroke" and "here is the 200-byte prompt that did it." A hash you can check. If you ship a report to a customer or a compliance person, attach the sha256 of the probe corpus. They can diff it. This is the whole game in one line: trust is replaced by a checksum. A score that means something. 0-100, but derived from which of your N probes flipped, not vibes. A CI gate then becomes a number: block the deploy at threshold. I built this as a small kit: 35 probes across 17 attack classes role-play, encoding hops, indirect injection, system-prompt extraction, tool abuse , each returning the raw exchange, with a per-class remediation note and a 0-100 score. The free tier is an 8-probe scan you can point at an endpoint you are authorised to test, no account. The point is not the 35 probes. The point is that "we tested it" stops being a claim and becomes a receipt. If you cannot re-run your red-team report tomorrow, you did not have a report. Free 8-probe scan no signup : https://llmrt-companion.manhliemcn4euwlu.workers.dev/review https://llmrt-companion.manhliemcn4euwlu.workers.dev/review