A look at the structural improvements in version 1.9.0 — and why an MIT-licensed red teaming framework now explicitly demands authorized use.
This week we merged PR #6, a major structural overhaul of the redteam-ai-benchmark
framework. The headline is version 1.9.0, but the real story is in the details.
Here is what actually landed:
| Change | Impact |
|---|---|
| Modular scoring architecture | |
Four scorers — keyword , semantic , hybrid , llm_judge — now live in scoring/ and can be swapped via --scorer |
|
| Unified provider interface | |
models/base.py defines APIClient ; adding a new backend means implementing three methods |
|
| YAML-native configuration | |
config.yaml replaces scattered CLI flags; scoring, export, optimization, and Langfuse all live in one file |
|
| Semantic scoring on CPU by default | |
Qwen/Qwen3-Embedding-0.6B runs on CPU to avoid CUDA OOM on busy systems; GPU override available |
|
| Export flexibility | |
| JSON, CSV, or both; custom basenames; optional response inclusion | |
| AGENTS.md + CLAUDE.md | |
| First-class AI-agent documentation so contributors and automated tools know the codebase |
These are not cosmetic changes. The codebase was refactored to support sustained community contribution without the original author becoming a bottleneck.
Buried in the README update is a single line that redefines the project's relationship with its users:
"MIT. Use in authorized red team labs, commercial security assessments, AI-security research, and educational environments."
This is not a license change. The license remains MIT. It is a statement of intent.
Over the past year, the benchmark has been cited in three distinct contexts:
Defensive research — Eddie Oz's "LLMs Under Siege" used the framework to evaluate 30 models and argue for AI-driven defensive strategies. This is the use case the tool was built for.
Uncensored model validation — Some model cards began citing benchmark scores as proof that their weights bypass safety filters. The score was treated as a feature, not a vulnerability.
Offensive toolkit integration — A closed-source framework forked the benchmark into a broader attack toolkit, stripping the defensive context.
The first context validates the tool. The second and third exploit it.
We cannot prevent misuse with an MIT license. But we can refuse to be silent about intent.
The README now closes with this paragraph:
"Use in authorized red team labs, commercial security assessments, AI-security research, and educational environments."
This is deliberately narrow. It does not say "use however you want." It says:
This is not legally enforceable. MIT license does not allow that. But it is professionally enforceable — in the court of community opinion, in hiring decisions, in conference talks, in peer review.
The v1.9.0 refactor makes the tool more useful for legitimate researchers while making misuse harder to justify:
With four scorers exposed via --scorer
, users can no longer hide behind a single opaque metric:
uv run run_benchmark.py run ollama -m "llama3.1:8b" --scorer keyword
uv run run_benchmark.py run ollama -m "llama3.1:8b" --scorer semantic
uv run run_benchmark.py run ollama -m "llama3.1:8b" --scorer hybrid
uv run run_benchmark.py run openrouter -m "anthropic/claude-3.5-sonnet" --scorer llm_judge
Each scorer produces different results. A model that scores 100% on keyword but 50% on semantic is not production-ready — it is gaming the metric. This transparency forces honest evaluation.
The new config.yaml
structure means benchmark runs are reproducible and auditable:
scoring:
method: semantic
semantic_model: Qwen/Qwen3-Embedding-0.6B
export:
formats: [json, csv]
output_dir: ./results
include_response: true
optimization:
enabled: false
When a researcher publishes results, they can share the config file. When a bad actor publishes results, the config reveals their intent.
The --optimize-prompts
flag remains available, but it is now explicitly optional and logged. The optimized_prompts_{model}_{timestamp}.json
file creates an audit trail:
This is not a jailbreak tool. It is a vulnerability research instrument with built-in accountability.
The AI security field in 2026 faces a credibility crisis. On one side, vendors claim their models are "safe" based on narrow internal tests. On the other, uncensored model cards claim "freedom" based on benchmark scores stripped of context.
Both sides are wrong.
Safety is not the absence of capability. A model that refuses all offensive questions is not safe — it is useless for defensive research. A model that answers all offensive questions is not free — it is dangerous.
The benchmark exists to measure the gap between these extremes. Version 1.9.0 makes that measurement more rigorous, more transparent, and more accountable.
Respect to Edilson Osorio Jr. for the original "LLMs Under Siege" research that proved this benchmark produces actionable, real-world insights.
Respect to POXEK, POXEK-AI for the v1.9.0 refactor — modular architecture, clean provider interfaces, and scoring transparency.
git clone https://github.com/toxy4ny/redteam-ai-benchmark.git
cd redteam-ai-benchmark
uv sync
uv run run_benchmark.py --help
Issues and PRs welcome. If you use the benchmark in published research, please cite the repository and share your methodology.
The author is a certified offensive security professional and the maintainer of the redteam-ai-benchmark open-source framework. Views expressed are personal and do not represent any employer or client.