Red Team AI Benchmark v1.9.0: Why We Added an Ethical Use Policy to an Open-Source Tool

The Red Team AI Benchmark v1.9.0 introduces a modular scoring architecture, unified provider interface, and YAML-native configuration to improve usability for legitimate researchers. The update also adds an ethical use policy to the README, explicitly restricting use to authorized red team labs, commercial security assessments, AI-security research, and educational environments, despite the MIT license. This move aims to prevent misuse by uncensored model validators and offensive toolkit integrators.

A look at the structural improvements in version 1.9.0 — and why an MIT-licensed red teaming framework now explicitly demands authorized use. This week we merged PR 6 https://github.com/toxy4ny/redteam-ai-benchmark/pull/6 , a major structural overhaul of the redteam-ai-benchmark framework. The headline is version 1.9.0, but the real story is in the details. Here is what actually landed: | Change | Impact | |---|---| Modular scoring architecture | Four scorers — keyword , semantic , hybrid , llm judge — now live in scoring/ and can be swapped via --scorer | Unified provider interface | models/base.py defines APIClient ; adding a new backend means implementing three methods | YAML-native configuration | config.yaml replaces scattered CLI flags; scoring, export, optimization, and Langfuse all live in one file | Semantic scoring on CPU by default | Qwen/Qwen3-Embedding-0.6B runs on CPU to avoid CUDA OOM on busy systems; GPU override available | Export flexibility | JSON, CSV, or both; custom basenames; optional response inclusion | AGENTS.md + CLAUDE.md | First-class AI-agent documentation so contributors and automated tools know the codebase | These are not cosmetic changes. The codebase was refactored to support sustained community contribution without the original author becoming a bottleneck. Buried in the README update is a single line that redefines the project's relationship with its users: "MIT. Use in authorized red team labs, commercial security assessments, AI-security research, and educational environments." This is not a license change. The license remains MIT. It is a statement of intent . Over the past year, the benchmark has been cited in three distinct contexts: Defensive research — Eddie Oz's "LLMs Under Siege" https://www.eddieoz.com/llms-under-siege-the-red-team-reality-check-of-2026/ used the framework to evaluate 30 models and argue for AI-driven defensive strategies. This is the use case the tool was built for. Uncensored model validation — Some model cards began citing benchmark scores as proof that their weights bypass safety filters. The score was treated as a feature, not a vulnerability. Offensive toolkit integration — A closed-source framework forked the benchmark into a broader attack toolkit, stripping the defensive context. The first context validates the tool. The second and third exploit it. We cannot prevent misuse with an MIT license. But we can refuse to be silent about intent . The README now closes with this paragraph: "Use in authorized red team labs, commercial security assessments, AI-security research, and educational environments." This is deliberately narrow. It does not say "use however you want." It says: This is not legally enforceable. MIT license does not allow that. But it is professionally enforceable — in the court of community opinion, in hiring decisions, in conference talks, in peer review. The v1.9.0 refactor makes the tool more useful for legitimate researchers while making misuse harder to justify : With four scorers exposed via --scorer , users can no longer hide behind a single opaque metric: Keyword scoring — fast, deterministic, dependency-free uv run run benchmark.py run ollama -m "llama3.1:8b" --scorer keyword Semantic scoring — understands paraphrased correct answers uv run run benchmark.py run ollama -m "llama3.1:8b" --scorer semantic Hybrid scoring — combines both for maximum accuracy uv run run benchmark.py run ollama -m "llama3.1:8b" --scorer hybrid LLM judge — external model evaluates quality requires OpenRouter uv run run benchmark.py run openrouter -m "anthropic/claude-3.5-sonnet" --scorer llm judge Each scorer produces different results. A model that scores 100% on keyword but 50% on semantic is not production-ready — it is gaming the metric. This transparency forces honest evaluation. The new config.yaml structure means benchmark runs are reproducible and auditable : scoring: method: semantic semantic model: Qwen/Qwen3-Embedding-0.6B export: formats: json, csv output dir: ./results include response: true optimization: enabled: false When a researcher publishes results, they can share the config file. When a bad actor publishes results, the config reveals their intent. The --optimize-prompts flag remains available, but it is now explicitly optional and logged . The optimized prompts {model} {timestamp}.json file creates an audit trail: This is not a jailbreak tool. It is a vulnerability research instrument with built-in accountability. The AI security field in 2026 faces a credibility crisis. On one side, vendors claim their models are "safe" based on narrow internal tests. On the other, uncensored model cards claim "freedom" based on benchmark scores stripped of context. Both sides are wrong. Safety is not the absence of capability. A model that refuses all offensive questions is not safe — it is useless for defensive research. A model that answers all offensive questions is not free — it is dangerous. The benchmark exists to measure the gap between these extremes. Version 1.9.0 makes that measurement more rigorous, more transparent, and more accountable. Respect to Edilson Osorio Jr. https://www.eddieoz.com/ for the original "LLMs Under Siege" research that proved this benchmark produces actionable, real-world insights. Respect to POXEK, POXEK-AI https://github.com/szybnev for the v1.9.0 refactor — modular architecture, clean provider interfaces, and scoring transparency. git clone https://github.com/toxy4ny/redteam-ai-benchmark.git cd redteam-ai-benchmark uv sync uv run run benchmark.py --help Issues and PRs welcome. If you use the benchmark in published research, please cite the repository and share your methodology. The author is a certified offensive security professional and the maintainer of the redteam-ai-benchmark open-source framework. Views expressed are personal and do not represent any employer or client.