Benchmark frontier reasoning LLMs as
SOC agentson raw NetFlow data.
socbench
benchmarks frontier reasoning models as SOC agents:
each model runs a bounded multi-turn agent loop against a deterministic,
pre-indexed NetFlow corpus, with persona-scoped read-only tools, fixed dollar
caps per investigation, and a strict final-answer JSON contract. Four personas
(SOC Analyst, Threat Analyst, Adversary Hunter, Detection Engineer) and three
providers (OpenAI, Anthropic, Google) share the same eval units, scoring
lenses, and ablation surface so the headline numbers and tools_off
/
playbooks_off
deltas are directly comparable.
The repository is local-first. A laptop, three API keys, and a sample parquet committed to the repo are enough to reproduce a smoke under a $10 budget.
Alpha. The full pipeline runs end-to-end. Build-out covered:
Step 1: package skeleton, contracts, configs, schema** Step 2**: the index builder (socbench build-index
) with deterministic content-addressed indexesStep 3: read-only tools layer with persona allowlist + sample builder** Step 4**: personas, playbooks, prompt compose + forbidden-token check** Step 5**: provider adapters (OpenAI / Anthropic / Gemini + always-on mock) and the multi-turn agent loop with budget caps and cost/latency rollupsStep 6: scoring (per-flow / per-pair / per-host F1), stratified sampling, ablation aggregation** Step 7**: quickstart + results-explorer notebooks; reproduction instructions inREPRODUCE.md
You can run a complete smoke today with no API keys via the mock provider
(see Quickstart step 3, or notebooks/quickstart.ipynb
).
socbench
ships as a standard PEP 621 / hatchling project. Either install path works.
curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/DeepTempo/socbench.git
cd socbench
uv venv --python 3.11
source .venv/bin/activate
uv pip install -e ".[dev,providers]"
git clone https://github.com/DeepTempo/socbench.git
cd socbench
python3.11 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,providers]"
Either way, socbench --help
should now list the available subcommands.
| Surface | Default | Lives in |
|---|---|---|
| Benchmark defaults (sampling, agent budgets, providers, persona × tool matrix) | benchmark_config.yaml |
|
config/ |
||
| Canonical NetFlow schema + normalization aliases | schema.json |
|
config/ |
||
| Provider pricing snapshot (USD per 1M tokens) | pricing.yaml |
|
config/ |
||
| Provider API keys | env vars OPENAI_API_KEY , ANTHROPIC_API_KEY , GOOGLE_API_KEY |
|
| shell env |
config/benchmark_config.yaml
ships safe defaults: smoke cost_budget_usd: 10
,
full cost_budget_usd: 900
, fixed cost_usd_cap_per_rendering: 0.50
. Paths
inside it that point at sibling config files (schema_path
, pricing_path
)
resolve relative to the YAML's own directory, so renaming or relocating
config/
doesn't require any code edits.
socbench build-index \
--config config/benchmark_config.yaml \
--dataset sample
This normalizes the parquet against config/schema.json
, sorts globally by
ts_start
with deterministic tie-breaking, assigns stable flow_id
s,
derives pair_timeline
/ host_egress
eval units, computes
rollups, and writes to indexes/<dataset_hash>/
.
Re-running the command on the same data is a no-op. Pass --rebuild
to force a rebuild.
socbench tools-smoke \
--dataset-hash <dataset_hash> \
--persona soc_analyst
This invokes every tool in the persona's allowlist against the built index and prints a summary, with no model calls.
socbench run --dataset-hash <dataset_hash> --providers mock --personas all
socbench run --dataset-hash <dataset_hash> --providers all --personas all
Unit selection defaults to stratified sampling, deterministic in
(dataset_hash, sample_seed, mode)
. Each (unit × persona × provider)
rendering runs a bounded multi-turn agent loop; results land under
runs/<run_id>/
with summary.json
(scoring + cost + cache rollups),
eval_units_summary.jsonl
, predictions_raw.jsonl
, renderings.jsonl
,
tool_calls.jsonl
, and prompts_used/
.
socbench run --dataset-hash <dataset_hash> --ablation tools_off --providers mock --personas all
socbench aggregate --dataset-hash <dataset_hash>
notebooks/quickstart.ipynb
runs the whole loop (it synthesizes a sample
dataset so it needs no committed data) and plots per-persona F1.
notebooks/results_explorer.ipynb
loads any runs/<run_id>/
and slices the
results by stratum, persona, and provider. Install with
pip install -e ".[notebooks]"
.
Every interface designed to evolve is a registry or a YAML key:
New tool: drop a new file undersrc/socbench/tools/catalog/<name>.py
with aTool
subclass, register it insrc/socbench/tools/catalog/__init__.py
by appending toALL_TOOLS
, then add its name to the appropriate personatools:
lists inconfig/benchmark_config.yaml
. Thetools_manifest_sha
shifts automatically. Filename, YAML name, and matrix entry are 1:1 by design.New eval-unit type: add an assigner tosrc/socbench/index.py
and a matchingLiteral
toEvalUnitType
insrc/socbench/models.py
.New provider adapter: implement theAdapter
ABC in a newsrc/socbench/providers/<name>_adapter.py
, register it in thebuild_adapter
factory inproviders/base.py
, and add an entry underproviders:
inconfig/benchmark_config.yaml
. Pricing goes inconfig/pricing.yaml
. SDK imports stay lazy so the dependency is optional.New persona: add a block underagent.personas:
inconfig/benchmark_config.yaml
with its budget andtools:
allowlist.New scoring lens: add a lens toscore_unit
insrc/socbench/scoring.py
and a matching field toEvalUnitSummary
inmodels.py
.New ablation: extend theAblation
handling inprompts.py
/agent.py
and the tag list inaggregate.py
.
The full methodology (eval units, persona x tool matrix, agent loop, scoring,
cost model, repair policy, sampling, ablations, run artifacts) is implemented
across the module-level files in src/socbench/
(each carries a focused module docstring).
Apache-2.0. See LICENSE.