{"slug": "show-hn-socbench-an-open-benchmark-for-ai-on-soc-tasks", "title": "Show HN: SOCBench – an open benchmark for AI on SoC tasks", "summary": "DeepTempo released SOCBench, an open benchmark for evaluating frontier reasoning LLMs as Security Operations Center (SOC) agents on raw NetFlow data. The benchmark supports multiple personas, providers, and a local-first reproducible pipeline with a $10 smoke test budget.", "body_md": "Benchmark frontier reasoning LLMs as\n\nSOC agentson raw NetFlow data.\n\n`socbench`\n\nbenchmarks frontier reasoning models as SOC agents:\neach model runs a bounded multi-turn agent loop against a deterministic,\npre-indexed NetFlow corpus, with persona-scoped read-only tools, fixed dollar\ncaps per investigation, and a strict final-answer JSON contract. Four personas\n(SOC Analyst, Threat Analyst, Adversary Hunter, Detection Engineer) and three\nproviders (OpenAI, Anthropic, Google) share the same eval units, scoring\nlenses, and ablation surface so the headline numbers and `tools_off`\n\n/\n`playbooks_off`\n\ndeltas are directly comparable.\n\nThe repository is **local-first**. A laptop, three API keys, and a sample\nparquet committed to the repo are enough to reproduce a smoke under a $10\nbudget.\n\nAlpha. The full pipeline runs end-to-end. Build-out covered:\n\n**Step 1**: package skeleton, contracts, configs, schema** Step 2**: the index builder (`socbench build-index`\n\n) with deterministic content-addressed indexes**Step 3**: read-only tools layer with persona allowlist + sample builder** Step 4**: personas, playbooks, prompt compose + forbidden-token check** Step 5**: provider adapters (OpenAI / Anthropic / Gemini + always-on mock) and the multi-turn agent loop with budget caps and cost/latency rollups**Step 6**: scoring (per-flow / per-pair / per-host F1), stratified sampling, ablation aggregation** Step 7**: quickstart + results-explorer notebooks; reproduction instructions in`REPRODUCE.md`\n\nYou can run a complete smoke today with **no API keys** via the mock provider\n(see Quickstart step 3, or `notebooks/quickstart.ipynb`\n\n).\n\n`socbench`\n\nships as a standard PEP 621 / hatchling project. Either install\npath works.\n\n```\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n\ngit clone https://github.com/DeepTempo/socbench.git\ncd socbench\n\nuv venv --python 3.11\nsource .venv/bin/activate\nuv pip install -e \".[dev,providers]\"\ngit clone https://github.com/DeepTempo/socbench.git\ncd socbench\n\npython3.11 -m venv .venv\nsource .venv/bin/activate\npip install -e \".[dev,providers]\"\n```\n\nEither way, `socbench --help`\n\nshould now list the available subcommands.\n\n| Surface | Default | Lives in |\n|---|---|---|\n| Benchmark defaults (sampling, agent budgets, providers, persona × tool matrix) | `benchmark_config.yaml` |\n`config/` |\n| Canonical NetFlow schema + normalization aliases | `schema.json` |\n`config/` |\n| Provider pricing snapshot (USD per 1M tokens) | `pricing.yaml` |\n`config/` |\n| Provider API keys | env vars `OPENAI_API_KEY` , `ANTHROPIC_API_KEY` , `GOOGLE_API_KEY` |\nshell env |\n\n`config/benchmark_config.yaml`\n\nships safe defaults: smoke `cost_budget_usd: 10`\n\n,\nfull `cost_budget_usd: 900`\n\n, fixed `cost_usd_cap_per_rendering: 0.50`\n\n. Paths\ninside it that point at sibling config files (`schema_path`\n\n, `pricing_path`\n\n)\nresolve relative to the YAML's own directory, so renaming or relocating\n`config/`\n\ndoesn't require any code edits.\n\n```\nsocbench build-index \\\n  --config config/benchmark_config.yaml \\\n  --dataset sample\n```\n\nThis normalizes the parquet against `config/schema.json`\n\n, sorts globally by\n`ts_start`\n\nwith deterministic tie-breaking, assigns stable `flow_id`\n\ns,\nderives `pair_timeline`\n\n/ `host_egress`\n\neval units, computes\nrollups, and writes to `indexes/<dataset_hash>/`\n\n.\n\nRe-running the command on the same data is a no-op. Pass `--rebuild`\n\nto\nforce a rebuild.\n\n```\nsocbench tools-smoke \\\n  --dataset-hash <dataset_hash> \\\n  --persona soc_analyst\n```\n\nThis invokes every tool in the persona's allowlist against the built index and prints a summary, with no model calls.\n\n```\n# Free, deterministic, no API keys (the mock provider):\nsocbench run --dataset-hash <dataset_hash> --providers mock --personas all\n\n# Real models (after `pip install -e \".[providers]\"` + exporting API keys):\nsocbench run --dataset-hash <dataset_hash> --providers all --personas all\n```\n\nUnit selection defaults to stratified sampling, deterministic in\n`(dataset_hash, sample_seed, mode)`\n\n. Each `(unit × persona × provider)`\n\nrendering runs a bounded multi-turn agent loop; results land under\n`runs/<run_id>/`\n\nwith `summary.json`\n\n(scoring + cost + cache rollups),\n`eval_units_summary.jsonl`\n\n, `predictions_raw.jsonl`\n\n, `renderings.jsonl`\n\n,\n`tool_calls.jsonl`\n\n, and `prompts_used/`\n\n.\n\n```\nsocbench run --dataset-hash <dataset_hash> --ablation tools_off --providers mock --personas all\nsocbench aggregate --dataset-hash <dataset_hash>\n# → ablations/<dataset_hash>/<seed>/ablation_summary.json  (tools_off → main deltas)\n```\n\n`notebooks/quickstart.ipynb`\n\nruns the whole loop (it synthesizes a sample\ndataset so it needs no committed data) and plots per-persona F1.\n`notebooks/results_explorer.ipynb`\n\nloads any `runs/<run_id>/`\n\nand slices the\nresults by stratum, persona, and provider. Install with\n`pip install -e \".[notebooks]\"`\n\n.\n\nEvery interface designed to evolve is a registry or a YAML key:\n\n**New tool**: drop a new file under`src/socbench/tools/catalog/<name>.py`\n\nwith a`Tool`\n\nsubclass, register it in`src/socbench/tools/catalog/__init__.py`\n\nby appending to`ALL_TOOLS`\n\n, then add its name to the appropriate persona`tools:`\n\nlists in`config/benchmark_config.yaml`\n\n. The`tools_manifest_sha`\n\nshifts automatically. Filename, YAML name, and matrix entry are 1:1 by design.**New eval-unit type**: add an assigner to`src/socbench/index.py`\n\nand a matching`Literal`\n\nto`EvalUnitType`\n\nin`src/socbench/models.py`\n\n.**New provider adapter**: implement the`Adapter`\n\nABC in a new`src/socbench/providers/<name>_adapter.py`\n\n, register it in the`build_adapter`\n\nfactory in`providers/base.py`\n\n, and add an entry under`providers:`\n\nin`config/benchmark_config.yaml`\n\n. Pricing goes in`config/pricing.yaml`\n\n. SDK imports stay lazy so the dependency is optional.**New persona**: add a block under`agent.personas:`\n\nin`config/benchmark_config.yaml`\n\nwith its budget and`tools:`\n\nallowlist.**New scoring lens**: add a lens to`score_unit`\n\nin`src/socbench/scoring.py`\n\nand a matching field to`EvalUnitSummary`\n\nin`models.py`\n\n.**New ablation**: extend the`Ablation`\n\nhandling in`prompts.py`\n\n/`agent.py`\n\nand the tag list in`aggregate.py`\n\n.\n\nThe full methodology (eval units, persona x tool matrix, agent loop, scoring,\ncost model, repair policy, sampling, ablations, run artifacts) is implemented\nacross the module-level files in `src/socbench/`\n\n(each carries a focused module\ndocstring).\n\nApache-2.0. See [LICENSE](/DeepTempo/socbench/blob/main/LICENSE).", "url": "https://wpnews.pro/news/show-hn-socbench-an-open-benchmark-for-ai-on-soc-tasks", "canonical_source": "https://github.com/DeepTempo/socbench", "published_at": "2026-07-07 15:25:32+00:00", "updated_at": "2026-07-07 15:30:11.032073+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-research", "ai-agents"], "entities": ["DeepTempo", "SOCBench", "OpenAI", "Anthropic", "Google", "NetFlow"], "alternates": {"html": "https://wpnews.pro/news/show-hn-socbench-an-open-benchmark-for-ai-on-soc-tasks", "markdown": "https://wpnews.pro/news/show-hn-socbench-an-open-benchmark-for-ai-on-soc-tasks.md", "text": "https://wpnews.pro/news/show-hn-socbench-an-open-benchmark-for-ai-on-soc-tasks.txt", "jsonld": "https://wpnews.pro/news/show-hn-socbench-an-open-benchmark-for-ai-on-soc-tasks.jsonld"}}