{"slug": "network-flow-ml-that-outputs-security-incidents", "title": "Network-flow ML that outputs security incidents", "summary": "Security Anomaly ML, an open-source network-flow detector from developer I. Bondarenko, released v0.1.0, a research/evaluation version that converts CICFlowMeter-compatible traffic into deterministic analyst-facing security incidents. The tool, which uses a frozen Random Forest model (context-rf-v2) and emits incident-v1 JSONL, achieved 98.3686% flow recall, 99.9339% promoted incident recall, and 93.46% promoted incident precision on a February 18 temporal holdout, but the developer warns it is not production-ready due to operational noise.", "body_md": "Open-source ML network-flow detector that turns CICFlowMeter-compatible traffic into deterministic analyst-facing security incidents.\n\n**v0.1.0 — usable research/evaluation release. Not production-ready.**\n\nSecurity Anomaly ML takes unlabeled CICFlowMeter flow records, builds causal temporal context, scores them with a frozen Random Forest detector, groups repeated alerts into incidents, and emits deterministic `incident-v1`\n\nJSONL.\n\nDocker is the recommended path. The published image includes the exact verified frozen model, so no separate model download is needed.\n\n```\ndocker pull ghcr.io/ibondarenko1/security-anomaly-ml:0.1.0\n\ndocker run --rm --network none \\\n  -v \"$PWD:/data\" \\\n  ghcr.io/ibondarenko1/security-anomaly-ml:0.1.0 \\\n  analyze /data/flows.csv \\\n  --output /data/incidents.jsonl\n```\n\nThe mounted directory must be writable by the container's non-root UID/GID `10001`\n\n. Normal inference works with networking disabled. Validation, Windows PowerShell, and source-build examples are in [ docs/DOCKER.md](/ibondarenko1/security-anomaly-ml/blob/main/docs/DOCKER.md).\n\nThe output is newline-delimited `incident-v1`\n\nJSON. Each object contains a deterministic incident ID, first/last timestamps, endpoints, destination port, protocols, flow count, aggregate attack scores, and frozen version metadata.\n\n```\n{\n  \"schema_version\": \"incident-v1\",\n  \"incident_id\": \"inc_e05487e368f9e26a2a6939c7be622de508056478d3876c344c77a53ba4214872\",\n  \"first_seen\": \"2026-08-20T13:00:00\",\n  \"last_seen\": \"2026-08-20T13:00:05\",\n  \"src_ip\": \"192.0.2.10\",\n  \"dst_ip\": \"198.51.100.20\",\n  \"dst_port\": 445,\n  \"protocols\": [6, 17],\n  \"flow_count\": 3,\n  \"max_attack_score\": 0.432590909091,\n  \"mean_attack_score\": 0.418368686869,\n  \"promoted\": true,\n  \"product_version\": \"0.1.0\",\n  \"model_version\": \"context-rf-v2\",\n  \"feature_contract\": \"cicflow-v2-128\"\n}\n```\n\nThe public interface emits promoted incidents rather than individual ML predictions, reducing repeated flow alerts into deterministic analyst-facing objects.\n\nThe synthetic regression fixture at [ tests/fixtures/product-v01/flows.csv](/ibondarenko1/security-anomaly-ml/blob/main/tests/fixtures/product-v01/flows.csv) deterministically produces:\n\n- 12 processed flows;\n- 9 flow alerts;\n- 5 aggregated incidents;\n- 2 promoted incidents.\n\nThese numbers test runtime stability; they are **not an accuracy benchmark** and contain no copied research-dataset rows.\n\nThe v0.1 pipeline was frozen before evaluation on the February 18 temporal holdout. No thresholds, features, model parameters, aggregation rules, promotion rules, suppression, or whitelisting were changed after opening it.\n\n| Metric | Locked holdout |\n|---|---|\n| Flow recall | 98.3686% |\n| Flow precision | 67.5499% |\n| Flow FPR | 2.1190% |\n| PR-AUC | 0.898915 |\n| Aggregated incident recall | 99.9917% |\n| Promoted incident recall | 99.9339% |\n| Promoted incident precision | 93.46% |\n| Flow-alert to incident reduction | 83.80% |\n| FP-object reduction | 96.74% |\n\n**Verdict: acceptable but operationally noisy.** This is one future capture day with overlapping hosts/environment from the same dataset and network family. It is not evidence of generalization across arbitrary networks, and the remaining workload is too high for normal Tier-1 production use.\n\nThe holdout was evaluated once after all model, feature, threshold, aggregation, and promotion decisions were frozen.\n\nOne input row represents one network flow. The product validates the label-free CSV, builds tie-safe causal context, scores each flow, groups flow alerts into deterministic five-minute incidents, and emits only promoted incidents.\n\n``` php\nflowchart LR\n    A[\"CICFlowMeter flows\"] --> B[\"Input validation\"]\n    B --> C[\"128 causal features\"]\n    C --> D[\"context-rf-v2 attack score\"]\n    D --> E[\"Flow alert: score >= 0.10\"]\n    E --> F[\"Policy B: src IP + dst IP + dst port / 5 min\"]\n    F --> G[\"Promotion: max score >= 0.25\"]\n    G --> H[\"incident-v1 JSONL\"]\n```\n\nThe frozen feature contract is:\n\n```\n76 CICFlowMeter flow features\n+ 9 static port/protocol behavioral features\n+ 43 causal temporal-context features\n= 128 model features\n```\n\nRaw IP addresses and timestamps provide temporal and incident context but are not model identity features. Same-timestamp flows are processed as one peer group: all peers are featurized before that timestamp updates state.\n\nInput must be a UTF-8 CICFlowMeter-compatible CSV containing:\n\n`Src IP`\n\n,`Src Port`\n\n,`Dst IP`\n\n,`Dst Port`\n\n,`Protocol`\n\n, and`Timestamp`\n\n;- all 76 baseline numeric fields defined by\n;`cicflow-v2-128`\n\n- optional\n`Flow ID`\n\n.\n\n`Label`\n\n, `label`\n\n, and `attack_cat`\n\nare not required and are removed if present. Validation rejects missing or duplicate columns, reserved derived fields, malformed timestamps, invalid ports/protocols, and non-numeric or non-finite model inputs. Source timestamps are timezone-naive; v0.1 does not invent a timezone.\n\nUse `validate`\n\nbefore analysis when integrating a new exporter:\n\n```\nsecurity-anomaly validate flows.csv\n```\n\nThe immutable release tag is:\n\n```\nghcr.io/ibondarenko1/security-anomaly-ml:0.1.0\n```\n\nIt includes Python 3.13, the installed package, contracts, pinned runtime dependencies, and the verified model. It runs as non-root and requires no network during inference. Build and operational details are in [ docs/DOCKER.md](/ibondarenko1/security-anomaly-ml/blob/main/docs/DOCKER.md).\n\nNo mutable `latest`\n\ntag is published for v0.1.0.\n\nDownload the wheel from the [v0.1.0 GitHub Release](https://github.com/ibondarenko1/security-anomaly-ml/releases/tag/v0.1.0), then install it into Python 3.13:\n\n```\npython3.13 -m venv .venv\nsource .venv/bin/activate\npython -m pip install security_anomaly_ml-0.1.0-py3-none-any.whl\nsecurity-anomaly version\n```\n\nThe Python wheel intentionally does not embed the model. Obtain the existing artifact from the [ model-context-rf-v2 release](https://github.com/ibondarenko1/security-anomaly-ml/releases/tag/model-context-rf-v2), or from a source checkout run:\n\n```\npython tools/fetch_frozen_model.py \\\n  --tag model-context-rf-v2 \\\n  --destination models/context-rf-v2.joblib\n\nsecurity-anomaly model-info --model models/context-rf-v2.joblib\nsecurity-anomaly analyze flows.csv \\\n  --model models/context-rf-v2.joblib \\\n  --output incidents.jsonl\n```\n\nThe downloader verifies the frozen SHA-256 before success and never silently replaces a different file. Full CLI behavior and exit codes are documented in [ docs/CLI.md](/ibondarenko1/security-anomaly-ml/blob/main/docs/CLI.md).\n\nPublic CI runs on every pull request and push to `main`\n\nand requires only a clean checkout plus the public frozen-model release. It verifies:\n\n- unit and frozen-contract tests;\n- pinned runtime dependency vulnerability audit;\n- clean wheel/sdist build and outside-checkout installation;\n- public model download and SHA verification;\n- real-model end-to-end golden regression;\n- non-root offline Docker build and byte-identical golden output;\n- exclusion of datasets and research artifacts from the runtime image.\n\nGoldens are never rewritten automatically. Details are in [ docs/CI.md](/ibondarenko1/security-anomaly-ml/blob/main/docs/CI.md).\n\n| Field | Frozen value |\n|---|---|\n| Model version | `context-rf-v2` |\n| Release tag |\n`model-context-rf-v2` |\n\n`context-rf-v2.joblib`\n\n`4730a06506d8c5f2af93679c492e1544b3c2b11acd16fe74120d64d4dbfc5c72`\n\n`3.13.7`\n\n`causal-temporal-v2`\n\n`cicflow-v2-128`\n\n`>= 0.10`\n\n`> 300s`\n\n`max_attack_score >= 0.25`\n\nScores are ranking signals, not calibrated real-world probabilities. The model is excluded from Git history and is never reserialized by the product build.\n\nSecurity Anomaly ML processes sensitive network-flow metadata locally. Normal inference performs no telemetry, cloud upload, hidden download, or other outbound network call; the Docker path is tested with `--network none`\n\n.\n\n- Vulnerability reporting:\n`SECURITY.md`\n\n- Data handling and privacy:\n`docs/PRIVACY.md`\n\nDo not attach real packet captures, raw flow exports, internal IP inventories, or unredacted incidents to public issues.\n\n- Research/evaluation grade; not production-ready and not a SOC replacement.\n- Batch CSV processing only; no streaming, API, dashboard, or persistent state service.\n- CICFlowMeter-compatible input only.\n- Frozen Python 3.13 serialization/runtime compatibility.\n- Input timestamps have source-defined, timezone-naive semantics and one-second granularity.\n- Validation covers one future day from an overlapping network/dataset family, not arbitrary networks.\n- Flow-level weaknesses remain concentrated in Fuzzers and Analysis traffic.\n- The score is not calibrated as a real-world attack probability.\n- Alert workload remains too high for normal Tier-1 production operations.\n\nThe production-facing v0.1 pipeline was selected using chronological capture days:\n\n``` php\n2015-01-22 -> training\n2015-02-17 -> validation\n2015-02-18 -> locked temporal holdout\n```\n\nTemporal context resets at split and batch boundaries. February 18 was not used for training, feature selection, threshold selection, aggregation selection, or promotion selection. Research scripts remain available for audit, but datasets and generated evaluation artifacts are not redistributed.\n\nThe research uses UNSW-NB15 and CIC-UNSW-NB15 under their publishers' terms. Third-party dataset terms are not covered by this repository's Apache-2.0 license.\n\n```\nsecurity-anomaly-ml/\n├── src/security_anomaly/      # label-free product runtime\n├── src/*.py                   # research and reproduction tooling\n├── contracts/                 # versioned feature/model/incident contracts\n├── tests/\n│   └── fixtures/product-v01/  # deterministic public regression fixture\n├── docs/                      # CLI, Docker, CI, privacy, and design docs\n├── tools/                     # artifact and parity verification helpers\n├── data/                      # ignored external datasets and derived data\n├── models/                    # ignored external model/evaluation artifacts\n├── Dockerfile\n├── pyproject.toml\n├── requirements-runtime.txt\n└── README.md\n```\n\nPython 3.13 is required for the frozen product runtime.\n\n```\npython3.13 -m venv .venv\nsource .venv/bin/activate\npython -m pip install -r requirements.txt\npython -m pytest -q\n```\n\nSome research-reproduction tests require non-redistributed datasets or generated artifacts and skip explicitly in a clean public checkout. Product CI is fully reproducible from public inputs.\n\nOriginal source code is licensed under the [Apache License 2.0](/ibondarenko1/security-anomaly-ml/blob/main/LICENSE). Dataset licenses and terms remain with their original publishers.", "url": "https://wpnews.pro/news/network-flow-ml-that-outputs-security-incidents", "canonical_source": "https://github.com/ibondarenko1/security-anomaly-ml", "published_at": "2026-08-20 22:12:31+00:00", "updated_at": "2026-08-20 22:44:38.658215+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence"], "entities": ["Security Anomaly ML", "I. Bondarenko", "CICFlowMeter", "Random Forest", "context-rf-v2"], "alternates": {"html": "https://wpnews.pro/news/network-flow-ml-that-outputs-security-incidents", "markdown": "https://wpnews.pro/news/network-flow-ml-that-outputs-security-incidents.md", "text": "https://wpnews.pro/news/network-flow-ml-that-outputs-security-incidents.txt", "jsonld": "https://wpnews.pro/news/network-flow-ml-that-outputs-security-incidents.jsonld"}}