Every morning, a DLP analyst logs in to thousands of security alerts. Most are the same pattern as yesterday — the same HR manager printing the same payroll report, the same sales rep emailing the same customer list. Yet each one runs through an expensive LLM pipeline, burns tokens, and produces the same verdict as the last 100 times.
This isn't security work. It's assembly line labor priced at LLM rates.
I spent the last few weeks building something to fix this: a DLP Alert Triage Agent with autonomous memory. The system gets smarter every time an analyst does their job — without requiring them to do anything extra. No "train" button. No rule writing. No prompt tuning.
Traditional DLP tools have a fundamental blind spot: they treat every alert as if it's the first time they've seen it.
An analyst can correct the same false positive 50 times, and the system will happily make the same mistake on attempt #51. There's no feedback loop between the analyst's decision and the model's behavior. Some teams try to work around this with custom rules or fine-tuning, but rules are brittle, fine-tuning is expensive, and neither scales across hundreds of alert types and dozens of departments.
The real issue isn't model accuracy — it's that the system doesn't learn from experience. An analyst shouldn't have to re-litigate the same alert twice.
I designed the pipeline around a simple principle: don't pay for what you can compute, and remember what you've already learned.
The pipeline has four stages, each capable of short-circuiting the rest:
Stage 1 — Metadata Analysis. Who initiated the action? What file? What time? What destination? Some alerts are clearly safe or clearly suspicious based on metadata alone. These never proceed further.
Stage 2 — Algorithms. Shannon entropy detects encoded or encrypted payloads. Structural fingerprinting identifies known file types and detects tampering. Encoding detection catches obfuscation attempts. This stage uses zero LLM tokens — pure applied math. In practice, it catches about 20% of clearly malicious content before any model is invoked.
Stage 3 — Memory Recall. This is where the system's learned memory lives. Every past analyst decision has been distilled into a pattern — a Markdown file with YAML frontmatter describing the conditions, the correct outcome, and the confidence level. If a new alert matches an existing pattern, the system returns the cached verdict in approximately 50 milliseconds with zero token cost. This stage is responsible for the bulk of the efficiency gains.
Stage 4 — LLM Pipeline. Only reached for genuinely novel alerts. A Lite LLM performs initial triage, then two Specialist Agents analyze independently. If they disagree, they enter up to two rounds of structured debate — each sees the other's reasoning and may revise its verdict. A Risk Scorer produces the final decision.
This was the hardest part to get right. The memory system needs to be autonomous, self-organizing, and human-readable.
Every learned pattern is a Markdown file with YAML frontmatter:
---
conditions:
role: HR Manager
action: print
destination: printer
confidence: 0.92
hitCount: 14
created: 2026-07-17
lastMatched: 2026-07-19
outcome: safe
---
HR managers routinely print payroll documents for distribution.
No policy violation detected. Pattern confirmed by analyst 14 times.
The system manages its own memory with zero human intervention:
hitCount × 0.35 + confidence × 0.35 + recency × 0.30
. The most relevant patterns enter the prompt.The most difficult engineering problem was merging overlapping patterns.
Consider two patterns: one says "HR manager printing payroll is safe," another says "HR manager printing Q4 payroll is safe." Should the system merge them? Keep both? Let the more specific one supersede?
Our first attempt was too aggressive — it merged everything, creating patterns so broad they matched irrelevant alerts. The second attempt was too conservative — it kept every pattern separate, and the memory count grew uncontrollably.
The third iteration finally worked: a dedicated merge agent, powered by Qwen3.7-plus, that analyzes both patterns and decides whether to merge, keep separate, or let one supersede the other. The merge agent considers semantic overlap, confidence scores, hit counts, and recency before making its decision. This runs as part of the compaction cycle and has proven remarkably reliable.
The system uses Qwen Cloud for everything:
For document processing, the system uses pdf-parse, mammoth (for .docx), and xlsx (for Excel files), so it supports the full range of file types that appear in real DLP workflows.
In testing across 16 scenarios spanning three companies (Bank Corp, Healthcare+, SaaS Inc) and six employee roles:
If I could share one insight from this project, it's this: most AI systems don't need to be smarter — they need to be better at remembering what they've already figured out.
The obsession with model capabilities often distracts from a simpler truth: the cheapest LLM call is the one you don't make. By investing in algorithmic pre-filtering and persistent memory, you can build systems that are simultaneously more efficient and more accurate than a pure LLM approach.
The code is open source under MIT: github.com/Ra-226/dlp-triage-agent And if you're building something similar, I'd love to hear about your approach to the memory merging problem. It's harder than it looks.