I let a local 27B LLM audit and fix my Splunk + Sysmon stack

A security analyst studying for CySA+ demonstrated that a local 27B-parameter LLM running entirely on their own GPU could audit and remediate a Splunk and Sysmon home SOC stack without any data leaving the machine. Across five audits the agent corrected the analyst's assumptions three times, diagnosed issues such as double log ingestion via a Universal Forwarder and three dead data inputs, and flagged that Sysmon was not detecting LSASS credential access. The analyst noted the model hit 97.4% of its 128K context window chasing an irrelevant detail during remediation, requiring a system-prompt rule to stop once a root cause was evidenced.

The question was not "can an LLM do SOC work". The question I actually wanted answered was narrower and harder: can a 27B model running on my own GPU, with zero bytes leaving the machine, audit my Splunk install, find what is broken and fix it — without me telling it how? After an afternoon of back and forth, the answer is yes, with caveats worth writing down. Including the mistakes, because those are the interesting part. Context first, since it shapes how you should read this: I am a security analyst without prior SOC experience — I am studying for CySA+, and building a home SOC is the only way I have to practise on something that behaves like a real environment. Everything here ran on my own hardware, was reviewed by the model and verified against real evidence. No third-party data, no borrowed infrastructure. Three moving parts: llama-server and driven by an agent. The goal was not a mentor that explains things to me. It was an agent that audits the real install, finds what is wrong, and remediates it: edits configs, proposes commands, removes what is redundant. I did not hand my Splunk over on day one. I ran five real audits against it: config inspection, data pipeline review, logging hardening, 24-hour log analysis, and IoC detection. The verdict: it reasons like a senior analyst. It corrected wrong assumptions of mine three times, debugged a dozen issues on its own, and — this matters — said "I cannot confirm this" instead of inventing an answer. Thoroughness is gold in an audit and poison in a remediation. When I handed it the first finding to fix, it went full forensic: tried performance-counter tools to verify counters, read the vendor spec, ran the failing binary by hand to reproduce it — and ran out of context. It hit 97.4% of its 128K window chasing the source name of a Windows Update log that had nothing to do with the fix. The model knew the answer. It did not know when to stop. The fix was a system-prompt rule: stop as soon as you have the root cause with evidence . Double ingestion. My indexer was already collecting logs locally, and a Universal Forwarder on the same box was sending them again. Double licence usage, double noise. The valuable part was not the diagnosis: it read the two product GUIDs out of the MSI registry entries and explicitly warned me which one to remove and which one never to touch , because that second one was Splunk itself with all my data in it. A careless msiexec /x there and the lab is gone. It marked that one as excluded. Also worth knowing: the straightforward msiexec /x ... /qn returned a phantom -1 , because msiexec is asynchronous and PowerShell does not capture its exit code reliably. Start-Process msiexec -Wait -PassThru returned a clean ExitCode: 0 and removed service, directory and registry entry. Three data inputs dead since install day. Zero events since July, three different root causes: % Processor Time but % de tiempo de procesador . Even the object name is localised. The fix was to force the tool to use the English API. And a detail I appreciate more than the fix itself: the config change was correct, but the binary still died when sampling. It documented that as "needs binary debugging" instead of claiming success. Sysmon was blind to credential theft. My config did not detect access to LSASS — the process that holds credentials in memory, and the thing Mimikatz goes after. An attacker could have dumped credentials and I would have learned about it later. Enabling ProcessAccess with a filter on lsass.exe is a three-line change, but along the way we discovered that both the model and I were carrying a wrong Sysmon event-ID mapping . The numbers I had been using were simply incorrect. The real config file disproved them. Verify technical numbers against reality; do not trust recall, including your own. One command, three traps. To log process creation with the command line: auditpol /set /subcategory:"{0CCE922B-69AE-11D9-BED3-505054503030}" /success:enable /failure:enable It failed for three reasons at once: the privilege was missing, the category name is localised so the English string does not resolve , and the braces of the GUID break PowerShell syntax unless quoted. Elevation, localisation, syntax. Two earlier findings turned out to be false positives, and the model is what caught them , correcting previous audits one of them a note of mine : An analyst that corrects its own assumptions with evidence beats one that gets lucky. After watching it blow through the context window, I fixed the cause rather than the symptom: temperature down from 0.8 to 0.3, plus a working-rules system prompt answer only what is asked; in remediation stop at root cause with evidence; in audit be exhaustive; use scripts for mechanical operations; verify IDs, ports and GUIDs before asserting; never invent; be concise . | | Before | After | |---|---|---| | Output | Huge, rambling | Short and direct | | Context used | 97.4% overflowed | 49.7% | | Assumptions | Taken for granted | Verified against reality | | Result | 1 finding in 30 min | 4 findings in a row | Out of ten findings: double ingestion resolved, two inputs fixed and one pending binary debugging, credential access now detected, process logging enabled, two false positives corrected, one of my own notes fixed. Not a perfect lab result — which is exactly why I trust the report. What I take away: An LLM does not replace your judgement or the urge to understand what is happening. But as a mentor that never sleeps and never leaves your machine, for someone starting out like me, it is a genuine advantage — and not a single log leaves my computer . The full technical breakdown of this audit — screenshots, exact commands and config files — is on my blog: A 27B SOC Analyst: Auditing and Remediating Splunk + Sysmon https://sammideblas.com/posts/0x73-a-27b-soc-analyst-auditing-and-remediating-splunk-sysmon . Related, if you are building the same kind of thing: Building Your Own Log Analyzer https://sammideblas.com/posts/0x74-building-your-own-log-analyzer and Gravity SOC, my home security operations center https://sammideblas.com/posts/0x46-gravity-soc-my-security-operations-center . I write up what I learn, mistakes included, at sammideblas.com https://sammideblas.com .