Lemmalog separates fuzzy LLM extraction from deterministic facts, provenance and retractions, with early benchmarks showing promise and gaps.
By RuntimeWire Staff · Published
Primary source: pwning.systems
Why it matters #
Lemmalog moves agent memory from transcript retrieval toward auditable state maintenance, giving developers a way to retract bad assumptions without asking an LLM to reconstruct every dependency.
Security researcher Jordy Zomer published the design and source code for Lemmalog on August 28th, turning a recurring failure in long-running AI-assisted vulnerability research into a Datalog engine that maintains what an agent currently knows.
Zomer started with a practical annoyance. After several hours inside a large codebase, an LLM agent would revive discarded exploit paths, forget that an assumption had failed or continue reasoning from an observation invalidated in a debugger. Retrieval could bring the old notes back, but it left the model to reconcile which version remained true.
"I want it to maintain what we currently know," Zomer wrote in his August 28th technical account. That distinction led Zomer back to the tools he already knew. His security work spans static analysis, fuzzing, reverse engineering, binary exploitation and open-source vulnerability research. Ubuntu has also credited Zomer and Alexandra Sandulescu with discovering two Linux kernel flaws involving speculative-execution barriers, CVE-2023-0458 and CVE-2023-0459.
Lemmalog applies the same discipline used in program analysis: record facts, derive conclusions through explicit rules, track their dependencies and update only the affected results when an input changes. The model still interprets source code, debugger output and natural-language notes. Once those observations become structured facts, the deterministic engine takes over.
Memory becomes maintained state
Zomer's example begins with three observations: an attacker controls one object, that object points to a second object, and the second object is a kernel object. A Datalog rule can derive that the attacker controls a kernel object.
If later debugging disproves the pointer relationship, Lemmalog retracts conclusions that depended on it. A conclusion supported by a second, independent path remains valid. The engine can also produce a provenance tree showing which observations and rules support a result. That gives an agent a harder boundary around unsupported claims. The extraction model can still misunderstand evidence or encode a bad fact, a limitation Zomer states directly. Lemmalog makes the resulting state inspectable and gives corrections mechanical consequences once the error is identified.
Temporal facts preserve a history of those changes. An exploit primitive can be recorded as viable during one interval and invalid after a later observation, allowing the agent to answer both what is believed now and why an earlier path was explored.
The repository, published under the MIT license, includes incremental evaluation, retractions, temporal facts, aggregation, entity resolution, persistence, hybrid retrieval and an MCP server for connecting Lemmalog to agent tools including Claude Code and Kimi CLI.
The architecture still uses semantic retrieval. Zomer combines structured facts and dependency-aware reasoning with the original source text, embeddings and lexical search. His point is narrower than replacing vector databases: similarity search answers which old information appears relevant, while the deductive state answers which conclusions remain supported.
The benchmarks support an experiment, not a victory lap
Zomer tested Lemmalog on LongMemEval and LoCoMo, two conversational-memory benchmarks rather than vulnerability investigations, as described in his technical account. He reports a LongMemEval F1 score of 0.463 +/- 0.010 across three runs, behind the published PropMem result of 0.550 and slightly below SimpleMem at 0.480.
The strongest LongMemEval category reported in the research material was knowledge updates, where Lemmalog scored 0.579 in a representative run. That category most closely matches Zomer's original problem: deciding what remains true after an earlier belief changes.
Multi-session performance exposed the weak point. Lemmalog scored 0.211, compared with 0.582 for PropMem. Zomer traced many misses to extraction failures. A perfect rule engine cannot reason over an event that the LLM never converted into a fact.
On LoCoMo, Zomer reports 0.533 +/- 0.001 F1 across three runs. The project also struggled with some inference tasks, illustrating how compressing a conditional preference into a simple tuple can discard the qualification that matters.
All of those results are project-reported and have not been independently validated. Zomer also cautions that the benchmarks do not reproduce a real vulnerability investigation.
The efficiency result is still notable. Zomer says the LongMemEval answering model received roughly 2,700 tokens per question through Lemmalog, compared with about 104,000 tokens under full-context prompting. Extraction carries its own cost, paid when new material enters the system. Repeated queries can then operate on maintained state instead of rereading the full transcript each time.
A different bet in a funded memory market
Agent memory has already become an infrastructure category with venture financing behind it. Mem0 said it raised $24 million across seed and Series A rounds in October 2025. Cognee announced a $7.5 million seed round on February 19th, 2026, for an engine combining graph, vector and relational storage.
Other builders are also moving beyond transcript retrieval. Letta's Context Repositories use Git-backed files to version changes to agent memory, while Weaviate's Engram is pitched as a managed context service that reconciles evolving facts.
Zomer's contribution comes from the security-research end of the problem. Lemmalog treats an agent's beliefs like an analysis artifact whose assumptions, derivations and invalidations should be visible. That approach is particularly suited to work where a stale claim can waste hours or corrupt every conclusion downstream.
The decisive test remains the one Zomer set for himself: run an agent through a long, complicated vulnerability investigation and measure whether maintained state stops dead hypotheses from returning. The current benchmarks put numbers around where the experiment performs well and where extraction and inference remain weak. They also show how much depends on the probabilistic front end correctly reading the evidence in the first place.