Series overview: Ontographia
Teams that query a graph database such as Neo4j in natural language usually choose between letting an AI write the query and locking the system to handwritten templates. This series examines the common compromise (agent skills plus template selection) and proposes another split: the AI extracts a structured Intent, and an ontology-checked, deterministic pipeline compiles the Cypher (Ontographia).
Part 1: The Graph AI Dilemma: Let LLMs Write Cypher or Suffer Template Maintenance Hell?. Production bottlenecks, research review, Intent-first hypothesis Part 2: The LLM Never Writes Cypher: Intent-First Design and Implementation. How the split works in practice Part 3: Evaluation and Roadmap (This article). What the measurements support, what they do not, and what comes next Japanese edition: Zenn article
Documentation: edgesentry.github.io/ontographia
Repository: github.com/edgesentry/ontographia
In Part 1, I laid out why the standard compromise in graph AI—agent skills paired with template selection—hits a maintenance ceiling as schemas expand, and proposed the Intent-first hypothesis. In Part 2, I walked through the architecture that enforces this division of labor: the model extracts an Intent JSON payload, and a deterministic core validates it against an ontology and builds parameter-bound Cypher. Part 2 also adapted three major directions from recent Text-to-Cypher literature to the Intent layer: dynamic ontology subsetting, self-repair driven by execution feedback, and difficulty-adaptive resource allocation.
This final installment examines whether the Intent-first architecture holds up under stress, where it succeeds, and where its limits lie. To answer those questions, I tested the system across a synthetic manufacturing stress benchmark of up to 2,000 properties and 15 third-party public database schemas from Neo4j's official text2cypher benchmark.
Before digging into the full data, here are the five primary takeaways from the evaluation:
First, static ontology validation breaks down against silent false positives as schemas grow. When an ontology contains thousands of properties, an LLM easily picks a distractor that exists in the schema but is wrong for the question (such as picking factory_id instead of plant_id). Because the chosen property is a real, registered vocabulary item, static schema validators and AST compilers accept it without error. The query compiles cleanly, runs without failing, and returns wrong numbers.
Second, dynamic ontology subsetting structurally resolves this silent failure mode. By extracting a minimal connected subgraph of classes and relationships matched to the user's question, irrelevant distractors are physically pruned from the prompt. In deterministic simulation, this raised property accuracy by 50 percentage points and cut prompt tokens by about 91%. In live repeated evaluations using Gemini 3.8 Flash on a 2,000-property ontology, our recommended subset-with-fallback policy lifted property accuracy by an average of 19.5 percentage points over the full schema while cutting prompt tokens by roughly 90.5% to 91% (13,765 down to 1,301 p95).
Third, execution feedback repairs empty results without compromising query governance. When a query is syntactically valid but returns zero rows because of an invalid filter value (an Empty@Valid condition), feeding the empty execution result back into an Intent revision prompt achieved 100% task recovery in deterministic simulation. Crucially, the model never touches raw Cypher strings; all regenerated queries matched the deterministic compiler output exactly.
Fourth, difficulty-adaptive resource allocation drastically reduces operational total cost of ownership. For routine questions, a subset-first policy with a single initial attempt matched the quality of an always-full-schema baseline while trimming token consumption by about 91%. For challenging questions where the subset pruned necessary vocabulary, the pipeline automatically escalated to the full schema with extended retries, recovering cleanly without human intervention.
Fifth, third-party schemas integrate smoothly, but public benchmark answer quality highlights fundamental evaluation constraints. All 15 external schemas from Neo4j's text2cypher benchmark converted into Ontographia's Canonical Ontology Model, achieving 100% relationship coverage, about 96% node label coverage, and about 81% property coverage. However, an end-to-end execution spot check on live public demo databases scored an exact row-set match of roughly 3%. Because public benchmarks treat raw Cypher as the gold standard rather than Intent structures, semantically valid Cypher queries written differently from the benchmark reference are penalized as total failures.
Before reviewing the numerical data, it helps to establish the experimental logic: why each test was necessary, what concrete value it delivers to operational teams, and why specific evaluation techniques were chosen.
The evaluation is split across two experimental tracks:
Track A is a synthetic stress test based on an enterprise manufacturing domain. It introduces controlled distractor properties across three scale tiers (16, 200, and 2,000 properties) to stress-test hypotheses H1, H2, and H3 under isolated conditions.
Track B is an external validity benchmark using 15 third-party database schemas published by Neo4j, spanning cinema, social networks, healthcare, and retail. It evaluates whether Ontographia's Canonical Ontology Model generalizes beyond schemas authored by its creators.
The primary motivation for this evaluation was the realization that the common belief—"constraining an LLM with an ontology guarantees query safety"—is flawed.
Static ontology validation cleanly rejects hallucinated vocabulary that does not exist in the database. But when an enterprise ontology swells to thousands of properties after years of system integrations, mergers, and ERP migrations, it fills with plausible near-duplicates. If an LLM selects an incorrect property that happens to be registered in the schema, mechanical validation passes it through. The failure is completely silent.
A second motivation was validating self-repair. When a query compiles cleanly but returns zero rows because a filter value does not exist, can an agent recover simply by correcting its structured Intent, without letting the model edit raw Cypher text directly?
A third motivation was verifying architectural generality. Any developer can build an abstraction that works on their own demo schema. Demonstrating that the pipeline functions across 15 external database models without modification was necessary to prove that the Canonical Ontology Model is not an overfitted toy.
For teams running graph databases in production, these verifications address real economic and operational bottlenecks:
To ensure that results reflect real architectural properties rather than random benchmark noise, the evaluation balances synthetic isolation with real-world testing:
Why use a synthetic benchmark for Track A?
In production datasets, failure causes are heavily confounded by missing data, inconsistent terminology, ambiguous questions, and network timeouts. By synthesizing near-duplicate distractors across three distinct scale milestones (16, 200, and 2,000 properties), we can cleanly isolate exactly how schema size distorts model attention and triggers silent false positives.
Why use public benchmarks for Track B?
Testing against 15 third-party schemas from Neo4j's official text2cypher-2025v1 dataset eliminates author bias and proves that the ontology ingestion engine handles diverse graph designs.
Why use deterministic simulation arms?
In clinical trials and statistics, an "arm" refers to a distinct experimental condition group. In our evaluation, a "deterministic simulation arm" replaces the commercial LLM with fixed, predictable heuristic rules.
Commercial LLMs introduce nondeterministic variance, daily API updates, rate limits, and sampling noise. If an evaluation mixes pipeline mechanics with live LLM calls from day one, it becomes impossible to determine whether a failure was caused by a bug in the state machine or a temporary lapse in model attention.
Just as software engineers separate unit and integration tests from third-party API dependencies, using deterministic simulation arms allows us to verify the correctness of the pipeline control loops with 100% mathematical reproducibility.
However, a deterministic simulation tests the mechanics of the control pipeline, not the intelligence of a model. To verify how real models perform, we follow simulation testing with repeated live commercial LLM evaluations using Gemini 3.8 Flash.
To evaluate the architectural extensions proposed in Part 2, I established three formal hypotheses. To prevent post-hoc rationalization, strict pass, partial, and fail criteria were defined directly in the evaluation harness code before running benchmarks.
--refine-on-exec flag.--schema-policy auto) with initial and escalated retry ceilings.
In enterprise knowledge graphs, property namespaces frequently become polluted. A manufacturing database might simultaneously contain plant_id, plant_code, factory_id, facility_name, and plant_name.
To measure how schema growth impacts model behavior, Track A uses a manufacturing ontology evaluated against 40 benchmark questions, each associated with an authoritative Gold Intent. We synthesized three scale tiers by systematically injecting distractors—near-duplicate property names and redundant entity types:
The test suite evaluates what happens when an Intent contains an incorrect property that nonetheless exists within the ontology.
Running the baseline pipeline across all three schema tiers produced the following results:
Small Schema (16 properties):
Mid Schema (200 properties):
Large Schema (2,000 properties):
On the clean 16-property schema, the system performed as intuitively expected: because distractor properties did not exist in the ontology, any Intent attempting to use an incorrect property was rejected by static validation as an unknown name.
However, once the schema grew to 200 and 2,000 properties, the safety net vanished. All 40 deliberately incorrect Intents sailed through ontology validation and compiled into executable Cypher (40 of 40).
This quantitatively confirms the primary failure mode of graph AI at scale: static validators and AST compilers verify lexical existence, not contextual appropriateness. If an LLM becomes confused by a wall of prompt text and selects factory_id instead of plant_id, the validator sees a registered, valid attribute and signs off on the query. The database executes it without raising an error, producing empty rows or corrupt business metrics.
Simultaneously, prompt token overhead on the full schema grew tenfold, jumping from 1,359 tokens on the small schema to 13,710 tokens on the large schema.
To reproduce this baseline experiment locally:
uv run python examples/llm/eval/run_track_a.py --record
To combat both silent false positives and prompt bloat, Ontographia implements dynamic ontology subsetting. When a user asks a question, the engine scans the text for vocabulary matching ontology classes and properties. It identifies the target nodes, computes the minimal connecting relationships between them, and prunes all unreferenced entities and distractor properties from the prompt.
We evaluated three schema delivery configurations on the 2,000-property ontology:
subset_fallback): The model receives the subset first; if Intent extraction or validation fails, the engine automatically falls back to the full schema. This is our recommended production configuration.
Testing these configurations in a deterministic simulation arm yielded the following measurements:
Full Schema:
Standalone Subset:
Subset with Fallback:
The simulation arm met all pre-registered criteria for Hypothesis H1:
However, a deterministic simulation relies on a key simplifying assumption: that the model only errs when distractors are visible in its prompt. To observe how real models behave when dealing with ambiguous phrasing and vocabulary pruning, we turn to commercial LLM evaluation.
To reproduce the simulation locally:
uv run python examples/llm/eval/run_track_a_h1.py --record
To measure real-world performance, we evaluated the published package (ontographia[llm]==0.1.6) against commercial LLM APIs via LiteLLM. We selected Gemini 3.8 Flash (model identifier ontographia-gemini-3.8, temperature 0).
Because commercial LLM calls are subject to sampling variance, single-run benchmarks can be deeply misleading. We designed a multi-run sampling plan across both the 200-property (mid) and 2,000-property (large) schemas:
The summary below details the mean and sample standard deviation across all 5 live runs on the 2,000-property ontology:
Subset with Fallback (Recommended):
Crucially, across all 5 runs and all conditions, the rate of hallucinated vocabulary ("invent-without-seeing", where the model invents terms absent from the prompt) was exactly 0. The rate of choosing near-duplicate distractors was also 0.
For comparison, the 3 runs on the 200-property ontology produced consistent trends:
Reviewing these measurements yields four critical insights:
First, Hypothesis H1 is supported under live commercial conditions. Across every single run in both the mid (n=3) and large (n=5) configurations, the pre-registered criteria were fully satisfied. The formal verdict across all runs is Support.
Second, the recommended subset_fallback policy demonstrates clear superiority over the full schema. On the 2,000-property ontology, subset_fallback boosted Property Hit rate by an average of +19.5 percentage points (improving from 0.683 to 0.878). Across the 5 individual runs, the measured accuracy gains over the full schema were +21.25, +31.25, +22.50, +11.25, and +11.25 percentage points. Clean compile completions rose from 28.4 to 36.4 of 40. Prompt tokens at p95 plunged from 13,765 down to 1,301, representing a 90.5% reduction (roughly 91%).
Third, standalone subsetting exhibits high variance and must not be used in production. While the standalone subset achieved a higher mean accuracy than the full schema (0.720 vs 0.683), its sample standard deviation was large (sd = 0.086). In individual runs, aggressive keyword pruning stripped out necessary vocabulary, causing the standalone subset to score lower than the full schema.
Fourth, the fallback mechanism is essential for operational stability. Under subset_fallback, the engine automatically escalates to the full schema whenever the subset fails extraction or validation. This fallback triggered on 17.0% ± 2.7% of queries, requiring an average of 1.08 LLM calls per question. By using the full schema as a safety net, subset_fallback recaptured missing vocabulary and maintained a high, stable success rate (0.878 ± 0.065). In production environments, subsetting without a fallback path is an anti-pattern.
To reproduce the live evaluation harness:
uv run python examples/llm/eval/run_track_a_h1_live.py --profiles mid,large --record
Any architecture can achieve high benchmark scores on an ontology designed by its own authors. To test whether Ontographia's Canonical Ontology Model (COM) generalizes to third-party databases, Track B evaluates the engine against external benchmarks.
We sourced 15 database schemas from Neo4j's official neo4j/text2cypher-2025v1 dataset on Hugging Face. These schemas represent diverse domains, including cinema, recommendations, social networks, retail, logistics, and healthcare. We built an automated ingestion tool (schema_convert.py) to parse these external schemas and map them directly into Ontographia's COM structure.
Across all 15 third-party databases, the ingestion tool produced the following results:
Engine.load success rate: 15 of 15 (100%).
The engine ingested every single external database schema without code changes, achieving perfect coverage on relationship topologies and 96% coverage on node entity types.
The remaining 19% gap in property coverage stems from structural quirks in the source benchmark dataset rather than architectural limitations. Several schemas in text2cypher-2025v1 contain inconsistent textual property descriptions, unstructured comments, or property names embedded inside freeform prose. While automated regex parsers missed these edge cases, the underlying COM schema representation accommodates all standard graph primitives.
To evaluate whether the pipeline could answer questions on these third-party databases, we conducted an end-to-end execution spot check. We stratified 60 sample questions (seed=53) from the movies and recommendations databases, pointed the engine at Neo4j's public demo instance (demo.neo4jlabs.com), and queried the database using Gemini 3.7 Flash ( gemini-3.7-flash via LiteLLM).
The pipeline executed the full cycle: Natural Language Question -> Intent JSON -> Deterministic Engine.build -> Neo4j Database Query -> Jaccard Row-Set Comparison against benchmark Gold Cypher queries.
The spot check yielded the following metrics:
An exact match score of roughly 3% is undeniably low. However, analyzing the failure modes reveals the structural challenges of evaluating Intent-first architectures against raw Cypher benchmarks:
MATCH ... WHERE block with distinct aliasing. Both queries may extract valid operational data, but minor differences in column ordering, aggregation grouping, or projection limits cause the strict row-set Jaccard comparison to score 0.
This experiment proves that Ontographia's ingestion and compilation pipeline executes end-to-end on arbitrary third-party databases without crashing. It does not prove that Ontographia currently matches human-level accuracy on open-domain Cypher benchmarks. We make no claims of State-of-the-Art (SOTA) execution parity on Track B.
A key takeaway from Track B was the necessity of decoupling domain heuristics from the core engine. In early prototypes, terminology repair rules were hardcoded for manufacturing concepts. In Track B, we moved these rules into declarative YAML configuration mappings. We verified that switching from manufacturing mappings to cinema domain mappings allowed the same compiler engine to handle movies without modification.
To run the Track B execution benchmark:
uv run --with datasets --with neo4j python examples/llm/eval/run_track_b_exec.py --record
In real enterprise operations, a query can be syntactically flawless, fully compliant with the ontology, and compile without warnings, yet still return zero records when executed against the database.
Consider a user asking for production numbers from "Plant 4." The ontology confirms that Plant nodes have a plant_code attribute, and the query compiles cleanly. But if the physical database only contains plants 1, 2, and 3, Neo4j returns an empty result set. We term this failure mode Empty@Valid.
In traditional Text-to-Cypher systems, the agent handles empty results by passing the failed Cypher query back to the LLM and asking it to rewrite the query. This reintroduces non-deterministic behavior: the model frequently hallucinates alternative relationship paths, drops required filters, or introduces syntax errors.
Ontographia takes a fundamentally different approach. The engine captures the empty execution result and feeds it back into the Intent extraction prompt, requesting a revised Intent JSON. The LLM never touches the Cypher string. The revised Intent passes through ontology validation and is compiled into Cypher by the deterministic AST engine.
To test whether this repair loop operates reliably, we constructed a test suite of 40 questions where filter values in the Gold Intents were deliberately corrupted. The queries compiled successfully, but mock database executions returned zero rows.
We evaluated two conditions in a deterministic simulation arm:
no_refine): The pipeline terminates immediately after the initial query returns an empty result.with_refine): The empty result triggers an Intent refinement prompt, and the revised Intent is re-compiled.
The simulation produced the following measurements:
Without Feedback (no_refine):
Engine.build: 1.00 (100%)
With Feedback (with_refine):
The simulation supported Hypothesis H2, achieving a 100 percentage point recovery from empty execution results.
Crucially, in 100% of cases, the final Cypher queries matched the output of the deterministic AST compiler. This demonstrates that self-repair can be achieved without allowing the LLM to write or alter query text directly.
This simulation confirms the integrity of the state machine and repair control loop: when provided with corrective feedback, the pipeline successfully re-parses, re-validates, and re-compiles the query.
In live production environments, the ultimate recovery rate depends on whether the LLM can infer the correct filter value from the execution error. The core value of Ontographia's approach is that even when an LLM fails to find the right filter, it cannot violate query safety or database governance while trying.
Teams can enable this self-repair loop in production via the CLI:
uv run python examples/run_llm_e2e.py \
--question "Which plant hosts production Line-1?" \
--execute \
--refine-on-exec \
--password ontographia
uv run python examples/llm/eval/run_track_a_h2.py --record
In enterprise conversational systems, user queries vary wildly in complexity. Asking "What is the name of Line 1?" requires minimal schema context and can be resolved in a single call. Conversely, asking for a cross-facility dependency bottleneck requires inspecting multiple relationship hops.
Defaulting to the most expensive configuration—dumping the entire 2,000-property ontology into every prompt and allowing up to 5 retries—wastes token budgets and introduces unnecessary latency. Conversely, capping all queries to minimal prompts causes difficult questions to fail.
Hypothesis H3 evaluates an adaptive resource allocation policy: start every query with an ontology subset and a single initial attempt (initial_retries=1); if validation or execution fails, automatically escalate to the full schema with extended retries (escalated_retries=5).
We evaluated this adaptive policy against an always-full control group on the 2,000-property ontology, testing across two query slices:
We compared two system configurations:
always_full_fixed): Always sends the full 2,000-property ontology with a fixed ceiling of 5 retries.adaptive): Starts with the subset and 1 retry; escalates to the full schema and 5 retries only upon failure.
The simulation arm yielded the following metrics:
Easy Slice under always_full_fixed:
Easy Slice under adaptive:
Hard Slice under always_full_fixed:
Hard Slice under adaptive:
The simulation supported Hypothesis H3 across all criteria:
This adaptive mechanism is already built into Ontographia as the default production mode (--schema-policy auto). Because this policy mirrors the subset_fallback behavior verified in our live commercial LLM evaluations, teams do not need a separate query classifier to benefit from adaptive spend.
To test the adaptive policy:
uv run python examples/run_llm_e2e.py \
--question "Which plant hosts production Line-1?" \
--schema-policy auto \
--initial-retries 1 --escalated-retries 5
uv run python examples/llm/eval/run_track_a_h3.py --record
Synthesizing our empirical findings across simulation arms, commercial LLM remeasurements, and public benchmarks yields a grounded assessment of the Intent-first paradigm:
These results indicate that Intent-first is not merely a theoretical framework. It operates effectively on commercial LLM APIs and external database schemas, delivering substantial cost reductions and structural error protection. At the same time, it highlights the need for better evaluation standards that look beyond syntactic Cypher matching.
The Ontographia project set out to test whether natural language interfaces to graph databases could achieve deterministic auditability without drowning in template maintenance. Here is an accounting of what has been proven, what was left out of scope, and what lies ahead.
schema_policy=auto already serves as the default operational path.
The full project roadmap is maintained on the Ontographia repository roadmap.
Over this three-part series, we have explored a fundamental architectural question: how do we build natural language interfaces to enterprise knowledge graphs that are flexible enough to answer ad-hoc questions, yet strict enough to deploy in production?
The central takeaway is clear: enterprise graph AI does not require choosing between safety and scalability. By drawing a clean boundary between semantic extraction and query compilation, teams can deploy conversational graph agents that are both cost-effective and strictly auditable.
Ontographia is open source. All code, ontology specifications, evaluation harnesses, and raw benchmark data are publicly accessible.
You can install the experimental Intent layer directly from PyPI:
pip install "ontographia[llm]"
Explore the test suites under examples/llm/eval/, review the Evaluation Guide, and test the pipeline against your own domain ontologies. Feedback, benchmark reports, and contributions are welcome on the GitHub repository.
Open source project: github.com/edgesentry/ontographia
Documentation: edgesentry.github.io/ontographia
Release notes: v0.1.6
Author: Yohei Onishi
Tags: #Neo4j #Cypher #KnowledgeGraph #GraphRAG #AIAgents #EnterpriseAI #LLMEvaluation