{"slug": "benchmarking-5-graph-database-platforms-on-the-same-352k-edge-graph-what-free", "title": "Benchmarking 5 Graph Database Platforms on the Same 352k-Edge Graph: What Free Tiers Hide", "summary": "An engineer from Wexa AI benchmarked five graph database platforms, including CognoDB Cloud, on a 352,768-edge citation network under free-tier constraints. The harness measured ingest, traversal, lookup, aggregation, and mixed workloads, documenting throttling and failures. Memgraph led in ingest speed, while the report highlights how free tiers can distort performance comparisons.", "body_md": "Graph database benchmarks are easy to get wrong.\n\nMost public benchmarks compare expensive production instances, use different datasets, ignore network latency, hide free-tier throttling, or quietly optimize one database more than the others.\n\nFor a recent engineering assignment from **Wexa AI**, I decided to do the opposite.\n\nI built a small benchmark harness that compared **CognoDB Cloud** against four other graph database platforms using:\n\nThe platforms tested were:\n\nThe dataset was the SNAP `cit-HepTh`\n\ncitation network:\n\nSmall enough to fit inside free tiers, but large enough to make the benchmark meaningful.\n\nThe full benchmark code, results, and documentation are available here:\n\n**GitHub repository:** [Add your GitHub repository link]\n\nThe assignment was clear: the goal was not to crown a database.\n\nThe goal was to evaluate engineering rigor:\n\nThat matters because real-world database selection is messy.\n\nThere is no universal “best graph database.”\n\nThere is only the best database for:\n\nSo this benchmark was designed to measure behavior under constrained free-tier conditions, not to simulate an unlimited production environment.\n\nBefore writing code, I set a few rules.\n\nEvery platform received the exact same graph.\n\nAll platforms ran the same workloads, even where minor Cypher dialect adjustments were required.\n\nNo paid production instance was allowed to sneak into the comparison.\n\nCognoDB’s free tier is intentionally small:\n\nSo the other platforms were run on their free, trial, or capped self-hosted tiers.\n\nEvery read workload was warmed up first. Cold-start behavior was excluded from the main latency numbers and documented as a caveat.\n\nAverages can hide bad requests.\n\nSo the benchmark reported:\n\nIf something failed, throttled, crashed, or behaved strangely, it got documented.\n\nThat rule ended up being one of the most valuable parts of the project.\n\nI used the SNAP `cit-HepTh`\n\ncitation network.\n\nIt represents arXiv High Energy Physics Theory papers and their citations.\n\nIf paper A cites paper B, the graph contains a directed edge from A to B.\n\nThe final cleaned graph contained:\n\n| Metric | Value |\n|---|---|\n| Nodes | 27,769 |\n| Relationships | 352,768 |\n| Node label | `Paper` |\n| Relationship type | `CITES` |\n\nEach paper node had:\n\n`id`\n\n`year`\n\n`field`\n\nThe `year`\n\nproperty came from the original SNAP metadata.\n\nThe `field`\n\nproperty was generated deterministically for aggregation testing, so every platform received identical values.\n\nThis dataset was a good fit because it had enough relationships to make traversals and aggregations interesting, but it was still small enough to fit into free tiers.\n\nI built the benchmark harness in C# using .NET 8 and the official Neo4j .NET driver.\n\nFor each platform, the harness did the following:\n\nThen a report generator converted the JSON files into Markdown tables.\n\nThe workflow was simple:\n\n```\ndotnet run -- cognodb\ndotnet run -- aura\ndotnet run -- sandbox\ndotnet run -- memgraph\ndotnet run -- docker\n```\n\nAnd then:\n\n```\ndotnet run -- report\n```\n\nNo passwords were stored in the repository. All credentials were read from environment variables.\n\nThe benchmark measured six categories.\n\nHow fast can the platform load the graph?\n\nMeasured as:\n\nOne-hop, two-hop, and three-hop queries from randomly selected start nodes.\n\nExample:\n\n``` php\nMATCH (p:Paper {id: $id})-[:CITES]->()-[:CITES]->(q)\nRETURN count(q) AS c;\n```\n\nPoint lookup by indexed `id`\n\n:\n\n```\nMATCH (p:Paper {id: $id})\nRETURN p.year AS year;\n```\n\nFiltered lookup using indexed `year`\n\n:\n\n```\nMATCH (p:Paper)\nWHERE p.year >= $y1 AND p.year <= $y2\nRETURN count(p) AS c;\n```\n\nGroup-by query over a node property:\n\n```\nMATCH (p:Paper)\nRETURN p.field AS field, count(*) AS c\nORDER BY c DESC;\n```\n\nThe mixed workload used:\n\nWhere observable, the benchmark recorded:\n\nWhere a platform did not expose internals, the result was marked as **not observable**.\n\nMemgraph dominated ingest.\n\n| Platform | Relationships/sec | Total Load Time |\n|---|---|---|\n| Memgraph | 14,966 | 25.7s |\n| Docker Neo4j | 8,144 | 55.2s |\n| Neo4j Aura | 7,865 | 47.6s |\n| Neo4j Sandbox | 2,339 | 167.9s |\n| CognoDB | 1,209 | 317.9s |\n\nVisual summary:\n\n```\nmemgraph   14966 rels/sec  ████████████████████████████████\ndocker      8144 rels/sec  █████████████████\naura        7865 rels/sec  ████████████████\nsandbox     2339 rels/sec  █████\ncognodb     1209 rels/sec  ██\n```\n\nMemgraph’s in-memory architecture gave it a major advantage during batched loading.\n\nAura also performed strongly.\n\nDocker Neo4j benefited from local execution and no cloud network overhead.\n\nCognoDB and Sandbox were slower under the tested free-tier conditions.\n\nFor traversal latency, Docker Neo4j had the lowest p50 numbers because it ran locally.\n\n| Platform | 1-hop p50 | 2-hop p50 | 3-hop p50 |\n|---|---|---|---|\n| Docker Neo4j | 3.1 ms | 3.2 ms | 3.3 ms |\n| Memgraph | 95.3 ms | 95.8 ms | 96.2 ms |\n| Neo4j Aura | 146.2 ms | 146.5 ms | 146.8 ms |\n| CognoDB | 644.2 ms | 652.3 ms | 654.2 ms |\n| Neo4j Sandbox | 697.8 ms | 697.7 ms | 698.0 ms |\n\nOne pattern stood out.\n\nFor most platforms, latency barely increased from one hop to three hops.\n\nFor example:\n\nThat tells us something important.\n\nAt this dataset size, the actual traversal work was small compared with fixed overhead.\n\nThat overhead may include:\n\nIn other words, the query engine may not have been the dominant cost. The request path was.\n\nLookup latency followed a similar pattern.\n\n| Platform | Point Lookup p50 |\n|---|---|\n| Docker Neo4j | 3.3 ms |\n| Memgraph | 95.1 ms |\n| Neo4j Aura | 146.1 ms |\n| CognoDB | 643.9 ms |\n| Neo4j Sandbox | 698.8 ms |\n\n| Platform | Aggregation p50 |\n|---|---|\n| Docker Neo4j | 13.9 ms |\n| Memgraph | 106.0 ms |\n| Neo4j Aura | 156.8 ms |\n| CognoDB | 683.7 ms |\n| Neo4j Sandbox | 704.6 ms |\n\nDocker’s low numbers again reflected its localhost advantage.\n\nAmong cloud platforms, Memgraph and Aura were the strongest in this test.\n\nCognoDB remained stable, but its latency was dominated by what appeared to be fixed request overhead under the tested free-tier environment.\n\nThe mixed workload used 80% reads and 20% writes.\n\nAt 40 concurrent clients:\n\n| Platform | QPS | Errors |\n|---|---|---|\n| Docker Neo4j | 378.4 | 0 |\n| Memgraph | 374.4 | 5 |\n| Neo4j Aura | 287.7 | 0 |\n| CognoDB | 61.2 | 0 |\n| Neo4j Sandbox | 51.3 | 0 |\n\nVisual summary:\n\n```\ndocker     378.4 QPS  ████████████████████████████████\nmemgraph   374.4 QPS  ███████████████████████████████\naura       287.7 QPS  ████████████████████████\ncognodb     61.2 QPS  █████\nsandbox     51.3 QPS  ████\n```\n\nDocker and Memgraph achieved the highest throughput.\n\nAura scaled well and completed the workload with no recorded errors.\n\nCognoDB and Sandbox showed lower absolute throughput, but both completed with zero recorded errors.\n\nMemgraph produced five errors during the 40-client workload. I kept those errors in the results because hiding them would defeat the purpose of an honest benchmark. The likely explanation is resource pressure on the entry-tier instance under concurrent writes.\n\nOne of the most interesting findings came from Docker Neo4j.\n\nCognoDB’s free tier gives you:\n\nI wanted the local Docker Neo4j instance to be as close to that as possible.\n\nSo I first capped the Docker container at 256 MB.\n\nNeo4j crashed during startup.\n\nIt did not fail during the benchmark. It failed before it could serve queries.\n\nThe fix was to distinguish between two different memory concepts:\n\nThe final Docker configuration used:\n\nThat distinction mattered.\n\nThe database heap was still limited to 256 MB, but the JVM needed additional headroom just to run.\n\nThis became one of my favorite architectural observations from the project:\n\nA database memory limit is not always the same thing as a process memory limit.\n\nIt also highlighted a difference between lightweight engines and JVM-based engines. CognoDB’s free tier operates within a very small total footprint, while Java-based Neo4j requires more baseline headroom.\n\nThe traversal results were interesting because latency barely changed as query complexity increased.\n\nIf the graph traversal itself were the dominant cost, we would expect three-hop queries to be noticeably slower than one-hop queries.\n\nBut for several platforms, they were almost identical.\n\nThat suggests that the measured latency was dominated by fixed overhead:\n\nThis is an important lesson for anyone reading database benchmarks.\n\nA slow result does not always mean the engine is slow.\n\nIt may mean:\n\nHonest benchmarks need to say that clearly.\n\nDocker Neo4j had the lowest latency in almost every category.\n\nBut that result must be interpreted carefully.\n\nDocker ran on the same machine as the benchmark client.\n\nThat means:\n\nDocker results are useful as a local engine baseline, but they are not directly comparable to cloud platforms over a network.\n\nThis is one of the reasons I included Docker in the first place.\n\nIt helped separate two questions:\n\nThose are related, but they are not the same.\n\nOne of the requirements I’m glad the assignment enforced was percentile reporting.\n\nDocker Neo4j had very low p50 latency:\n\nBut its p95 latency jumped to around 80 ms in several workloads.\n\nThat gap matters.\n\nAverages could make Docker look almost perfect. But p95 revealed occasional pauses, likely related to JVM behavior, caching, or local scheduling.\n\nIn production, users often notice the tail more than the average.\n\nA system that usually responds in 5 ms but occasionally responds in 100 ms can feel worse than a system that consistently responds in 30 ms.\n\nPercentiles expose that.\n\nUnder the conditions of this benchmark:\n\nMemgraph showed strong ingest throughput and strong cloud latency. Its in-memory architecture appears to benefit both batch loading and read workloads.\n\nAura showed strong managed-cloud behavior, especially considering it was running on a free tier. It scaled well under concurrency and completed mixed workloads with zero recorded errors.\n\nDocker Neo4j produced the lowest local latency, but its results include a localhost advantage and should not be treated as a direct cloud comparison.\n\nSandbox behaved like a managed trial environment with relatively high fixed request latency. It is useful for evaluation, but its internal resource guarantees are not publicly detailed.\n\nCognoDB completed all benchmark workloads with zero mixed-workload errors. Its free tier is intentionally small, and under the tested network path it showed higher request latency and lower throughput than some other platforms.\n\nThe important caveat is that this reflects the free-tier conditions and client network path used in this test. It does not necessarily represent CognoDB’s behavior under larger instances, closer regions, or production configurations.\n\nThis benchmark has limitations, and they matter.\n\nFree instances may be shared, throttled, or burstable. Results can vary over time.\n\nCloud platforms were tested from a residential internet connection. Different regions or networks could produce different results.\n\nSome platforms do not expose exact vCPU, RAM, or storage details for free tiers.\n\nDocker Neo4j avoids cloud network latency, so it should be interpreted as a local baseline.\n\nNeo4j’s JVM required extra headroom beyond the 256 MB database heap.\n\nThe reported latency numbers are warm-run numbers. Cold-start behavior was not separately benchmarked.\n\nEach read workload used 100 iterations, but the full suite was not repeated across multiple days due to the 48-hour assignment window.\n\nAura, Sandbox, and Docker all use Neo4j-based engines. They were included to compare different deployment models. Memgraph provided a different in-memory architecture. Future work could include additional distinct engines such as FalkorDB, ArangoDB, NebulaGraph, or TigerGraph.\n\nIf I had more time, I would extend the benchmark in several ways.\n\nFalkorDB, ArangoDB, NebulaGraph, TigerGraph, and Kùzu would make the comparison more diverse.\n\nMultiple runs across different times would help measure variance.\n\nCold-start latency is important for serverless and scale-to-zero environments.\n\nWhere available, I would collect:\n\nThe current report uses Markdown tables and simple text charts. A chart generator would make the results easier to read.\n\nThe current dataset fits free tiers. A larger dataset would stress indexing, memory, and storage more aggressively.\n\nThe biggest lesson from this project was not “Database X is faster.”\n\nThe bigger lesson was that benchmarking is a systems problem.\n\nYou are not only measuring the database engine.\n\nYou are measuring:\n\nIf you ignore those things, your benchmark may be technically reproducible but practically misleading.\n\nIf you document them, your benchmark becomes useful.\n\nThat was the spirit of this assignment.\n\nNot to hide the messy parts.\n\nNot to smooth over the caveats.\n\nNot to declare a winner.\n\nBut to measure carefully, explain clearly, and leave the next engineer enough information to reproduce the work.\n\nThe benchmark harness, results, and full documentation are available here:\n\n**GitHub repository:** [https://github.com/kalbashi09/Benchmark](https://github.com/kalbashi09/Benchmark)\n\nThe README includes:\n\nIf you are evaluating graph databases, I hope this helps you ask better questions before trusting any benchmark.\n\nIncluding mine.", "url": "https://wpnews.pro/news/benchmarking-5-graph-database-platforms-on-the-same-352k-edge-graph-what-free", "canonical_source": "https://dev.to/kalbashi09/benchmarking-5-graph-database-platforms-on-the-same-352k-edge-graph-what-free-tiers-hide-39jm", "published_at": "2026-08-27 03:38:43+00:00", "updated_at": "2026-08-27 04:18:25.368021+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["Wexa AI", "CognoDB Cloud", "Memgraph", "SNAP", "Neo4j", ".NET"], "alternates": {"html": "https://wpnews.pro/news/benchmarking-5-graph-database-platforms-on-the-same-352k-edge-graph-what-free", "markdown": "https://wpnews.pro/news/benchmarking-5-graph-database-platforms-on-the-same-352k-edge-graph-what-free.md", "text": "https://wpnews.pro/news/benchmarking-5-graph-database-platforms-on-the-same-352k-edge-graph-what-free.txt", "jsonld": "https://wpnews.pro/news/benchmarking-5-graph-database-platforms-on-the-same-352k-edge-graph-what-free.jsonld"}}