Benchmarking 5 Graph Database Platforms on the Same 352k-Edge Graph: What Free Tiers Hide

An engineer from Wexa AI benchmarked five graph database platforms, including CognoDB Cloud, on a 352,768-edge citation network under free-tier constraints. The harness measured ingest, traversal, lookup, aggregation, and mixed workloads, documenting throttling and failures. Memgraph led in ingest speed, while the report highlights how free tiers can distort performance comparisons.

Graph database benchmarks are easy to get wrong. Most public benchmarks compare expensive production instances, use different datasets, ignore network latency, hide free-tier throttling, or quietly optimize one database more than the others. For a recent engineering assignment from Wexa AI , I decided to do the opposite. I built a small benchmark harness that compared CognoDB Cloud against four other graph database platforms using: The platforms tested were: The dataset was the SNAP cit-HepTh citation network: Small enough to fit inside free tiers, but large enough to make the benchmark meaningful. The full benchmark code, results, and documentation are available here: GitHub repository: Add your GitHub repository link The assignment was clear: the goal was not to crown a database. The goal was to evaluate engineering rigor: That matters because real-world database selection is messy. There is no universal “best graph database.” There is only the best database for: So this benchmark was designed to measure behavior under constrained free-tier conditions, not to simulate an unlimited production environment. Before writing code, I set a few rules. Every platform received the exact same graph. All platforms ran the same workloads, even where minor Cypher dialect adjustments were required. No paid production instance was allowed to sneak into the comparison. CognoDB’s free tier is intentionally small: So the other platforms were run on their free, trial, or capped self-hosted tiers. Every read workload was warmed up first. Cold-start behavior was excluded from the main latency numbers and documented as a caveat. Averages can hide bad requests. So the benchmark reported: If something failed, throttled, crashed, or behaved strangely, it got documented. That rule ended up being one of the most valuable parts of the project. I used the SNAP cit-HepTh citation network. It represents arXiv High Energy Physics Theory papers and their citations. If paper A cites paper B, the graph contains a directed edge from A to B. The final cleaned graph contained: | Metric | Value | |---|---| | Nodes | 27,769 | | Relationships | 352,768 | | Node label | Paper | | Relationship type | CITES | Each paper node had: id year field The year property came from the original SNAP metadata. The field property was generated deterministically for aggregation testing, so every platform received identical values. This dataset was a good fit because it had enough relationships to make traversals and aggregations interesting, but it was still small enough to fit into free tiers. I built the benchmark harness in C using .NET 8 and the official Neo4j .NET driver. For each platform, the harness did the following: Then a report generator converted the JSON files into Markdown tables. The workflow was simple: dotnet run -- cognodb dotnet run -- aura dotnet run -- sandbox dotnet run -- memgraph dotnet run -- docker And then: dotnet run -- report No passwords were stored in the repository. All credentials were read from environment variables. The benchmark measured six categories. How fast can the platform load the graph? Measured as: One-hop, two-hop, and three-hop queries from randomly selected start nodes. Example: php MATCH p:Paper {id: $id} - :CITES - - :CITES - q RETURN count q AS c; Point lookup by indexed id : MATCH p:Paper {id: $id} RETURN p.year AS year; Filtered lookup using indexed year : MATCH p:Paper WHERE p.year = $y1 AND p.year <= $y2 RETURN count p AS c; Group-by query over a node property: MATCH p:Paper RETURN p.field AS field, count AS c ORDER BY c DESC; The mixed workload used: Where observable, the benchmark recorded: Where a platform did not expose internals, the result was marked as not observable . Memgraph dominated ingest. | Platform | Relationships/sec | Total Load Time | |---|---|---| | Memgraph | 14,966 | 25.7s | | Docker Neo4j | 8,144 | 55.2s | | Neo4j Aura | 7,865 | 47.6s | | Neo4j Sandbox | 2,339 | 167.9s | | CognoDB | 1,209 | 317.9s | Visual summary: memgraph 14966 rels/sec ████████████████████████████████ docker 8144 rels/sec █████████████████ aura 7865 rels/sec ████████████████ sandbox 2339 rels/sec █████ cognodb 1209 rels/sec ██ Memgraph’s in-memory architecture gave it a major advantage during batched loading. Aura also performed strongly. Docker Neo4j benefited from local execution and no cloud network overhead. CognoDB and Sandbox were slower under the tested free-tier conditions. For traversal latency, Docker Neo4j had the lowest p50 numbers because it ran locally. | Platform | 1-hop p50 | 2-hop p50 | 3-hop p50 | |---|---|---|---| | Docker Neo4j | 3.1 ms | 3.2 ms | 3.3 ms | | Memgraph | 95.3 ms | 95.8 ms | 96.2 ms | | Neo4j Aura | 146.2 ms | 146.5 ms | 146.8 ms | | CognoDB | 644.2 ms | 652.3 ms | 654.2 ms | | Neo4j Sandbox | 697.8 ms | 697.7 ms | 698.0 ms | One pattern stood out. For most platforms, latency barely increased from one hop to three hops. For example: That tells us something important. At this dataset size, the actual traversal work was small compared with fixed overhead. That overhead may include: In other words, the query engine may not have been the dominant cost. The request path was. Lookup latency followed a similar pattern. | Platform | Point Lookup p50 | |---|---| | Docker Neo4j | 3.3 ms | | Memgraph | 95.1 ms | | Neo4j Aura | 146.1 ms | | CognoDB | 643.9 ms | | Neo4j Sandbox | 698.8 ms | | Platform | Aggregation p50 | |---|---| | Docker Neo4j | 13.9 ms | | Memgraph | 106.0 ms | | Neo4j Aura | 156.8 ms | | CognoDB | 683.7 ms | | Neo4j Sandbox | 704.6 ms | Docker’s low numbers again reflected its localhost advantage. Among cloud platforms, Memgraph and Aura were the strongest in this test. CognoDB remained stable, but its latency was dominated by what appeared to be fixed request overhead under the tested free-tier environment. The mixed workload used 80% reads and 20% writes. At 40 concurrent clients: | Platform | QPS | Errors | |---|---|---| | Docker Neo4j | 378.4 | 0 | | Memgraph | 374.4 | 5 | | Neo4j Aura | 287.7 | 0 | | CognoDB | 61.2 | 0 | | Neo4j Sandbox | 51.3 | 0 | Visual summary: docker 378.4 QPS ████████████████████████████████ memgraph 374.4 QPS ███████████████████████████████ aura 287.7 QPS ████████████████████████ cognodb 61.2 QPS █████ sandbox 51.3 QPS ████ Docker and Memgraph achieved the highest throughput. Aura scaled well and completed the workload with no recorded errors. CognoDB and Sandbox showed lower absolute throughput, but both completed with zero recorded errors. Memgraph produced five errors during the 40-client workload. I kept those errors in the results because hiding them would defeat the purpose of an honest benchmark. The likely explanation is resource pressure on the entry-tier instance under concurrent writes. One of the most interesting findings came from Docker Neo4j. CognoDB’s free tier gives you: I wanted the local Docker Neo4j instance to be as close to that as possible. So I first capped the Docker container at 256 MB. Neo4j crashed during startup. It did not fail during the benchmark. It failed before it could serve queries. The fix was to distinguish between two different memory concepts: The final Docker configuration used: That distinction mattered. The database heap was still limited to 256 MB, but the JVM needed additional headroom just to run. This became one of my favorite architectural observations from the project: A database memory limit is not always the same thing as a process memory limit. It also highlighted a difference between lightweight engines and JVM-based engines. CognoDB’s free tier operates within a very small total footprint, while Java-based Neo4j requires more baseline headroom. The traversal results were interesting because latency barely changed as query complexity increased. If the graph traversal itself were the dominant cost, we would expect three-hop queries to be noticeably slower than one-hop queries. But for several platforms, they were almost identical. That suggests that the measured latency was dominated by fixed overhead: This is an important lesson for anyone reading database benchmarks. A slow result does not always mean the engine is slow. It may mean: Honest benchmarks need to say that clearly. Docker Neo4j had the lowest latency in almost every category. But that result must be interpreted carefully. Docker ran on the same machine as the benchmark client. That means: Docker results are useful as a local engine baseline, but they are not directly comparable to cloud platforms over a network. This is one of the reasons I included Docker in the first place. It helped separate two questions: Those are related, but they are not the same. One of the requirements I’m glad the assignment enforced was percentile reporting. Docker Neo4j had very low p50 latency: But its p95 latency jumped to around 80 ms in several workloads. That gap matters. Averages could make Docker look almost perfect. But p95 revealed occasional pauses, likely related to JVM behavior, caching, or local scheduling. In production, users often notice the tail more than the average. A system that usually responds in 5 ms but occasionally responds in 100 ms can feel worse than a system that consistently responds in 30 ms. Percentiles expose that. Under the conditions of this benchmark: Memgraph showed strong ingest throughput and strong cloud latency. Its in-memory architecture appears to benefit both batch loading and read workloads. Aura showed strong managed-cloud behavior, especially considering it was running on a free tier. It scaled well under concurrency and completed mixed workloads with zero recorded errors. Docker Neo4j produced the lowest local latency, but its results include a localhost advantage and should not be treated as a direct cloud comparison. Sandbox behaved like a managed trial environment with relatively high fixed request latency. It is useful for evaluation, but its internal resource guarantees are not publicly detailed. CognoDB completed all benchmark workloads with zero mixed-workload errors. Its free tier is intentionally small, and under the tested network path it showed higher request latency and lower throughput than some other platforms. The important caveat is that this reflects the free-tier conditions and client network path used in this test. It does not necessarily represent CognoDB’s behavior under larger instances, closer regions, or production configurations. This benchmark has limitations, and they matter. Free instances may be shared, throttled, or burstable. Results can vary over time. Cloud platforms were tested from a residential internet connection. Different regions or networks could produce different results. Some platforms do not expose exact vCPU, RAM, or storage details for free tiers. Docker Neo4j avoids cloud network latency, so it should be interpreted as a local baseline. Neo4j’s JVM required extra headroom beyond the 256 MB database heap. The reported latency numbers are warm-run numbers. Cold-start behavior was not separately benchmarked. Each read workload used 100 iterations, but the full suite was not repeated across multiple days due to the 48-hour assignment window. Aura, Sandbox, and Docker all use Neo4j-based engines. They were included to compare different deployment models. Memgraph provided a different in-memory architecture. Future work could include additional distinct engines such as FalkorDB, ArangoDB, NebulaGraph, or TigerGraph. If I had more time, I would extend the benchmark in several ways. FalkorDB, ArangoDB, NebulaGraph, TigerGraph, and Kùzu would make the comparison more diverse. Multiple runs across different times would help measure variance. Cold-start latency is important for serverless and scale-to-zero environments. Where available, I would collect: The current report uses Markdown tables and simple text charts. A chart generator would make the results easier to read. The current dataset fits free tiers. A larger dataset would stress indexing, memory, and storage more aggressively. The biggest lesson from this project was not “Database X is faster.” The bigger lesson was that benchmarking is a systems problem. You are not only measuring the database engine. You are measuring: If you ignore those things, your benchmark may be technically reproducible but practically misleading. If you document them, your benchmark becomes useful. That was the spirit of this assignment. Not to hide the messy parts. Not to smooth over the caveats. Not to declare a winner. But to measure carefully, explain clearly, and leave the next engineer enough information to reproduce the work. The benchmark harness, results, and full documentation are available here: GitHub repository: https://github.com/kalbashi09/Benchmark https://github.com/kalbashi09/Benchmark The README includes: If you are evaluating graph databases, I hope this helps you ask better questions before trusting any benchmark. Including mine.