# We tried to benchmark SLayer and Cube: we failed

> Source: <https://motley.ai/blog-posts/we-tried-to-benchmark-slayer-and-cube-we-failed>
> Published: 2026-09-02 00:00:00+00:00

[← All posts](https://motley.ai/blog)

# We tried to benchmark SLayer and Cube: we failed

After [our experiments on LiveSQLBench-Large](https://motley.ai/blog-posts/when-the-database-gets-big-the-semantic-layer-earns-its-keep), which showed a SLayer-using agent passing over 30% more tasks than one writing raw SQL, I wanted to compare that to an agent using Cube, one of the incumbent open-core semantic layers.

For this evaluation, I focussed on the open source version of Cube. Since that doesn’t come with an MCP server, I wrote tools for the agent that wrapped the REST API, for Cube introspection and for running queries. Open source Cube doesn’t expose an API to modify models directly (it does provide a `RepositoryFactory`

approach that lets you manage your own model store and trigger recompiles when needed, but that was a step too far for a first attempt).

To keep the setup close to the agents described in the previous post, I auto-generated the Cube data models from the database schemas, and created custom dimensions and measures from the leaves of the documented JSON columns in the benchmark. Since Cube has no equivalent to SLayer’s memories or semantic retrieval tools, I gave the agent the same tools to access the knowledge base text items that the raw SQL-writing agent had.

For cost reasons, I selected a random set of 10 tasks from LiveSQLBench-Lite that SLayer had already succeeded on, to give the Cube agent an easy start (they’re listed in the table at the bottom). The SQL-writing agent had managed to solve 5 of these; the sample is too small to read much into the exact numbers.

The number of tasks the Cube-using agent could solve: zero.

## Why it failed

The natural question was whether this was a bug in the agent, or something similarly mundane. Inspecting the run logs turned up two blockers instead:

**The schema metadata didn’t fit.** What Cube returns when inspecting the models is extremely verbose, verbose enough that in many cases it went over the 25K limit on tool outputs that Claude Code sets (the agent ran on the Claude SDK). The agent never saw the full schema, and was left trying to hallucinate the right query.**Most tasks needed custom measures the agent couldn’t add mid-task.** Every one of the 10 tasks is in principle solvable with Cube queries, but 9 of them require custom measure or dimension definitions derived from the knowledge base items. Those definitions would need to be created during task solving, since earlier experiments showed that converting knowledge base items into configs ahead of time only helped when each conversion could be immediately validated by the task grader.

I considered writing a model-editing tool for the agent, but that means building a custom external model store, then a mechanism to validate every agent edit (triggering a recompile on each edit and getting the results back to the agent). That’s a fair amount of engineering to turn open source Cube into something it isn’t, and even then, refreshing the models that way would still take seconds.

So for Cube to become usable by agents at all, it would need its own schema introspection and smart retrieval, plus a live(ish) mechanism for updating schemas, just to get started. Whether agents would then handle its multistage query syntax well is a question I can’t answer until that exists.

These are the same kinds of challenges, and the same recurring quirks in how it handled multistage queries, that pushed us to migrate off Cube and build SLayer in the first place.

So if you’re planning to use Cube with agents, budget for writing a solid part of that scaffolding yourself, or use a semantic layer built agent-native from the start, like SLayer, and skip the trouble.

## Appendix: the chosen tasks, and why each needed custom model configs

A task clears the “no custom config needed” bar only if its projected and aggregated quantities are raw columns (or their auto-generated aggregates), and its predicates compare a single existing member to constants. A regular Cube query has no expression layer of its own.

| Task | Doable with the schema-generated Cube model? | The blocking item (first of possibly several) |
|---|---|---|
`disaster_relief_12` | ✅ Yes | None: `break_rate_avg` (auto-generated avg over a documented JSON leaf) plus a `transport_access = 'Minimal'` filter; the knowledge base content reduces to a name-to-member mapping, which the embedded column descriptions already carry. |
`fake_account_1` | ❌ | Projected `√(FollGrow² + FingGrow²)` : arithmetic in the output column, and the formula exists only in knowledge base prose. |
`cybermarket_pattern_10` | ❌ | `avg(kmc/mct)` : both JSON leaves exist as numeric dimensions, but avg-of-ratio isn’t any combination of the generated `_avg` /`_sum` measures. |
`planets_data_1` | ❌ | Measure `avg(mass·317.83 / (radius·11.209)²)` . The filters could be emulated on existing dimensions (converting the mass range through the constant, `densvalue > 3` directly), but the aggregated ratio can’t be. |
`labor_certification_applications_10` | ❌ | Grouping by a CASE (`complexity_tier` ): group-by only takes existing dimensions. Plus `apt_days` is a date-diff over two text columns in different date formats. |
`polar_equipment_10` | ❌ | Projected cross-cube `vpc_score` . Also `speed` is trapped as a string dimension (`"22.30 m/s"` fails the numeric-cast guard and needs unit parsing). |
`cross_border_10` | ❌ | Projected `acp` chain (afs·dsrl·…) plus `days_overdue` date arithmetic with a task-supplied anchor date. |
`mental_healths_10` | ❌ | Ordinal CASE mappings (severe→3…) and the EAS/RDD formulas and thresholds, all knowledge-base-only. Also needs a non-FK value join and `count_distinct` /`sub_query` members the generator doesn’t emit. |
`museum_artifact_11` | ❌ | Windowed deviation-from-year-average: an analysis concept absent from schema and docs. `count_distinct` enrichment alone doesn’t create it. |
`exchange_traded_funds_1` | ❌ | `yter` ratio, rating whitelist, `sies` product, thresholds, all knowledge-base-only, plus the rank column (Cube multistage). |