{"slug": "making-a-trino-lake-as-fast-as-a-sql-database", "title": "Making a Trino lake as fast as a SQL database", "summary": "A developer benchmarked an AI agent answering business questions over Iceberg tables on Azure storage via Trino against the same data in Azure SQL, and closed the gap with four fixes. The largest gain came not from Trino tuning but from replacing per-query remote Polaris catalog lookups with a local Postgres-backed JDBC Iceberg catalog, cutting analysis time on a 25-table question from about 13 seconds to 0.8 seconds with byte-identical results. Precomputing the 25-table accounting model into a single table after each sync brought the full question from roughly 30 seconds to parity with the database.", "body_md": "*Photo by Joshua Kettle on Unsplash*\n\nMoving analytics data from a SQL database to a lake usually comes with a quiet assumption: you will pay for cheap, unlimited storage with slower answers. I tested that assumption end to end, with an AI agent that answers business questions over synced SaaS data, and the result surprised me. With four fixes, full answers on the lake took the same time as on the database. The biggest fix had nothing to do with Trino's speed.\n\n**The setup, in one breath**\n\nThe data is CRM and accounting records, copied by a managed ingestion service into Iceberg tables on Azure storage (ADLS). Iceberg is an open table format: plain files plus a small \"description file\" per table saying which files make up the table right now. A hosted Polaris catalog keeps the list of where each table's description lives. Trino is the query engine that reads all of it. The comparison point is the same data in Azure SQL.\n\nThe agent doesn't run one query per answer. It runs five to seven: it lists tables, reads column names, samples a few values, then asks the real question. That detail turned out to matter more than anything else.\n\n**Out of the box, the lake lost badly**\n\nA small question touching four tables took about 2 seconds in Trino, against about 0.1 seconds in the database. A big accounting question that combines 25 tables took about 30 seconds, against about 6.\n\nI expected the time to be in reading data. It wasn't. Trino splits its own work into stages, and most of the time sat in \"analysis\", the step before any data is read.\n\nTrino asks the catalog about every table, in every query, with no memory between queries. Each ask is a network call. Our catalog sat in one region and the engine in another, so each lookup cost about half a second. Twenty-five tables, twenty-five calls, about 13 seconds of waiting before the engine did anything useful.\n\nThink of a delivery driver who phones head office for the address of every stop, every single day, even though the addresses never changed.\n\nThe fix was to give the driver a notebook. Trino supports an Iceberg catalog stored in a plain Postgres database (the [JDBC catalog in Trino's docs](https://trino.io/docs/current/object-storage/metastores.html)). \"Registering\" a table there is just writing one row: its name and the location of its current description file, which I read from the table's own `$metadata_log_entries`. No data is copied. Nothing is written to the lake.\n\n*Before and after: the engine stops phoning the catalog and reads a local address book instead.*\n\nThe result was the clearest number of the whole project. Analysis on the 25-table question went from about 13 seconds to 0.8. On the small question it went from 1.5 seconds to 0.17. Answers were byte-for-byte identical.\n\nThe price is a job you now own. Every time new data lands, the notebook has to be updated with each table's new description file, or the engine reads yesterday's data.\n\nA newer Trino setting, `snapshot-loading-mode=REFS`, asks the catalog for a shorter answer that leaves out old table history. Someone on another catalog [measured a 6x faster lookup with it](https://github.com/trinodb/trino/pull/30612), but their tables had about 9,000 old versions. Mine had one. I tried it anyway: no gain. The time was the trip, not the size of the reply.\n\nTrino's metadata cache was already on, and it caches description files, not the catalog lookup. Rewriting or compacting the source tables was off the table too, since the ingestion service owns them and would undo it on the next sync.\n\nRemoving the lookups took the big accounting question from about 30 seconds to about 17. What remained was real work: about 8 seconds planning a query with hundreds of stages, and about 8 seconds running it.\n\nSo I stopped doing that work at question time. The accounting \"model\" (the recipe that turns 25 raw tables into one clean ledger) runs once after each sync and saves its result as a single table. A question like \"income in March\" then reads one small table instead of rebuilding the ledger.\n\n*The heavy combining happens once per data refresh, not once per question.*\n\nI tried the ready table in three places: in Trino's memory, in Postgres, and as an Iceberg table on the lake itself. All three answered in about 0.15 to 0.3 seconds, against about 6 seconds in the SQL database. Writing it to the lake took about 30 seconds. Writing it into Postgres took about 90, because rows go in one by one.\n\nThat last point matters if you worried, like I did, that precompute means \"we're back to a database.\" The data stays in the lake. Postgres only holds pointers, and you can rebuild it from the catalog at any time.\n\nThe engine first ran on a VM on the US west coast, with the data on the east coast. A single round trip across the country took 65 to 100 milliseconds, and each catalog call needed several. Moving the engine east cut a call from 0.21 to 0.40 seconds down to 0.06 to 0.16, and small questions got about twice as fast again, landing around half a second.\n\nThen I made the mistake that taught me the most. I timed full answers with the app on my laptop.\n\nFull answers on the lake averaged 53.7 seconds. On the SQL database, 44.3. Same correct answers, nine seconds worse. The engine was fine; the app was talking to it five to seven times per answer, across the internet, several round trips each time.\n\nWhen I ran the app on the same machine as the engine, the lake averaged **42.5 seconds**. Inside the noise of the database's 44.3.\n\n*Same engine, same questions. Only the app's location changed.*\n\nAbout 40 seconds of every answer is the AI thinking, on either system, and it swings by five seconds from run to run. Per-query benchmarks hid that. If you are timing a database for an agent, time the whole answer, and put the app where it will live in production.\n\nAn answer cache in the app, keyed on the exact SQL text, did what it promised. A repeated 18-second question came back in under a millisecond, and by the third question in a session only the real query reached the engine.\n\nIt saved about a second per answer. I've seen caching cut query latency by 60% on a high-traffic payment system before, where queries were the whole cost. Here the AI is the whole cost, so the same trick barely registered. Worth keeping, not worth leading with.\n\nStarRocks read the same catalog and the same storage with the VM's managed identity (the machine's own Azure login, no passwords) on the first try ([its Azure auth docs](https://docs.starrocks.io/docs/integrations/csp_auth/authenticate_to_azure_storage/) cover it), and gave identical answers. With none of the fixes above, small questions took 0.4 to 0.7 seconds and the big accounting questions 7 to 10. That is faster than plain Trino out of the box, likely because it keeps table information between queries and shares repeated query blocks instead of planning each copy. The catch is a different SQL dialect, so every query the agent writes would need porting.\n\nApache Doris connected to the catalog and listed every table, then failed on the first read. Its [Azure storage docs](https://doris.apache.org/docs/4.x/lakehouse/storages/azure-blob) list account-key access only, and I wasn't going to hand a query engine a key to the whole storage account. If you use managed identities on Azure, check this before anything else.\n\nProfile the step before the data. On a lake, the slow part is often looking things up, not reading. Keep a local copy of the catalog's answers, precompute the few heavy models, and put the engine and the app next to the data. Then time whole answers, not queries.\n\nThe lake was never slow. It was asking for directions.", "url": "https://wpnews.pro/news/making-a-trino-lake-as-fast-as-a-sql-database", "canonical_source": "https://dev.to/thisismairaj/making-a-trino-lake-as-fast-as-a-sql-database-5ccj", "published_at": "2026-11-20 05:39:13+00:00", "updated_at": "2026-11-20 05:43:15.297288+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "ai-tools"], "entities": ["Trino", "Iceberg", "Polaris", "Azure SQL", "Azure", "Postgres", "ADLS"], "also_reported_by": [], "takeaway": "Swapping Trino's remote Polaris catalog lookups for a local Postgres JDBC Iceberg catalog cut analysis time on a 25-table question from about 13 seconds to 0.8 seconds, and precomputing the multi-table model after each sync removed the remaining planning cost.", "recommendation": {"advice": "If your agent fires many short queries against a remote catalog, register Iceberg tables in a local JDBC catalog and refresh it after each sync to avoid per-query network lookups.", "playbook": "production-ops", "playbook_url": "https://wpnews.pro/playbooks/production-ops"}, "alternates": {"html": "https://wpnews.pro/news/making-a-trino-lake-as-fast-as-a-sql-database", "markdown": "https://wpnews.pro/news/making-a-trino-lake-as-fast-as-a-sql-database.md", "text": "https://wpnews.pro/news/making-a-trino-lake-as-fast-as-a-sql-database.txt", "jsonld": "https://wpnews.pro/news/making-a-trino-lake-as-fast-as-a-sql-database.jsonld"}}