AlloyDB Ships Proxy Models That Replace LLM Calls with Local Inference Inside the Database

Google announced the general availability of AlloyDB AI functions, which allow developers to call large language models directly within SQL queries, along with two acceleration techniques: smart batching and proxy models. The proxy model architecture trains a lightweight local model inside the database using a sample of data and a frontier LLM, then applies that model locally for inference, achieving up to 100,000 rows per second and significant cost reductions. This pattern inverts the typical database-to-LLM relationship, turning the LLM into a teacher rather than a runtime dependency.

Google recently announced https://cloud.google.com/blog/products/databases/boost-performance-and-lower-costs-with-alloydb-ai-functions the general availability GA of AlloyDB AI functions https://docs.cloud.google.com/alloydb/docs/ai/ai-query-engine-landing , along with two acceleration techniques that fundamentally change how databases interact with large language models LLMs . According to Google's internal testing, smart batching delivers a 2,400x improvement in throughput over row-at-a-time processing, while optimized proxy models push that to 23,000x with a 6,000x reduction in cost. While these headline numbers warrant engineering scrutiny, the underlying proxy model architecture is what practitioners should focus on. With AlloyDB AI functions, developers can call LLMs directly within standard SQL queries. The GA release includes ai.generate for text generation, ai.if for semantic filtering, ai.rank for semantic reranking, ai.forecast for time-series prediction, and three new additions: ai.summarize , ai.agg summarize for group-level summaries,- and ai.analyze sentiment . Because these execute as standard SQL operators, a query can filter rows by meaning rather than rigid keyword matches. The problem with calling an LLM per row is obvious at scale: a table with 100,000 products means 100,000 round trips to Vertex AI, each carrying the same system prompt, each waiting for model inference, and each incurring per-token costs. Two acceleration layers address this. Smart batching, now available for ai.if and ai.rank , groups multiple rows into a single model call. Instead of sending the system prompt with every row, AlloyDB sends it once and batches the data. Google reports throughput of up to 10,000 rows per second in internal testing, a 2,400x improvement over the row-at-a-time baseline. The technique is straightforward, and the gains are credible for workloads where the per-row payload is small relative to the prompt. The proxy model is the more architecturally significant move. For ai.if queries currently in preview , AlloyDB introduces a two-phase workflow: -- Phase 1: Train the local proxy model using a sample of data and the frontier LLM PREPARE underwater suitability proxy FROM SELECT description FROM products; -- Phase 2: Execute the query at database speed using the local proxy model SELECT FROM products WHERE ai.if description, 'suitable for underwater use deeper than 60 meters' USING proxy underwater suitability proxy ; First, a PREPARE statement sends a sample of your data to a frontier model and uses the results to train a lightweight local model inside the database. Then, EXECUTE runs the query using the local proxy instead of calling the external LLM. AlloyDB falls back to the frontier model if the proxy's confidence is too low or if no trained model is available. Google reports throughput of 100,000 rows per second with this approach. The proxy model pattern inverts the usual database-to-LLM relationship. Instead of the database being a client that calls an external model for every decision, the database becomes a student that learns the model's judgment on a sample, then applies that judgment locally at database speed. The LLM becomes a teacher rather than a runtime dependency. This pattern has implications beyond AlloyDB. Any database that calls external models for per-row decisions faces the same cost and latency wall. The question is whether competitors Aurora, Azure SQL, CockroachDB, PlanetScale will adopt similar distillation-at-query-time approaches, or whether they will rely on users building this logic in application code. Practitioners should note the caveats in Google's own announcement https://cloud.google.com/blog/products/databases/boost-performance-and-lower-costs-with-alloydb-ai-functions : the 23,000x and 6,000x numbers come from internal testing, apply specifically to ai.if in preview, and do not represent the performance of all AI functions generally. The optimized proxy model is not yet GA. Teams evaluating AlloyDB for production AI workloads should benchmark against their own data distributions and query patterns before committing. Raimundas Juodvalkis, a Starburst architect, offered practical framing on LinkedIn https://www.linkedin.com/posts/raimundas-juodvalkis-63b5a916 enterpriseai-aiarchitecture-dataengineering-share-7478431237833949185-Mf3G : Treat them as governed database extensions, not magic WHERE clauses. He recommended starting with read-heavy review workflows before writing model-derived fields back into core systems, and tracking model cost separately from query cost. The release also includes a managed MCP server for AlloyDB, letting AI agents query database content through the Model Context Protocol without teams running and scaling their own MCP infrastructure. Combined with existing vector search using Google's ScaNN index up to 10 billion vectors , AlloyDB is positioning itself as a PostgreSQL-compatible database where structured queries, semantic search, and LLM-powered analysis coexist in the same SQL layer. AlloyDB AI functions are generally available https://cloud.google.com/alloydb/ai on PostgreSQL 17 instances. Smart batching is GA for ai.if and ai.rank . The optimized proxy model for ai.if is in preview. AI function acceleration requires setting a database flag google ml integration.enable ai function acceleration and is not enabled by default https://docs.cloud.google.com/alloydb/docs/ai/accelerate-ai-queries .