How to upgrade embedding models in AlloyDB AI with zero downtime

Google's AlloyDB AI now supports zero-downtime embedding model upgrades by combining model endpoint management, auto embeddings that generate up to 3,000 embeddings per second in the background, and ScaNN indexes built 10x faster than open source HNSW, the company said in a technical blog post. The post walks through migrating a production table, ecomm.product_reviews, from text-embedding-005 (768 dimensions) to gemini-embedding-001 (3072 dimensions) while serving live traffic, using a nullable column added without defaults, a 2-second lock_timeout, concurrent index creation, and atomic column renames for blue/green and canary rollouts. The upgrade requires the google_ml_integration extension at version 1.5.6 or later, tested on 1.6.1.

Imagine you've just launched a new semantic search feature for product reviews to help customers quickly research products on your site. Users love it. But the next day, you see an announcement for a new vector embedding model promising higher accuracy, better nuance, and multilingual search out of the box. After testing it in your dev environment, you confirm it's a perfect fit. But now you're left with a common dilemma: How do you upgrade the model without disrupting your users? Upgrading embedding models in production requires re-embedding your entire dataset, rebuilding indexes, and coordinating schema migrations without interrupting user queries. You may encounter several potential problems: Embedding generation: All existing data must be re-embedded using the new embedding model, which typically requires updating complex pipelines and using dual write strategies while the new and old models run side-by-side for a period of time. Mathematical incompatibility: Embeddings generated by different models inhabit distinct vector spaces and cannot be compared directly. Application query vectors and stored column vectors must switch models in unison to prevent severe search degradation. Index rebuild overhead: Vector indexes must be built on the new vectors before similarity search can perform efficiently at scale. Degraded user experience: Switching embedding models requires multiple DDL changes and other operations that acquire exclusive locks and can stall incoming read and write transactions if not carefully managed. In this blog, we’ll highlight how the rich feature set in AlloyDB AI https://docs.cloud.google.com/alloydb/docs/ai/what-is-alloydb-ai combines to enable zero-downtime embedding model upgrades, so that your users can benefit from the latest model advances without scheduled downtime maintenance. Model endpoint management https://docs.cloud.google.com/alloydb/docs/ai/model-endpoint-overview gives you native in-database access to any embedding model first party, third party, or even self-hosted , while auto embeddings https://docs.cloud.google.com/alloydb/docs/ai/generate-manage-auto-embeddings-for-tables rapidly generates new embeddings in the background up to 3000/second and keeps them up to date as source data changes. Using AlloyDB’s ScaNN index https://docs.cloud.google.com/alloydb/docs/ai/create-scann-index can also significantly reduce the wall clock time of your upgrade with index builds that are 10x faster than open source HNSW, and native PostgreSQL functionality like concurrent index creation, set-returning functions SRF , and atomic column renames allow transparent blue/green and canary-style rollouts. In this example, we demonstrate migrating a production table ecomm.product reviews from text-embedding-005 768-d to gemini-embedding-001 3072-d while serving live traffic. Verify that the google ml integration extension is updated to version 1.5.6 or later tested on 1.6.1 and confirm model connectivity: Adding a nullable column without default values updates catalog metadata without creating new versions of each row. This makes the operation nearly instant. Note that the size of your vector column 3072 in this example must match the number of dimensions output by your embedding model. We also set the lock timeout to 2 seconds to prevent the operation from blocking critical production traffic while it waits to acquire an exclusive lock. If the operation fails due to a longer wait for an exclusive lock, simply retry. Next, we initialize embeddings on our new embedding column using gemini-embedding-001 . Note that there are two modes available in auto embeddings to keep embeddings up to date as source data changes: Transactional: When you set incremental refresh mode to transactional , embeddings are updated immediately when data changes. This keeps your embeddings transactionally consistent with your source data, but it also introduces additional write latency with an inline embedding model inference required for every INSERT/UPDATE operation. Use transactional mode when you want to keep your embedding data transactionally consistent with the source data and your rate of change is low. Manual: When you set incremental refresh mode to manual , AlloyDB tracks which rows have changed but does not immediately update the embedding. This allows you to quickly regenerate embeddings for only the changed rows not the full dataset on a schedule that you define using ai.refresh embeddings , which uses efficient batching techniques just like the initial backfill operation. Use manual mode when you have a high rate of change on your source data and you can tolerate embedding drift for the period of time defined by your scheduled refresh interval. We use manual mode in this example to keep writes on the base table performant, while still having full control of how quickly changes are reflected in the embeddings by periodically running ai.refresh embeddings . Managing concurrent writer wait times During backfill, AlloyDB Auto Embeddings processes rows in parallel groups batch size × max parallel requests . While the process for each micro batch is usually fast, concurrent updates or deletes have to wait for the batch to commit. To cap the maximum wait times for writers on busy transactional tables, you can lower max parallel requests during the backfill session: This change generally improves the commit time for each micro batch, while increasing the total wall clock time for the backfill operation. Our benchmark results on a 10,000-row table demonstrate the trade-off: | max parallel requests | Longest Concurrent Writer Wait | Total Backfill Duration | |---|---|---| | 400 Default | 5.1 seconds | 5.2 seconds | | 20 | 0.39 seconds | 7.6 seconds +46% | AlloyDB provides a view to track progress for auto embedding jobs, including state, rows processed, and estimated time remaining. To see how things are progressing, run the query below from a separate session: Once the initial backfill operation completes, we can optionally regenerate embeddings for rows that were modified while it was running. When using manual mode, Auto Embeddings automatically tracks which rows were changed so that they can be quickly updated without re-embedding data that is already current. To catch those rows up, simply execute a quick refresh using ai.refresh embeddings . You should also run this refresh command on a schedule using pg cron or an external scheduler to ensure the latest changes continue to be reflected in your embeddings . Now that the embeddings are generated and caught up, we’ll ensure completeness and correctness prior to traffic cutover. Note that in a live product environment, you will likely have active INSERT / UPDATE / DELETE traffic that modifies rows between steps 3 and 4. To prevent the results below from being skewed, we added a filter on google ml track stale embedding = ' false ' to include only rows that haven’t been changed since the last invocation of ai.refresh embeddings . Note that your table may use a different name for this column, such as google ml track stale embedding 2 , so be sure to update the query accordingly. At this point, you should also validate your application logic using the new embedding model. In internal benchmarks comparing text-embedding-005 and gemini-embedding-001 , the top-10 nearest neighbor results overlapped by only 31%. This is a good thing because the shift reflects the higher semantic fidelity of the 3072-d model. However, the new results for recall@10 may impact assumptions in your application, so be sure to conduct offline testing against a set of sample golden queries before routing production traffic to the new embeddings. Searching across a large number of vectors without an index is a computationally expensive and slow process. To avoid this, we’ll build a ScaNN index to enable industry leading filtered vector search performance, which is 10x faster than HNSW on open source PostgreSQL. High dimensional models like gemini-embedding-001 require additional memory during index construction compared to lower dimensional models, so in this step we increase maintenance work mem to accommodate the additional overhead. We also build the ScaNN index using CONCURRENTLY to avoid blocking concurrent writes: Building a ScaNN index is also up to 10x faster than HNSW on open source PostgreSQL. In our tests on 100,000 rows, a 3072-d ScaNN index built in approximately 50 seconds, whereas an HNSW index using halfvec took 7 to 13 minutes. Be sure to also tune scann.num leaves to search to balance recall and latency. In our tests, increasing num leaves to search from the default value to 25 improved Recall@10 from 0.35 to 0.95 at a search latency of 8.8 ms: At this stage, you have two primary options for your application cutover strategy: Pattern A: Atomic column rename swap — This is a simple approach that supports blue/green deployments with zero application code changes. It does not, however, support canary-style releases or auto embeddings in transactional mode supports manual mode only . Pattern B: Custom GUC session variables + unified UNION ALL set-returning function SRF — This more advanced approach enables transparent query embedding model selection, live incremental traffic splitting, blue/green deployments, and canary A/B testing, while requiring only one application config change. This technique also supports auto embeddings in both transactional and manual modes. First, we route application query embedding generation through a custom database function to abstract the model from the application. Application queries will use this function instead of directly using ai.embedding : Then, when we’re ready to cut over, we simply execute a stored procedure like the example below to swap the column names and update the function to use the new model. Create the procedure: Then execute the swap: Because the column rename and model switch execute in the same transaction, queries never encounter a model mismatch. If you need to roll back, simply call the procedure with the column arguments reversed. One advantage to this approach is that there are zero changes required on the application side. Your application code consistently uses the ecomm.product reviews query embedding function to generate an embedding, and the column name used by the application query does not change. While this column rename technique is simple, it has two limitations: First, it applies exclusively to manual mode for auto embeddings. In transactional mode, AlloyDB-managed triggers reference columns by hardcoded names, so renaming a column causes write operations to fail. Second, this pattern only supports blue/green style releases. You don’t have per-query control over which model you use, so you’re unable to use canary-style releases with, for example, 5% of traffic using the new model for testing while 95% of the traffic continues to use the existing model. To execute canary rollouts, traffic splitting, or instant rollbacks, you can use custom PostgreSQL Custom Grand Unified Configuration GUC variables paired with a set-returning function abstraction. 1. Define the table-valued function abstraction layer Create a database SRF that inspects a session-level GUC variable app.embedding model to dynamically evaluate search queries against either the legacy or the new embedding column. 2. Standardized application queries The application queries the unified SRF while generating query vectors via ai.embedding using the active model parameter: 3. Execute canary traffic splitting and A/B testing Application connections or middleware can dynamically control which model processes a query by setting the session variable: Using this pattern, engineering teams can route a small percentage of application sessions to the new embedding model, validate accuracy and latency metrics, then gradually increase traffic to 100%. Or, they can revert instantly to the prior model by simply altering the session parameter. A note on connection pooling: When using connection poolers such as PgBouncer in transaction-mode, a plain SET stays on the pooled server connection. This means it can leak to other clients, and it may not always apply to your own subsequent transactions. To mitigate this, you can use SET LOCAL inside an explicit transaction so the setting is cleared at COMMIT or ROLLBACK . You can also prevent this condition entirely by using PgBouncer in session mode, which clears session parameters when a connection is returned to the pool. Once you have completed the cutover to the new model and you are happy with its functionality and stability, you can decommission the legacy embedding column, index, and auto embedding configs to reclaim disk space. Drop the auto embedding config and the old index: Executing ALTER TABLE DROP COLUMN removes the column from PostgreSQL catalog tables, but the underlying TOAST space remains allocated on disk until data pages are rewritten. To make this disk space reusable for future operations, you can set the old vector column to NULL in batches before dropping it. Note that this operation is a trade-off: It creates new row versions for every row in the table in order to make disk space reusable, which requires a VACUUM to clean up the old row versions. You should determine whether it is a priority for you to immediately free up disk space for reuse on the same table versus allowing the old row versions to naturally age out through the course of normal operations. In our tests on 100,000 rows, zeroing out the vectors and vacuuming before dropping the column reclaimed 381 MB of TOAST space for reuse by subsequent model upgrades. If your application uses auto embeddings with incremental refresh mode = 'transactional' , vectors are generated synchronously inside each write transaction. Upgrading in transactional mode follows the same sequence shown above, with three key differences: Step 3 is skipped: You don’t need to run ai.refresh embeddings , because new writes update both columns automatically. Column swap technique is not supported: You must use the GUC + SRF abstraction method, which does not impact the triggers managed by auto embeddings. Write latency increases during migration: While both models are configured, each row write executes two remote embedding calls. Because gemini-embedding-001 generates a more expressive 3072-dimensional embedding, its remote inference call takes longer. In our tests, we observed the following latency profiles for single-row UPDATE operations with transactional mode enabled for auto embeddings: | Migration Phase | Active Models | Median Single-Row Write Latency | |---|---|---| | Baseline | text-embedding-005 | ~37.5 ms | | Active Migration | Both Models Active | ~191.0 ms | | Post-Cutover | gemini-embedding-001 | ~172.0 ms | Be sure to review your application's write latency budget before choosing transactional mode for auto embeddings, especially with higher-dimension models. Because old and new vector columns coexist during migration, your table storage will grow based on peak usage. Based on our testing, we observed the following average storage per row for each embedding type: | Embedding Model | Output Dimensions | Estimated Storage Per Row | |---|---|---| | text-embedding-005 | 768 | ~4.5 KB / row | | gemini-embedding-001 | 3072 | ~13.3 KB / row | Based on these estimates, you can use the formula below to estimate your storage space: New vector column: Total Rows × ~13.3 KB approx. 3x legacy storage . New ScaNN index: Total Rows × ~0.6 KB. Headroom Allocation: Add ~25% overhead for WAL generation and page updates during backfill and vacuum operations. Example: A table with 10 million rows requires approximately 133 GB of disk space for the new vector column, 6 GB for the ScaNN index, plus headroom. AlloyDB combines native PostgreSQL primitives with built-in AI capabilities to make upgrading vector embedding models in production faster and earlier, with no maintenance downtime or complex application-side changes. By combining AlloyDB AI auto embeddings and Model Endpoint Management with standard PostgreSQL operational controls, database administrators and developers can keep their semantic search models up to date with the latest embedding model capabilities as soon as they are released. To get started, launch an AlloyDB trial cluster https://console.cloud.google.com/alloydb/create-trial-cluster and consult the official AlloyDB AI documentation https://docs.cloud.google.com/alloydb/docs/ai/what-is-alloydb-ai .