Production RAG on the Lakehouse with BigQuery Vector Search and Apache Iceberg

A developer argues that conventional RAG architectures create a dangerous chasm between the data lakehouse and the AI stack, forcing teams to duplicate data and reimplement security in a separate vector database. The post details how this separation causes metadata drift, where stale vectors can lead a RAG chatbot to quote outdated prices or recommend recalled products, and security fragmentation, where access revocations in the lakehouse may not propagate to the vector store. The proposed fix is running production RAG directly on the lakehouse using BigQuery Vector Search and Apache Iceberg so vectors stay governed by the source of truth.

As teams rush to build with Generative AI, they're creating a dangerous chasm between their data and AI stacks. This common architectural flaw introduces massive technical debt and business risk. The explosion of Generative AI and Building a RAG Context Manager with Apps Script and Gemini Pro https://votuduc.com/building-a-rag-context-manager-with-apps-script-and-gemini-pro-p-20260505625174 RAG has unlocked incredible potential, but it has also exposed a fundamental architectural flaw in how many organizations are building these systems. As teams rush to production, they often inadvertently create a deep chasm between their core data infrastructure and their new AI stack. This divide isn't just an inconvenience; it's a source of significant technical debt, operational complexity, and business risk. At the heart of the issue is the separation of concerns gone awry. The data lives in one universe—the data lakehouse, governed by decades of best practices in security, governance, and reliability. The AI, particularly the vector search component, lives in another—a specialized, often external, database. Bridging this gap requires brittle pipelines, data duplication, and fragmented security models, ultimately undermining the very reliability and trustworthiness we seek to build into our AI applications. The conventional approach to building a RAG system follows a familiar, yet problematic, pattern. You begin with your curated, high-quality data residing in a centralized platform like a data lakehouse. This is your source of truth. To make this data accessible to a Large Language Model LLM , you must: This new silo is completely disconnected from the original data's lifecycle. It has its own infrastructure to manage, its own APIs to learn, and its own failure modes to handle. More importantly, it requires a complex and often fragile synchronization process to keep it from becoming stale. Every time data is updated, deleted, or added in the source system, a corresponding change must be perfectly orchestrated and propagated to the vector database. This adds immense operational overhead and introduces a new, critical point of failure in your AI stack. The consequences of this data silo extend far beyond mere operational complexity. Two critical challenges emerge that directly impact the quality and security of your AI application: metadata drift and security fragmentation. Metadata Drift is the silent killer of RAG system accuracy. It occurs when the data in your source-of-truth lakehouse changes, but those changes aren't immediately and atomically reflected in the vector database. Consider a product catalog table in your lakehouse. A product's price is updated, or its status changes to "recalled." If your synchronization pipeline fails or runs on a delay, your RAG-powered chatbot could retrieve the old, stale context from the vector database and confidently provide a customer with an incorrect price or, worse, recommend a recalled product. This isn't just a technical glitch; it's a direct erosion of user trust and a potential business liability. The vector index has "drifted" from the ground truth, and your RAG system is now hallucinating based on outdated facts. Security Fragmentation presents an equally severe governance and compliance risk. Your enterprise data lakehouse is built upon a robust, unified security model. You have fine-grained controls—IAM roles, row-level access policies, and column-level security—that dictate precisely who can see what data. When you copy that data into a separate vector database, you are forced to reimplement that entire security model from scratch in a new environment. This is not only a duplication of effort but also a massive security risk. It's incredibly difficult to keep two disparate security models perfectly in sync. An employee who leaves the company might have their access revoked in the lakehouse, but their access to the sensitive data copied in the vector store might persist. This fragmentation creates security gaps, doubles the administrative burden, and makes compliance audits a nightmare. What if we could eliminate the divide? What if, instead of moving the data to a separate AI system, we brought the AI capabilities directly to the data? This is the foundational principle of building production-grade AI on the Lakehouse. The modern data lakehouse, combining the scalability of a data lake with the performance and transactional integrity of a data warehouse, is already the established single source of truth for enterprise analytics. It houses your most valuable, curated, and governed data assets. By integrating vector search as a native feature within this platform—as BigQuery has done—we can fundamentally change the architectural paradigm. In this model, vector embeddings are not shipped to an external system; they become just another data type, a new column ARRAY<FLOAT64 in your existing Apache Iceberg or BigQuery native tables. The vector index is built directly on top of this column, co-located with the source data it represents. This elegant simplification solves our critical challenges: To build a robust, production-grade RAG system on the lakehouse, we need more than just a collection of services; we need a cohesive architecture where each component plays a specific, complementary role. Our blueprint unifies data management, machine learning, and analytics within a single, governable ecosystem on Google Cloud. This approach moves beyond siloed vector databases, bringing AI capabilities directly to your data's center of gravity—the data lakehouse. The result is a streamlined, scalable, and cost-effective pipeline that transforms raw information into intelligent, contextual responses. The power of this architecture lies in the synergy between four key Google Cloud and open-source technologies. Let's break down the role of each player. text-embedding-004 . These managed, scalable endpoints take our processed text chunks as input and convert them into high-dimensional numerical vectors embeddings . This process is the heart of the "retrieval" mechanism, as it encodes the semantic meaning of our text into a format that machines can compare for similarity. The process of converting raw documents into a searchable vector index follows a clear, automated data pipeline. This is the "indexing" half of the RAG workflow. annual-report-2023.pdf are uploaded to a designated GCS bucket. VECTOR INDEX . BigQuery automatically handles the complex process of building the ANN index in the background. This index is what enables lightning-fast similarity searches across potentially billions of vectors during the retrieval step. The most transformative aspect of this architecture is how BigQuery Vector Search acts as the unifying force, collapsing what were once disparate systems into a single, cohesive plane. Traditionally, a RAG pipeline required managing at least three separate systems: an object store for raw files GCS , a dedicated vector database for ANN search e.g., Pinecone, Weaviate , and a data warehouse for structured metadata and analytics BigQuery . This separation introduces complexity in data movement ETL , security, governance, and operational overhead. Our blueprint eliminates this fragmentation. The foundation of any high-performing RAG system isn't the LLM—it's the data. The quality, structure, and semantic representation of your knowledge base directly dictate the relevance and accuracy of the generated responses. In a Lakehouse architecture, this first step is about establishing a robust, scalable, and open foundation for your data and then transforming it into a format that machine learning models can understand: high-dimensional vectors. We'll tackle this by first defining our data's home using Apache Iceberg tables in BigQuery, and then processing our raw documents into vectorized "chunks" using Vertex AI's powerful embedding models. Before we can ingest anything, we need a destination. Why Apache Iceberg? In the context of a Lakehouse, Iceberg provides critical features that traditional data warehousing tables lack. It's an open table format that decouples the table structure from the physical storage in our case, Google Cloud Storage , offering schema evolution, time travel, and efficient file-level operations. This makes it perfect for managing large, evolving datasets of document chunks and their corresponding embeddings. We'll create a BigQuery "BigLake" table backed by Iceberg. This table will serve as our "vector store" source of truth, holding the original text chunks, their vector embeddings, and any relevant metadata. Here’s the DDL to create our core table, doc embeddings iceberg : CREATE OR REPLACE TABLE your dataset.doc embeddings iceberg chunk id STRING NOT NULL OPTIONS description="Unique identifier for the text chunk" , doc source STRING OPTIONS description="Identifier for the original source document, e.g., GCS path" , chunk text STRING OPTIONS description="The actual text content of the chunk" , embedding ARRAY<FLOAT64 OPTIONS description="The 768-dimension vector embedding from Vertex AI" , created at TIMESTAMP OPTIONS format = 'ICEBERG', table version = 2, uris = 'gs://your-gcs-bucket/iceberg-warehouse/doc embeddings' , connector = 'biglake-connector-v1' -- Ensure your BigLake connection is set up ; Let's break down the key components of this schema: chunk id doc source chunk text embedding ARRAY<FLOAT64 column will hold the numerical vector generated by our embedding model. OPTIONS ICEBERG and specify the GCS path where the underlying Parquet and metadata files will be stored. This is the core of the Lakehouse pattern—SQL on open files in your data lake. With our table ready, we need a way to convert text into meaningful vectors. An embedding is a dense vector representation of a piece of data in our case, text where semantically similar items are located closer together in the vector space. Google's Vertex AI offers state-of-the-art embedding models that are managed, scalable, and optimized for various tasks. For our RAG use case, we'll use the text-embedding-004 model, which generates a 768-dimensional vector. Its task type parameter is specifically designed to optimize embeddings for retrieval, making it ideal for creating a searchable knowledge base. Here’s a JSON-to-Video Automated Rendering Engine https://votuduc.com/JSON-to-Video-Automated-Rendering-Engine-p618510 snippet demonstrating how to generate embeddings for a batch of text chunks using the Vertex AI SDK: python import vertexai from vertexai.language models import TextEmbeddingModel def generate embeddings project id: str, location: str, text chunks: list str - list list float : """Generates embeddings for a list of text chunks.""" vertexai.init project=project id, location=location We use the latest text embedding model, optimized for retrieval model = TextEmbeddingModel.from pretrained "text-embedding-004" The 'task type' is critical for optimizing vectors for RAG 'RETRIEVAL DOCUMENT' is used for the text being indexed. 'RETRIEVAL QUERY' would be used for the user's input query. embeddings = model.get embeddings text chunks, task type="RETRIEVAL DOCUMENT" Extract the numerical vector from the response object return embedding.values for embedding in embeddings --- Example Usage --- my project id = "gcp-project-id" my location = "us-central1" my chunks = "Apache Iceberg is an open table format for huge analytic datasets.", "BigQuery vector search enables efficient similarity search on embeddings.", "A Lakehouse architecture combines the benefits of data lakes and data warehouses." vector embeddings = generate embeddings my project id, my location, my chunks The output 'vector embeddings' is a list of lists, where each inner list is a 768-dimension vector. print f"Generated {len vector embeddings } embeddings." print f"Dimension of first embedding: {len vector embeddings 0 }" This function is the core of our text-to-vector transformation. It takes a list of strings and returns a corresponding list of 768-dimension floating-point vectors, ready to be inserted into our Iceberg table. Now we connect the pieces. The final step is to create a scalable batch pipeline that reads raw documents, processes them into chunks, generates embeddings, and loads the results into our BigQuery Iceberg table. 1. Data Sourcing and Chunking Your enterprise data likely lives in various formats PDFs, DOCX, HTML and locations GCS, Confluence, etc. . The first task is to extract the raw text. Once you have the text, you must break it down into smaller, semantically meaningful chunks. This is perhaps the most important tuning parameter in a RAG system. \n\n , then sentences . , then spaces This is a conceptual pipeline structure, not a complete, runnable script. You would use libraries like 'google-cloud-bigquery' and 'pypdf' from google.cloud import bigquery import uuid def process and ingest documents documents to process: list str : """ Conceptual pipeline to chunk, embed, and ingest documents. """ all rows to insert = for doc path in documents to process: Step 1: Extract text from the source document e.g., a PDF in GCS raw text = extract text from pdf doc path Your custom text extraction logic Step 2: Chunk the text using a chosen strategy text chunks = chunk text recursively raw text, chunk size=512, chunk overlap=50 Step 3: Generate embeddings for the chunks in batches API has a limit on items per call chunk embeddings = generate embeddings project id="gcp-project-id", location="us-central1", text chunks=text chunks Step 4: Structure the data for insertion for i, chunk in enumerate text chunks : row = { "chunk id": str uuid.uuid4 , "doc source": doc path, "chunk text": chunk, "embedding": chunk embeddings i , "created at": "CURRENT TIMESTAMP " Let BigQuery handle this } all rows to insert.append row Step 5: Batch load the data into the BigQuery Iceberg table The BigQuery Python client can handle streaming inserts or batch loads from GCS if all rows to insert: client = bigquery.Client table id = "your dataset.doc embeddings iceberg" For large volumes, loading from a file JSON, Parquet in GCS is more robust For simplicity, this example uses streaming inserts errors = client.insert rows json table id, all rows to insert if not errors: print f"Successfully inserted {len all rows to insert } rows." else: print f"Encountered errors while inserting rows: {errors}" --- Example Invocation --- In a real pipeline, this list would come from scanning a GCS bucket source docs = "gs://my-knowledge-base/doc1.pdf", "gs://my-knowledge-base/doc2.pdf" process and ingest documents source docs By executing this pipeline, you systematically convert your unstructured enterprise documents into a structured, vectorized dataset within your Lakehouse. This Iceberg table is now the single source of truth for your knowledge base, ready to be indexed for lightning-fast similarity search in the next step. With our embeddings now residing in an Apache Iceberg table managed by BigQuery, we can unlock the power of high-performance retrieval without any data movement. This is where the tight integration between the Lakehouse storage format and BigQuery's analytical engine truly shines. We will create a vector index directly on the Iceberg table, enabling low-latency similarity searches that are essential for a responsive RAG application. A vector index is a specialized data structure that reorganizes your high-dimensional embedding data to enable Approximate Nearest Neighbor ANN search. Instead of exhaustively comparing a query vector to every single vector in your table a brute-force approach , the index allows the system to quickly narrow down the search to a small, promising subset of candidates. This is the key to achieving millisecond-level latency on datasets with millions or even billions of vectors. In BigQuery, creating a vector index is a straightforward DDL operation. Let's assume our Iceberg table is named rag documents and has the following simplified schema: doc id STRING, PRIMARY KEY CREATE VECTOR INDEX statement. CREATE VECTOR INDEX my doc index ON my dataset.rag documents embedding OPTIONS index type = 'IVF', distance type = 'COSINE', ivf options = '{"num lists": 500}' ; Let's break down the OPTIONS : index type = 'IVF' distance type = 'COSINE' COSINE similarity is almost always the correct choice. It measures the angle between two vectors, making it robust to differences in vector magnitude. Other options include EUCLIDEAN L2 distance and DOT PRODUCT . ivf options = '{"num lists": 500}' num lists , which sets the number of partitions to create. probe count which we'll cover next to maintain high recall. Index creation is an asynchronous background job. You can monitor its progress by querying the information schema: SELECT index name, table name, coverage percentage, last refresh time FROM my dataset.INFORMATION SCHEMA.VECTOR INDEXES WHERE table name = 'rag documents'; A coverage percentage of 100 indicates that the index is fully built and ready for use. BigQuery automatically keeps the index updated as new data is inserted into your Iceberg table. Once the index is active, you can perform searches using the VECTOR SEARCH function. This function is the core of the retrieval step in your RAG pipeline. It takes a query vector and efficiently finds the top k most similar vectors from your indexed table. The basic syntax is: VECTOR SEARCH TABLE table name, column to search, query vector, top k = k, options = '...' Here is a practical example. Imagine your application has generated an embedding for the user's question, "What are the latest query optimization techniques?". You would use that embedding to find the most relevant document chunks. -- Assume @query embedding is a query parameter passed from your application -- For this example, we'll use a placeholder array. DECLARE query embedding ARRAY<FLOAT64 ; SET query embedding = 0.1, 0.2, 0.3, ... ; -- Your 768 or 1536-dimension query vector SELECT base.doc id, base.chunk text, search results.distance FROM VECTOR SEARCH TABLE my dataset.rag documents, -- The table with the index 'embedding', -- The indexed column query embedding, -- The vector to search for top k = 10, -- Number of results to return options = '{"probe count": 20}' AS search results -- Join back to the base table to retrieve the actual text content JOIN my dataset.rag documents AS base ON search results.doc id = base.doc id ORDER BY search results.distance; -- COSINE distance is 0 for identical, 2 for opposite Key Points: options Parameter sqrt num lists . JOIN Pattern VECTOR SEARCH returns the primary key columns of your table distance for each match. You must For a production RAG system, performance and accuracy are paramount. Simply running a basic vector search is often not enough. You need to consider filtering and tuning to ensure your application is both fast and relevant. A common requirement in RAG is to search only within a subset of documents. For example, you might want to find information relevant only to a specific user, product, or date range. There are two ways to apply these filters: WHERE clause to the final result set. This is highly inefficient because the vector search wastes resources finding top matches that are immediately discarded by the filter. source year column to our table and compare the two approaches. -- AVOID THIS PATTERN SELECT base.doc id, base.chunk text FROM VECTOR SEARCH TABLE my dataset.rag documents, 'embedding', @query embedding, top k = 10 AS s JOIN my dataset.rag documents AS base ON s.doc id = base.doc id WHERE base.source year 2022; -- Filter is applied AFTER the expensive search Efficient Pre-filtering: -- USE THIS PATTERN SELECT base.doc id, base.chunk text FROM VECTOR SEARCH -- The filter is applied to a subquery on the base table TABLE SELECT FROM my dataset.rag documents WHERE source year 2022 , 'embedding', @query embedding, top k = 10 AS s JOIN my dataset.rag documents AS base ON s.doc id = base.doc id; By filtering the table before it's passed to VECTOR SEARCH , you ensure the ANN search is performed only on the relevant slice of your data, leading to significant performance gains in production workloads. This is a critical optimization for building scalable, multi-tenant RAG applications on the Lakehouse. With a robust retrieval mechanism in place, the next critical step is to use the retrieved information to generate a coherent, accurate, and contextually relevant answer. This is the "Generation" part of Retrieval-Augmented Generation RAG . It involves skillfully weaving the search results from BigQuery into a prompt that instructs a Large Language Model LLM on how to synthesize a final response. This process transforms raw, retrieved data into a conversational and helpful answer, ensuring the model's output is grounded in the facts contained within our Iceberg table. The core of grounding lies in Prompt Engineering for Reliable Autonomous Workspace Agents https://votuduc.com/prompt-engineering-for-reliable-autonomous-workspace-agents-p-20260504436320 for Reliable Autonomous Workspace Agents https://votuduc.com/prompt-engineering-for-reliable-autonomous-workspace-agents-p-20260319404106 . We are not simply asking the LLM a question; we are providing it with a specific set of instructions and the exact context it must use to formulate its answer. A well-structured RAG prompt is the key to minimizing hallucinations and ensuring factual consistency. A typical RAG prompt consists of three main components: {context} and {question} which our application logic will replace with the actual data at runtime. You are an expert Q&A system that is a world-class expert on internal company documentation. Your instructions are: 1. Answer the user's QUESTION based ONLY on the provided CONTEXT. 2. Do not use any prior knowledge or information outside of the CONTEXT. 3. If the CONTEXT does not contain the answer, you MUST state that you cannot answer the question with the information provided. 4. Synthesize the information from the CONTEXT into a clear and concise answer. Do not simply copy and paste sections. 5. If the CONTEXT includes source URIs, cite the relevant sources in your answer. --- CONTEXT: & 123;context& 125; --- QUESTION: & 123;question& 125; Final Answer: The orchestration logic is the glue that connects our BigQuery vector index to the LLM. This logic, typically implemented in an application backend e.g., a Python service running on Cloud Run or a Cloud Function , executes a precise sequence of operations for every incoming user query. The end-to-end flow is as follows: textembedding-gecko@003 that was used to embed the documents in our Iceberg table. This generates a query vector. base document the original text chunk and any other relevant metadata for the top-k most similar documents. \n---\n . This string will replace the gemini-1.0-pro . Let's translate the logic above into a practical Python implementation. This example uses the google-cloud-bigquery and vertexai client libraries to orchestrate the entire pipeline. This function encapsulates the full RAG process: embedding the query, searching BigQuery, and generating the final answer with Gemini. python import vertexai from vertexai.language models import TextEmbeddingModel, TextGenerationModel from google.cloud import bigquery --- Configuration --- PROJECT ID = "your-gcp-project-id" LOCATION = "US" BQ DATASET = "rag dataset" BQ TABLE = "iceberg docs embedded" EMBEDDING MODEL NAME = "textembedding-gecko@003" GENERATION MODEL NAME = "gemini-1.0-pro" Or your preferred Gemini model --- Initialize clients --- vertexai.init project=PROJECT ID, location=LOCATION bq client = bigquery.Client project=PROJECT ID embedding model = TextEmbeddingModel.from pretrained EMBEDDING MODEL NAME It's best practice to initialize the model once generation model = TextGenerationModel.from pretrained GENERATION MODEL NAME PROMPT TEMPLATE = """ You are an expert Q&A system that is a world-class expert on internal company documentation. Your instructions are: 1. Answer the user's QUESTION based ONLY on the provided CONTEXT. 2. Do not use any prior knowledge or information outside of the CONTEXT. 3. If the CONTEXT does not contain the answer, you MUST state that you cannot answer the question with the information provided. 4. Synthesize the information from the CONTEXT into a clear and concise answer. --- CONTEXT: {context} --- QUESTION: {question} Final Answer: """ def get rag response question: str, top k: int = 5 - str: """ Orchestrates the RAG pipeline: 1. Embeds the user question. 2. Searches BigQuery for relevant documents. 3. Generates a response using an LLM. """ 1. Embed the user's question question embedding = embedding model.get embeddings question 0 .values 2. Execute VECTOR SEARCH in BigQuery sql query = f""" SELECT base document, distance FROM VECTOR SEARCH TABLE {PROJECT ID}.{BQ DATASET}.{BQ TABLE} , 'embedding', SELECT {question embedding} AS embedding , top k = {top k}, distance type = 'COSINE' """ query job = bq client.query sql query results = query job.result 3. Format the retrieved context context chunks = row.base document for row in results if not context chunks: return "I could not find any relevant information to answer your question." context string = "\n\n---\n\n".join context chunks 4. Construct the final prompt final prompt = PROMPT TEMPLATE.format context=context string, question=question 5. Invoke the LLM to generate the final answer response = generation model.predict prompt=final prompt, temperature=0.2, max output tokens=1024, top k=40, top p=0.95, return response.text --- Example Usage --- if name == ' main ': user question = "What are the key performance metrics for the Q3 marketing campaign?" answer = get rag response user question print "--- Question ---" print user question print "\n--- Answer ---" print answer Moving a Retrieval-Augmented Generation RAG system from a proof-of-concept to a production environment introduces a host of non-functional requirements that are critical for success. It's no longer just about getting the right answer; it's about delivering that answer securely, performantly, and in a way that aligns with your organization's governance and cost management principles. Building your RAG system on a lakehouse architecture with BigQuery and Iceberg provides a powerful foundation to address these challenges head-on, allowing you to leverage existing enterprise-grade features rather than building new solutions from scratch. One of the most significant advantages of this architecture is the ability to extend your existing data security and governance framework to your AI workloads. Your vector embeddings and source documents are not siloed in a separate, specialized database; they are first-class citizens within your BigQuery lakehouse, inheriting its robust security posture. Unified Access Control with IAM: Access to both the source Apache Iceberg tables and the BigQuery vector indexes is managed through Google Cloud's Identity and Access Management IAM . This means you can use the same roles and permissions you've already defined for your analytical workloads. There's no need to manage a separate set of credentials or access policies for your RAG application's data layer. A service account for your RAG application can be granted a fine-grained role, like roles/bigquery.dataViewer , on only the specific datasets it needs to access. Fine-Grained Data Segmentation: For sensitive data, you can enforce granular control using BigQuery's built-in security features: department = 'HR' . -- Example of a Row-Level Access Policy -- This policy ensures that users can only query vectors -- related to their own department. CREATE ROW ACCESS POLICY department filter ON my project.my dataset.document embeddings GRANT TO "group:sales-team@example.com" FILTER USING department = 'Sales' ; Auditing and Lineage: Every query, including vector searches, is logged in Cloud Audit Logs. This provides an immutable record of what data was accessed, by whom, and when. This is invaluable for compliance, security audits, and debugging. You can trace a specific generated response back to the exact VECTOR SEARCH query that was run, providing full data lineage for your AI application's knowledge retrieval step. Network Security with VPC Service Controls: For organizations with stringent data exfiltration requirements, you can place your BigQuery datasets and the underlying Cloud Storage buckets for your Iceberg tables within a VPC Service Controls perimeter. This creates a virtual network boundary, ensuring that your sensitive data and embeddings can only be accessed by authorized services and networks, effectively preventing data from leaving your trusted environment. Performance in a RAG system is a multi-faceted concern, primarily revolving around the trade-off between search quality recall and speed latency . A systematic benchmarking approach is essential to find the right balance for your application's Service Level Objectives SLOs . Indexing Performance: The creation of a vector index in BigQuery is an asynchronous, back-end process. The time it takes is influenced by the number of vectors, their dimensionality, and the index configuration. While you don't need to manage the underlying compute, you should monitor the build process. You can track the progress of index creation using the INFORMATION SCHEMA : SELECT table name, index name, coverage percentage, last refresh time FROM my project.my dataset.INFORMATION SCHEMA.VECTOR INDEXES WHERE table name = 'document embeddings'; An index is queryable before it reaches 100% coverage, but performance and recall will improve as it approaches full coverage. For production systems, your data ingestion pipeline should have a step to verify that the coverage percentage is 100 before routing live traffic to a newly refreshed index. Query Performance: The Latency vs. Recall Trade-off: Approximate Nearest Neighbor ANN search, which powers VECTOR SEARCH , is designed to be fast by trading perfect accuracy for speed. num lists to search option within ivf options . A higher value instructs the query engine to scan more of the index's "inverted file" lists, increasing the probability of finding the true nearest neighbors higher recall at the cost of increased processing and higher latency. number of true neighbors found / K . Generative AI workloads can become expensive if not managed carefully. A proactive approach to cost optimization is crucial for building a sustainable, production-grade RAG system. 1. Embedding Costs: The initial and ongoing cost of generating embeddings via an external model API is often the largest component. 2. Indexing and Storage Costs: INFORMATION SCHEMA.JOBS view to monitor the bytes processed by your vector search queries. Set up Cloud Billing budgets and alerts to get notified if costs exceed your forecasts, allowing you to take corrective action before you get a surprise at the end of the month. We've journeyed through a paradigm shift in building AI applications—moving from complex, fragmented architectures to a streamlined, powerful model centered on the data lakehouse. By integrating vector search capabilities directly into BigQuery and leveraging the open standard of Apache Iceberg, we've demonstrated that production-grade RAG is not just feasible but fundamentally more efficient and secure when AI is brought to the data. This approach dissolves the traditional boundaries between analytical and AI workloads, paving the way for a new generation of data-driven, intelligent applications built directly on your source of truth. The advantages of this unified architecture are immediate and substantial, primarily revolving around simplification and fortification. What we've built here is not an endpoint but a glimpse into the future of data platforms. The trend is clear: databases are evolving from passive repositories into active, intelligent engines capable of handling diverse workloads, including AI. We can anticipate several exciting developments on this front: You now have the architectural blueprint for building a scalable, secure, and efficient RAG system on the lakehouse. The next step is to put it into practice.