{"slug": "unifying-structured-and-unstructured-data-insights-with-bq-search-innovations", "title": "Unifying Structured and Unstructured Data Insights with BQ Search Innovations", "summary": "Google Cloud announced the general availability of Autonomous Embedding Generation and AI.SEARCH in BigQuery, along with the public preview of Hybrid Search, to simplify unstructured data analysis. AI.SEARCH now delivers up to a 133x gain in slot efficiency for single-query execution, and Autonomous Embedding Generation supports embeddings over images using ObjectRefs, enabling multimodal search.", "body_md": "Modern enterprises possess a vast amount of unstructured data, yet they frequently encounter significant challenges in managing and extracting value from it. Historically, unlocking the insights hidden within PDFs, audio files, images, and unstructured text required a fragmented architecture: moving data out of your warehouse, stitching together complex LLM pipelines, and managing disparate search indexes.\n\nBigQuery has worked with many enterprises to make sense of their unstructured data sources. For example, consider an **advanced healthcare company** **managing thousands of clinical trial documents in PDF** form. BigQuery helps unlock insights from these documents through a simple, five-step lifecycle: **Access, Process, Ground, Relate, and Activate**.\n\nIn this post, we are highlighting three major milestones focused heavily on the **\"Ground\"** phase of this framework:\n\n**General Availability (GA) of Autonomous Embedding Generation**\n\n**General Availability (GA) of AI.SEARCH with massive single-query performance gains**\n\n**Public Preview of Hybrid Search**\n\nLet’s dive into how these features work together to simplify your AI architecture, using a real-world clinical trial research platform as an example.\n\nBuilding a retrieval-augmented generation (RAG) pipeline or search application usually requires managing complex, asynchronous embedding infrastructure. You have to handle retries, error logging, and pipeline orchestration every time a new record arrives.\n\nWith the General Availability of **Autonomous Embedding Generation**, BigQuery manages this entirely for you. By simply defining a column in your schema, BigQuery asynchronously and continuously generates embeddings as new data is ingested. You have the flexibility to choose external models (like Vertex AI text-embeddings) or natively utilize **Gemma embedding models** directly within BigQuery.\n\n**How it works in practice:**\n\nImagine you are building a research platform analyzing clinical trial PDFs stored in Google Cloud Storage. After extracting the study titles and disease areas into a table, you can automatically embed those titles:\n\nBigQuery eliminates the need for complex third-party vector databases by managing\n\nenterprise-scale processing with one configuration. This autonomous embedding generation keeps data synchronized automatically as source text changes, removing the need for manual machine learning pipelines. This integrated approach streamlines workflows for dynamic datasets and reduces the operational burden of maintaining custom data scripts.\n\nFinally, with this GA launch, Autonomous Embedding Generation now also supports generating embeddings natively over images using [ObjectRefs](https://cloud.google.com/bigquery/docs/object-tables), unlocking true multimodal search and analytics.\n\nTo truly enable conversational analytics agents and snappier user experiences, your underlying search infrastructure needs to be intuitive and performant.\n\nOnce your data is seamlessly embedded, you need an efficient way to query it. Today, we are announcing the **General Availability of **[ AI.SEARCH()](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-search). This function provides a streamlined, natural-language-focused search experience, allowing you to easily find semantically related records without generating embeddings in your search path. In pairing this with the Autonomous Embedding Generation, we leverage the same embedding model used in your dataset for easier use.\n\nFurthermore, as part of efficiency investments in the last year, we have heavily optimized AI.SEARCH for single-query execution. For online applications and single-query searches (those most common in agentic searches), we have observed **up to a 133x gain in slot efficiency..**\n\nThis means you can serve highly concurrent, user-facing natural language searches directly out of BigQuery faster and more cost-effectively than ever before.\n\nSemantic (vector) search is incredibly powerful; for example, the query above will successfully return conceptually related terms like \"chemotherapy.\" However, semantic search isn't always enough. What if a researcher is searching for a specific, highly technical immunotherapy drug designation like \"MK3475\"? Because this alphanumeric string lacks broad semantic meaning, pure vector search might struggle to rank it correctly.\n\nBy merging lexical search with existing semantic capabilities, BigQuery's [hybrid search](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/search_functions#hybrid-search) allows for data retrieval based on both keyword similarity and underlying meaning. This approach unites the conceptual depth of semantic vector search with the pinpoint accuracy of lexical matching, utilizing algorithms such as Reciprocal Rank Fusion and BM25. The result is a significant boost in search precision and a reduction in LLM hallucination costs through the reranking of results based on keyword frequency and semantic relevance. Users can implement this via the AI.SEARCH and VECTOR_SEARCH functions by employing the `hybrid mode`\n\nor `lexical_search_columns`\n\nparameters. Furthermore, performance can be optimized by extending vector indexes to include keyword data, which accelerates the lexical search process.\n\nYou can now perform hybrid searches effortlessly using the AI.SEARCH() function by simply setting the mode to HYBRID:\n\nTo speed up these hybrid queries at scale, you can easily [extend your CREATE VECTOR INDEX DDL](https://docs.cloud.google.com/bigquery/docs/vector-index#use_vector_indexes_with_hybrid_search) to include the keyword columns you want to use for the lexical portion of the search, natively combining your indexes.\n\nThe search and embedding features launching today are part of a much broader vision. We are building an end-to-end unstructured data analytics platform. One common type of unstructured data is documents, and BigQuery now provides the complete toolset to manage this workflow from end to end:\n\n**Access:** Enable zero-ETL workflows by querying unstructured PDFs and documents directly where they live in Google Cloud Storage using Object Tables.\n\n**Process:** Utilize embedded AI capabilities like AI.PARSE and AI.CHUNK_DOC (coming soon) for layout-aware chunking (perfect for RAG), [AI.GENERATE](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-generate) for entity extraction/summarization, and [AI.CLASSIFY](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-classify) to instantly categorize records using foundational models directly in your SQL pipelines.\n\n**Ground:** Build highly accurate context using Autonomous Embeddings and Hybrid Search. Hybrid search combines the conceptual understanding of semantic vector search with the exact precision of lexical keyword matching By reranking results based on both semantic relevance and keyword frequency, you drastically increase search precision and drive down LLM hallucinations.\n\n**Relate:** Uncover hidden multi-hop insights by mapping extracted entities (like Sponsors, Trials, and Drugs) into a [BigQuery Graph](https://docs.cloud.google.com/bigquery/docs/graph-overview)—no specialized graph database required.\n\n**Activate:** Bring it all together with [BigQuery's Conversational Analytics](https://docs.cloud.google.com/bigquery/docs/conversational-analytics) agents. Using functions like [AI.AGG](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-ai-agg), you can chat directly with your complex data, generate visualizations, and perform trend analysis at massive scale.\n\nWith unstructured data as a first-class citizen in BigQuery, you can finally bridge the gap between your raw documents and conversational AI.\n\n**Ready to get started?**\n\n**Explore the Code:** Check out the complete end-to-end clinical trials demonstration in our [Document Analytics on BigQuery GitHub Repository](https://github.com/GoogleCloudPlatform/document-analytics-on-bigquery).\n\n**Read the Docs:** Dive into the official documentation for [Autonomous Embeddings](https://docs.cloud.google.com/bigquery/docs/autonomous-embedding-generation) and [Hybrid Search](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/search_functions#hybrid-search) to start building your own unified data pipelines today.", "url": "https://wpnews.pro/news/unifying-structured-and-unstructured-data-insights-with-bq-search-innovations", "canonical_source": "https://cloud.google.com/blog/products/data-analytics/bigquery-search-innovations-unify-structured-unstructured-data/", "published_at": "2026-08-07 16:00:00+00:00", "updated_at": "2026-08-09 11:40:51.303492+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "generative-ai", "ai-infrastructure", "ai-tools"], "entities": ["Google Cloud", "BigQuery", "Vertex AI", "Gemma", "ObjectRefs"], "alternates": {"html": "https://wpnews.pro/news/unifying-structured-and-unstructured-data-insights-with-bq-search-innovations", "markdown": "https://wpnews.pro/news/unifying-structured-and-unstructured-data-insights-with-bq-search-innovations.md", "text": "https://wpnews.pro/news/unifying-structured-and-unstructured-data-insights-with-bq-search-innovations.txt", "jsonld": "https://wpnews.pro/news/unifying-structured-and-unstructured-data-insights-with-bq-search-innovations.jsonld"}}