Building Production-Grade Semantic Search with GPT-5 and Microsoft Foundry, From Scratch

A developer built a production-grade semantic search pipeline using GPT-5 and Microsoft Foundry's Foundry IQ, an agentic retrieval layer on Azure AI Search. The system treats retrieval as a reasoning task, planning sub-queries, executing them in parallel, and synthesizing cited answers. The knowledge base exposes an MCP endpoint for integration with Foundry Agent Service or other MCP-compatible clients.

Most "RAG tutorials" stop at a single embedding query against a single index. That works for a demo and falls over the moment a real user asks something like "compare our Q3 and Q4 vendor contracts and flag anything that changed" — a question that needs multiple sub-queries, reasoning about what's still missing, and synthesis across documents. Microsoft Foundry's answer to this is Foundry IQ : an agentic retrieval layer built on Azure AI Search that treats retrieval as a reasoning task rather than a single keyword or vector lookup. This is a from-scratch build of a semantic search pipeline using GPT-5 for query planning/synthesis and Foundry IQ for retrieval. Instead of one query hitting one index once, Foundry IQ's knowledge base plans sub-queries, executes them in parallel against one or more knowledge sources, evaluates whether it has enough signal, and iterates before synthesizing a final, cited answer. The knowledge base sits between your agent and the underlying content. Your Foundry agent doesn't talk to Azure AI Search directly — it calls the knowledge base's MCP endpoint, which handles planning, retrieval, and synthesis behind a single tool call. A knowledge source is a reusable reference to your underlying content — in this example, a Blob Storage container of documents. Creating it also triggers Azure AI Search to generate the index, skillset, and indexer needed to chunk and vectorize the content automatically. python from azure.search.documents.indexes import SearchIndexClient from azure.search.documents.indexes.models import SearchIndexKnowledgeSource, SearchIndexKnowledgeSourceParameters, from azure.identity import DefaultAzureCredential index client = SearchIndexClient endpoint="https://<your-search-service .search.windows.net", credential=DefaultAzureCredential , knowledge source = SearchIndexKnowledgeSource name="vendor-contracts-ks", search index parameters=SearchIndexKnowledgeSourceParameters search index name="vendor-contracts-index", , index client.create or update knowledge source knowledge source=knowledge source The knowledge base ties one or more knowledge sources together with an LLM deployment that handles query planning and answer synthesis. from azure.search.documents.indexes.models import KnowledgeBase, KnowledgeBaseAzureOpenAIModel, AzureOpenAIVectorizerParameters, knowledge base = KnowledgeBase name="vendor-contracts-kb", knowledge sources= {"name": "vendor-contracts-ks"} , models= KnowledgeBaseAzureOpenAIModel azure open ai parameters=AzureOpenAIVectorizerParameters resource url="https://<your-foundry-resource .openai.azure.com", deployment name="gpt-5-mini", model name="gpt-5-mini", , output configuration={ "modality": "answerSynthesis", verbatim extractive data is the alternative }, index client.create or update knowledge base knowledge base=knowledge base output configuration is the key lever here: answerSynthesis returns a pre-generated, cited answer, while extractive mode returns verbatim source chunks and leaves reasoning entirely to your agent's own model. Extractive mode costs less and gives your agent more control; synthesis mode does more work up front at the retrieval layer. Each knowledge base exposes a standalone MCP endpoint. Any MCP-compatible client — including Foundry Agent Service, but also GitHub Copilot or other MCP clients — can call its knowledge base retrieve tool directly. python from azure.ai.projects import AIProjectClient from azure.identity import DefaultAzureCredential project = AIProjectClient endpoint="https://<your-foundry-project-endpoint ", credential=DefaultAzureCredential , agent = project.agents.create agent model="gpt-5-mini", name="contract-search-agent", instructions="Answer questions about vendor contracts using the knowledge base tool. Always cite sources.", tools= { "type": "mcp", "server url": "https://<your-search-service .search.windows.net/knowledgebases/vendor-contracts-kb/mcp?api-version=2026-05-01-preview", "allowed tools": "knowledge base retrieve" , } , The knowledge base's retrieval reasoning effort setting controls how much LLM-driven planning happens before retrieval, and it's the main dial for balancing latency, cost, and answer quality. | Reasoning effort | What happens | Best for | |---|---|---| | Minimal | Bypasses LLM query planning entirely; direct hybrid search | Simple factual lookups, latency-sensitive paths | | Medium | LLM reformulates and may decompose the query into sub-queries | Multi-part or ambiguous questions | | High | Full iterative planning, evaluates sufficiency, re-queries as needed | Complex, multi-hop questions across many documents | Lowering reasoning effort is also the primary way to control the number of GPT-5 tokens consumed per query — fewer planning passes and less iteration directly reduce both latency and inference cost, without needing to touch the underlying index. The reason this matters versus classic RAG: a query like "which vendors had payment terms that changed between the Q3 and Q4 renewals" can't be answered by a single embedding lookup. With medium or high reasoning effort, the knowledge base decomposes it into sub-queries find Q3 renewals, find Q4 renewals, compare payment terms fields , runs them in parallel against the knowledge source, and only synthesizes a final answer once it judges the retrieved context sufficient — re-querying automatically if it isn't.