Grounded Search Agents: Microsoft Foundry + Azure AI Search + Cosmos DB + Function Calling

Microsoft Foundry, Azure AI Search, and Azure Cosmos DB are combined to build a grounded search agent that uses function calling to route between unstructured document retrieval and structured data queries, always returning citations. The system outperforms plain RAG pipelines by integrating vector search with semantic ranking and maintaining conversation state in Cosmos DB.

A plain RAG pipeline — embed a query, hit a vector index, stuff the top-k chunks into a prompt — works fine for a single-turn FAQ bot. It falls apart the moment you need: This post builds a small but complete grounded search agent : Azure OpenAI via Microsoft Foundry decides when to call a search tool versus a structured data tool, Azure AI Search handles retrieval and semantic ranking over unstructured documents, and Azure Cosmos DB stores conversation state and structured order/customer metadata. The agent uses OpenAI function calling to route between the two data sources and always returns citations. We'll build this from the ground up in Python. The system has two phases: ingestion getting documents into a searchable index and runtime the agent loop that answers questions . At runtime, the agent sits between the user and two tools, using function calling to decide which one or both to invoke: Create the index with both a vector field for semantic similarity and a semantic configuration for ranking , since combining the two consistently outperforms either alone. python create index.py from azure.search.documents.indexes import SearchIndexClient from azure.search.documents.indexes.models import SearchIndex, SimpleField, SearchableField, SearchField, SearchFieldDataType, VectorSearch, HnswAlgorithmConfiguration, VectorSearchProfile, SemanticConfiguration, SemanticPrioritizedFields, SemanticField, SemanticSearch from azure.core.credentials import AzureKeyCredential endpoint = "https://<your-search-service .search.windows.net" credential = AzureKeyCredential "<admin-key " index client = SearchIndexClient endpoint, credential fields = SimpleField name="id", type=SearchFieldDataType.String, key=True , SearchableField name="content", type=SearchFieldDataType.String , SimpleField name="source url", type=SearchFieldDataType.String , SearchField name="content vector", type=SearchFieldDataType.Collection SearchFieldDataType.Single , searchable=True, vector search dimensions=1536, vector search profile name="default-profile", , vector search = VectorSearch profiles= VectorSearchProfile name="default-profile", algorithm configuration name="hnsw-config" , algorithms= HnswAlgorithmConfiguration name="hnsw-config" , semantic config = SemanticConfiguration name="default-semantic", prioritized fields=SemanticPrioritizedFields content fields= SemanticField field name="content" , index = SearchIndex name="support-docs-index", fields=fields, vector search=vector search, semantic search=SemanticSearch configurations= semantic config , index client.create or update index index print "Index created." Now embed and upload chunks. Chunking matters more than most tutorials admit — 512-token chunks with ~15% overlap is a reasonable default for policy/support documents. python ingest.py from openai import AzureOpenAI from azure.search.documents import SearchClient from azure.core.credentials import AzureKeyCredential import uuid aoai = AzureOpenAI azure endpoint="https://<your-foundry-resource .openai.azure.com", api key="<key ", api version="2024-10-21", search client = SearchClient endpoint=endpoint, index name="support-docs-index", credential=AzureKeyCredential "<admin-key " , def chunk text text: str, max tokens: int = 512, overlap: int = 80 : words = text.split step = max tokens - overlap return " ".join words i:i + max tokens for i in range 0, len words , step def embed text: str - list float : resp = aoai.embeddings.create model="text-embedding-3-small", input=text return resp.data 0 .embedding def ingest document text: str, source url: str : docs = for chunk in chunk text text : docs.append { "id": str uuid.uuid4 , "content": chunk, "source url": source url, "content vector": embed chunk , } search client.upload documents documents=docs with open "return policy.txt" as f: ingest document f.read , source url="policies/return policy.txt" Not every question is a document-retrieval problem. "Where's my order?" is a point lookup. Store operational data in Cosmos DB, partitioned by customer id for predictable query performance. python cosmos setup.py from azure.cosmos import CosmosClient, PartitionKey client = CosmosClient "<cosmos-endpoint ", credential="<cosmos-key " db = client.create database if not exists "retail" orders = db.create container if not exists id="orders", partition key=PartitionKey path="/customer id" , offer throughput=400, orders.upsert item { "id": "4471", "customer id": "cust 9001", "status": "delayed", "delay reason": "carrier weather disruption", "eta": "2026-08-02", } This is the core piece: Azure OpenAI decides, per turn, whether to call search docs , get order , both, or neither. We define the tools as JSON schemas and let the model pick. python agent.py import json from openai import AzureOpenAI from azure.search.documents import SearchClient from azure.cosmos import CosmosClient aoai = AzureOpenAI azure endpoint="https://<your-foundry-resource .openai.azure.com", api key="<key ", api version="2024-10-21", search client = SearchClient endpoint, "support-docs-index", AzureKeyCredential "<admin-key " cosmos = CosmosClient "<cosmos-endpoint ", credential="<cosmos-key " orders container = cosmos.get database client "retail" .get container client "orders" tools = { "type": "function", "function": { "name": "search docs", "description": "Semantic search over policy and support documents. Returns passages with citations.", "parameters": { "type": "object", "properties": {"query": {"type": "string"}}, "required": "query" , }, }, }, { "type": "function", "function": { "name": "get order", "description": "Look up a customer order by ID for status, ETA, and delay reason.", "parameters": { "type": "object", "properties": { "order id": {"type": "string"}, "customer id": {"type": "string"}, }, "required": "order id", "customer id" , }, }, }, def search docs query: str : results = search client.search search text=query, query type="semantic", semantic configuration name="default-semantic", top=3, return {"content": r "content" , "source": r "source url" } for r in results def get order order id: str, customer id: str : item = orders container.read item item=order id, partition key=customer id return {"status": item "status" , "reason": item.get "delay reason" , "eta": item.get "eta" } def run agent user message: str, customer id: str, history: list : messages = history + {"role": "user", "content": user message} response = aoai.chat.completions.create model="gpt-4o", messages=messages, tools=tools, tool choice="auto", msg = response.choices 0 .message if msg.tool calls: messages.append msg for call in msg.tool calls: args = json.loads call.function.arguments if call.function.name == "search docs": result = search docs args elif call.function.name == "get order": args "customer id" = customer id enforce, don't trust model-provided id result = get order args messages.append { "role": "tool", "tool call id": call.id, "content": json.dumps result , } final = aoai.chat.completions.create model="gpt-4o", messages=messages return final.choices 0 .message.content return msg.content A few things worth calling out: customer id is overwritten server-side before hitting Cosmos DB, even though the model could technically supply one — this prevents cross-tenant data leaks via prompt injection. tool choice="auto" chat.completions.create call after tool results are appended is what produces the grounded, cited final answer.Conversation history should not live in application memory — store it in Cosmos DB so any instance of the service can resume a session. php session store.py def load history session id: str - list: container = cosmos.get database client "retail" .get container client "sessions" try: item = container.read item item=session id, partition key=session id return item "messages" except Exception: return def save history session id: str, messages: list : container = cosmos.get database client "retail" .get container client "sessions" container.upsert item {"id": session id, "messages": messages} Choosing between search modes is a real architectural decision, not just a config toggle: | Retrieval Mode | Best for | Weakness | |---|---|---| | Keyword BM25 | Exact terms, SKUs, error codes | Misses paraphrased queries | | Pure vector search | Conceptual / paraphrased questions | Can surface plausible-but-wrong matches | | Hybrid keyword + vector | General-purpose document Q&A | Slightly higher latency than either alone | | Semantic ranking on top of hybrid | Conversational, nuanced queries | Adds a reranking pass; not needed for simple lookups | | Structured query Cosmos DB point read | Known-ID lookups orders, accounts | Not applicable to unstructured content | For most support/document agents, hybrid retrieval with semantic ranking is the right default, reserving structured lookups for anything with a stable ID.