The AI Agent Tech Stack Explained

Gartner predicts 40% of enterprise apps will feature AI agents by 2026, up from under 5% in 2025. A seven-layer tech stack—from foundation models like GPT-5.5 and Claude Sonnet 4.6 to deployment infrastructure—enables agents to perform complex tasks such as researching competitors and generating reports. Understanding each layer is critical for engineers building production systems.

In this article, you will learn how the seven layers of a production AI agent stack fit together, from the foundation model down to deployment infrastructure. Topics we will cover include: - What each layer of the stack does, from the foundation model and orchestration framework through memory, retrieval, tools, observability, and deployment. - How to implement each layer with working code, including a stateful agent, a memory system, a RAG pipeline, custom tools, and tracing. - Which combination of technologies to use at each layer depending on whether you are prototyping, scaling a startup, or operating in an enterprise environment. Introduction Picture this: you ask an AI agent to research three competitors, pull the pricing data from each of their websites, summarize the findings into a structured report, and drop it in a Slack channel by 9am. You hit enter. Thirty seconds later, the report is there. What just happened under the hood is not magic, and it is not one thing. It is seven distinct layers of technology working in sequence, each one handling a specific job, each one capable of breaking in its own specific way. The model at the top gets all the attention. The six layers beneath it are what determine whether the agent actually works. According to Gartner https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025 , 40% of enterprise applications will be integrated with task-specific AI agents by the end of 2026, up from less than 5% in 2025. That is not a gradual curve. That is a near-vertical adoption line, and the engineers and technical leads responsible for those deployments need to understand the full stack, not just the layer they happen to own. This article goes through each layer in order, from the foundation model down to deployment infrastructure. By the end, you will know what every piece is, why it exists, how the layers connect to each other, and what to actually use at each level. Layer 1: The Foundation Model The foundation model is the cognitive core of an agent. It is where reasoning happens, language is understood, and decisions about what to do next are made. Everything else in the stack is either feeding context into it or acting on what it produces. In practical terms, your main options in 2026 are OpenAI’s GPT-5.5 https://openai.com/index/introducing-gpt-5-5/ , Anthropic’s Claude Sonnet 4.6 https://www.anthropic.com/news/claude-sonnet-4-6 or Claude Opus 4.8 https://www.anthropic.com/news/claude-opus-4-8 for harder reasoning , Google’s Gemini 3.1 Pro https://deepmind.google/models/gemini/pro/ , and open-weight models like Meta’s Llama 4 https://ai.meta.com/blog/llama-4-multimodal-intelligence/ and Mistral Large 3 https://huggingface.co/collections/mistralai/mistral-large-3 . Each has trade-offs worth understanding before you commit. GPT-5.5 is fast for everyday calls and reliable at tool-calling, and it has the most mature ecosystem of integrations and the widest community of developers who have already run into and solved the edge cases you will encounter. Claude Sonnet 4.6 handles long documents and nuanced instruction-following well at a lower price point than Anthropic’s Opus tier, which matters in document-heavy workflows; reach for Claude Opus 4.8 when a task needs deeper, longer-horizon reasoning. Gemini 3.1 Pro has a 1 million token context window, which is relevant if your agent needs to process large codebases or lengthy knowledge bases in a single pass. Open-weight models like Llama 4 give you full control over deployment and data residency, at the cost of the infrastructure overhead of running them yourself. There is no longer a hard split between “ standard ” and “ reasoning ” model families, the way there was in 2025; OpenAI https://openai.com/ , Anthropic https://www.anthropic.com/ , and Google https://www.google.com/ have each folded reasoning into a single model that decides how long to think. GPT-5.5 ships with adjustable reasoning effort levels from none up to xhigh , and the same applies to Claude’s effort parameter and Gemini’s thinking levels. For most agent workflows, the default or low-effort setting is the right choice: fast and cheap. For tasks that require careful planning or mathematical reasoning, dialling the effort level up earns back its cost in correctness. Layer 2: The Orchestration Framework If the foundation model is the brain, the orchestration framework is the nervous system. It handles the control flow: deciding what the agent should do next, when it should call a tool, how it should handle the result, and how the whole reasoning loop stays coherent across multiple steps. The pattern that most frameworks implement is called ReAct Reasoning and Acting . The agent produces a thought, decides on an action, executes the action through a tool, observes the result, and then thinks again. This loop repeats until the agent produces a final answer. It sounds simple. In practice, it is where most production failures occur: the agent calls the wrong tool, gets stuck in a loop, or fails to recognise when it has enough information to stop. LangChain https://www.langchain.com/ is the most widely adopted framework. It offers a large ecosystem of integrations and good documentation. The criticism that it adds too much abstraction is fair at the prototype stage, but less relevant once you need the features that abstraction provides. LangGraph https://langchain-ai.github.io/langgraph/ , built by the same team, is better suited for stateful multi-agent workflows where you need fine-grained control over the execution graph. If your agent involves multiple specialists coordinating on a task, LangGraph is the cleaner choice. CrewAI https://www.crewai.com/ is designed specifically for multi-agent coordination. It lets you define agents with roles, assign them tasks, and have them collaborate within a structured workflow. It is higher-level than LangGraph and faster to get running, but gives you less control over the execution details. AutoGen https://microsoft.github.io/autogen/ , from Microsoft, takes a conversational approach to multi-agent systems. Agents interact with each other through a message-passing interface, which makes the interaction logic very readable. Semantic Kernel https://learn.microsoft.com/en-us/semantic-kernel/overview/ is Microsoft’s enterprise-focused option, with production-ready support for C , Python, and Java. If you are operating in an enterprise environment already running on the Microsoft stack, it fits naturally. LlamaIndex https://www.llamaindex.ai/ started as a document ingestion and retrieval framework and has since grown into a full agent framework, with particularly strong support for RAG-heavy workflows. The right choice depends on what your agent needs to do. For a single-agent task runner: LangGraph or LangChain. For a coordinated team of specialized agents: CrewAI https://crewai.com/ or AutoGen https://microsoft.github.io/autogen/stable//index.html . For enterprise environments: Semantic Kernel. For document-heavy retrieval workflows: LlamaIndex https://www.llamaindex.ai/ . Here is a minimal working agent in LangGraph that handles tool use and maintains state. Prerequisites : pip install langgraph langchain-openai langchain-community python-dotenv 1 pip install langgraph langchain-openai langchain-community python-dotenv How to run: Save as agent.py , add your OPENAI API KEY to a .env file, then run python agent.py agent.py Minimal stateful agent with tool use built on LangGraph Python 3.10+ | LangGraph 0.2+ | LangChain 0.3+ import os from dotenv import load dotenv from langchain openai import ChatOpenAI from langchain community.tools import DuckDuckGoSearchRun from langchain core.messages import HumanMessage from langgraph.prebuilt import create react agent Load API key from .env file load dotenv Initialize the language model temperature=0 for deterministic, focused responses in agentic tasks llm = ChatOpenAI model="gpt-5.5", temperature=0, api key=os.getenv "OPENAI API KEY" Register the tools the agent can use DuckDuckGoSearchRun requires no API key -- good for development tools = DuckDuckGoSearchRun create react agent from LangGraph wires together the LLM, tools, and a built-in ReAct loop -- no boilerplate required agent = create react agent llm, tools Run the agent with a sample query The agent will decide whether to use a tool based on the question result = agent.invoke { "messages": HumanMessage content="What is the current market cap of Nvidia?" } The final response is the last message in the messages list print result "messages" -1 .content 1234567891011121314151617181920212223242526272829303132333435363738 agent.py Minimal stateful agent with tool use built on LangGraph Python 3.10+ | LangGraph 0.2+ | LangChain 0.3+ import osfrom dotenv import load dotenvfrom langchain openai import ChatOpenAIfrom langchain community.tools import DuckDuckGoSearchRunfrom langchain core.messages import HumanMessagefrom langgraph.prebuilt import create react agent Load API key from .env fileload dotenv Initialize the language model temperature=0 for deterministic, focused responses in agentic tasksllm = ChatOpenAI model="gpt-5.5", temperature=0, api key=os.getenv "OPENAI API KEY" Register the tools the agent can use DuckDuckGoSearchRun requires no API key -- good for developmenttools = DuckDuckGoSearchRun create react agent from LangGraph wires together the LLM, tools, and a built-in ReAct loop -- no boilerplate requiredagent = create react agent llm, tools Run the agent with a sample query The agent will decide whether to use a tool based on the questionresult = agent.invoke { "messages": HumanMessage content="What is the current market cap of Nvidia?" } The final response is the last message in the messages listprint result "messages" -1 .content What this does: create react agent handles the full ReAct loop automatically. The agent receives the question, decides it needs current data, calls the DuckDuckGo search tool, reads the result, and synthesizes a final answer. The messages list in the output contains the full trace of that reasoning process. Layer 3: Memory Systems Statelessness is the default behavior of any LLM. Every call starts from scratch, with no knowledge of what came before unless you explicitly pass that context in. For a one-shot question, that is fine. For an agent that needs to track a conversation, remember a user’s preferences, or build on work it did yesterday, it is a fundamental problem. According to Atlan’s research https://atlan.com/know/memory-layer-for-ai-agents/ on AI agent memory, 95% of enterprise generative AI pilots delivered zero measurable ROI in 2025, with failure attributed to context readiness rather than model quality. Agents are failing not because the model is wrong, but because the memory layer is not there. There are four types of memory in a production agent, and each one handles a different job: Working memory in-context is the active context window. It holds the current conversation, any documents you have passed in, and the results of recent tool calls. It is fast and requires no infrastructure, but it is session-bound. When the session ends, it is gone. Episodic memory is a log of prior interactions. As described in the research on memory types https://atlan.com/know/ai-memory-system/ , episodic memory stores what happened: timestamp, task, actions taken, outcome. This is what allows an agent to answer “ What did we work on last Tuesday? ” or “ What did the user say about this project three sessions ago? “ Semantic memory is factual knowledge stored externally, including definitions, entity relationships, and domain-specific facts that the model was not trained on. This is where your RAG pipeline feeds in more on that in the next layer . Procedural memory encodes workflows and tool-use patterns, repeatable behaviors the agent should always follow. This lives in the system prompt or a version-controlled instruction file, and it shapes every response the agent produces. Here is how to implement working and episodic memory together using LangChain’s recommended pattern for LangChain 0.3+ : Prerequisites: pip install langchain langchain-openai python-dotenv 1 pip install langchain langchain-openai python-dotenv How to run: Save as memory.py , ensure your .env has OPENAI API KEY , then run python memory.py memory.py Working memory + episodic memory for persistent agent context Uses the current LangChain 0.3+ pattern legacy ConversationBufferMemory is deprecated import os import json from datetime import datetime from dotenv import load dotenv from langchain openai import ChatOpenAI from langchain core.messages import HumanMessage, AIMessage, SystemMessage, trim messages load dotenv llm = ChatOpenAI model="gpt-5.5", temperature=0.2, api key=os.getenv "OPENAI API KEY" ── EPISODIC MEMORY STORE ───────────────────────────────────────────────────── In production, replace this list with a database SQLite, Postgres, Redis . The structure here: each episode is a dict with timestamp, user input, and agent response. episodic store: list dict = def save episode user input: str, agent response: str - None: """Save a conversation turn to the episodic store.""" episodic store.append { "timestamp": datetime.now .isoformat , "user": user input, "agent": agent response } def load recent episodes n: int = 5 - str: """Retrieve the last N episodes as a formatted string for injection into context.""" if not episodic store: return "No prior conversation history." recent = episodic store -n: return "\n".join f" {ep 'timestamp' } User: {ep 'user' } | Agent: {ep 'agent' }" for ep in recent ── WORKING MEMORY IN-CONTEXT ─────────────────────────────────────────────── We manage the message list ourselves and pass it through trim messages before each LLM call to stay within the model's context limit. max tokens=4000 leaves headroom for the model's response. working memory: list = def chat user input: str - str: """ Send a message to the agent. Episodic history is injected into the system prompt. Working memory is trimmed before each call to prevent context overflow. """ Inject episodic memory into the system prompt so the model has long-term context system = SystemMessage content= "You are a helpful, context-aware assistant.\n\n" "Recent conversation history:\n" f"{load recent episodes }" Add the new user message to working memory working memory.append HumanMessage content=user input Trim working memory to stay within the context window This compresses older messages rather than dropping them entirely trimmed = trim messages working memory, max tokens=4000, strategy="last", Keep the most recent messages token counter=llm, Use the model's tokenizer for accurate counts include system=True, allow partial=False Call the model with system context + trimmed working memory response = llm.invoke system + trimmed reply = response.content Save the exchange to episodic memory and add the reply to working memory save episode user input, reply working memory.append AIMessage content=reply return reply ── DEMO ────────────────────────────────────────────────────────────────────── if name == " main ": print chat "My name is Alex and I'm building a RAG pipeline for legal documents." print chat "What's the best vector database for my use case?" print chat "What did I tell you I was building?" Tests episodic recall 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091 memory.py Working memory + episodic memory for persistent agent context Uses the current LangChain 0.3+ pattern legacy ConversationBufferMemory is deprecated import osimport jsonfrom datetime import datetimefrom dotenv import load dotenvfrom langchain openai import ChatOpenAIfrom langchain core.messages import HumanMessage, AIMessage, SystemMessage, trim messages load dotenv llm = ChatOpenAI model="gpt-5.5", temperature=0.2, api key=os.getenv "OPENAI API KEY" ── EPISODIC MEMORY STORE ───────────────────────────────────────────────────── In production, replace this list with a database SQLite, Postgres, Redis . The structure here: each episode is a dict with timestamp, user input, and agent response.episodic store: list dict = def save episode user input: str, agent response: str - None: """Save a conversation turn to the episodic store.""" episodic store.append { "timestamp": datetime.now .isoformat , "user": user input, "agent": agent response } def load recent episodes n: int = 5 - str: """Retrieve the last N episodes as a formatted string for injection into context.""" if not episodic store: return "No prior conversation history." recent = episodic store -n: return "\n".join f" {ep 'timestamp' } User: {ep 'user' } | Agent: {ep 'agent' }" for ep in recent ── WORKING MEMORY IN-CONTEXT ─────────────────────────────────────────────── We manage the message list ourselves and pass it through trim messages before each LLM call to stay within the model's context limit. max tokens=4000 leaves headroom for the model's response.working memory: list = def chat user input: str - str: """ Send a message to the agent. Episodic history is injected into the system prompt. Working memory is trimmed before each call to prevent context overflow. """ Inject episodic memory into the system prompt so the model has long-term context system = SystemMessage content= "You are a helpful, context-aware assistant.\n\n" "Recent conversation history:\n" f"{load recent episodes }" Add the new user message to working memory working memory.append HumanMessage content=user input Trim working memory to stay within the context window This compresses older messages rather than dropping them entirely trimmed = trim messages working memory, max tokens=4000, strategy="last", Keep the most recent messages token counter=llm, Use the model's tokenizer for accurate counts include system=True, allow partial=False Call the model with system context + trimmed working memory response = llm.invoke system + trimmed reply = response.content Save the exchange to episodic memory and add the reply to working memory save episode user input, reply working memory.append AIMessage content=reply return reply ── DEMO ──────────────────────────────────────────────────────────────────────if name == " main ": print chat "My name is Alex and I'm building a RAG pipeline for legal documents." print chat "What's the best vector database for my use case?" print chat "What did I tell you I was building?" Tests episodic recall What this does : The episodic store acts as a lightweight persistent log that gets summarized into the system prompt on every call. The working memory list holds the in-session message history and gets trimmed by trim messages before each LLM call to prevent token overflow. The final test question, “ What did I tell you I was building? ” verifies that episodic recall is working correctly even after the context window has moved on. Layer 4: Vector Databases and Retrieval RAG Foundation models know a lot, but they do not know your documents. They were not trained on your internal knowledge base, your customer support history, your proprietary research, or anything that has happened since their training cutoff. Retrieval-Augmented Generation RAG is how you fix that. The concept is straightforward : instead of trying to fit an entire knowledge base into the context window, you convert your documents into numerical representations embeddings , store them in a vector database, and retrieve only the most relevant chunks at query time. The agent gets a context window full of precisely the right information rather than everything you have ever written. The global vector database market reached \$3.2 billion in 2025 and is growing at 24% annually https://www.groovyweb.co/blog/vector-database-comparison-2026 , which reflects how central retrieval has become to production AI systems. The leading options each serve a different use case: Pinecone https://www.pinecone.io/ is fully managed with zero infrastructure overhead. You pay for it, push vectors to it, and query it. At 100 million vectors, it maintains recall without tuning. The right choice when you want to ship and not think about infrastructure. Weaviate https://weaviate.io/ is open-source with a managed cloud option, and it leads the field on hybrid search combining vector similarity, keyword matching BM25 , and metadata filtering in a single query. If your retrieval needs require more than pure semantic search, Weaviate handles it natively. Chroma https://www.trychroma.com/ is developer-first and runs locally with no infrastructure. The 2025 Rust rewrite made it significantly faster. It is the right choice for prototyping and small-to-medium production workloads where developer experience matters more than scale. pgvector https://github.com/pgvector/pgvector is a PostgreSQL extension that adds vector search to a database you may already be running. If your team already runs Postgres, pgvector is the lowest-friction path to production RAG. It handles millions of vectors with HNSW indexing and stays within single-node PostgreSQL limits for most production workloads. Here is a working RAG pipeline using Chroma and OpenAI embeddings. Prerequisites : pip install langchain langchain-openai langchain-chroma langchain-text-splitters chromadb python-dotenv 1 pip install langchain langchain-openai langchain-chroma langchain-text-splitters chromadb python-dotenv How to run: Save as rag pipeline.py , add OPENAI API KEY to your .env , then run python rag pipeline.py . rag pipeline.py Minimal RAG pipeline: ingest documents → embed → store in Chroma → retrieve and answer Python 3.10+ | ChromaDB 0.5+ | LangChain 0.3+ import os from dotenv import load dotenv from langchain openai import ChatOpenAI, OpenAIEmbeddings from langchain chroma import Chroma from langchain text splitters import RecursiveCharacterTextSplitter from langchain core.documents import Document from langchain core.prompts import ChatPromptTemplate load dotenv ── STEP 1: SAMPLE DOCUMENTS ────────────────────────────────────────────────── Replace this list with real documents from your knowledge base. In production, load from PDFs, databases, APIs, or file systems. documents = Document page content="Pinecone is a managed vector database optimized for fast, " "low-latency similarity search at scale. It handles infrastructure automatically " "and is best for production RAG when you don't want to manage servers.", metadata={"source": "vector db guide", "topic": "pinecone"} , Document page content="Weaviate is an open-source vector database with native hybrid search " "support, combining BM25 keyword search with dense vector search in a single query. " "It can be self-hosted or used via Weaviate Cloud.", metadata={"source": "vector db guide", "topic": "weaviate"} , Document page content="Chroma is a developer-friendly, local-first vector database ideal for " "prototyping. The 2025 Rust rewrite significantly improved performance. " "Best for small-to-medium production workloads and local development.", metadata={"source": "vector db guide", "topic": "chroma"} , Document page content="pgvector is a PostgreSQL extension that adds vector similarity search " "to an existing Postgres database. With HNSW indexing, it handles millions of vectors " "at low latency. Best choice if your team already runs PostgreSQL in production.", metadata={"source": "vector db guide", "topic": "pgvector"} , ── STEP 2: CHUNK THE DOCUMENTS ─────────────────────────────────────────────── Large documents are split into smaller chunks before embedding. chunk size=500 characters; chunk overlap=50 preserves context across chunk boundaries. splitter = RecursiveCharacterTextSplitter chunk size=500, chunk overlap=50 chunks = splitter.split documents documents ── STEP 3: EMBED AND STORE IN CHROMA ──────────────────────────────────────── OpenAIEmbeddings converts each chunk into a high-dimensional vector. Chroma stores those vectors locally in the ./chroma db directory. On subsequent runs, the existing store is loaded rather than rebuilt. embeddings = OpenAIEmbeddings model="text-embedding-3-small", Fast and cost-effective for most RAG tasks api key=os.getenv "OPENAI API KEY" vectorstore = Chroma.from documents documents=chunks, embedding=embeddings, persist directory="./chroma db" Persist to disk so you don't re-embed on every run ── STEP 4: RETRIEVAL ────────────────────────────────────────────────────────── Converts the query into an embedding and finds the most similar chunks. k=3 returns the top 3 most relevant chunks. retriever = vectorstore.as retriever search kwargs={"k": 3} ── STEP 5: GENERATION ──────────────────────────────────────────────────────── llm = ChatOpenAI model="gpt-5.5", temperature=0, api key=os.getenv "OPENAI API KEY" The prompt tells the model to use only the retrieved context. This prevents the model from hallucinating facts not in your knowledge base. rag prompt = ChatPromptTemplate.from messages "system", "Answer the question using only the provided context. " "If the answer isn't in the context, say so clearly.\n\n" "Context:\n{context}" , "human", "{question}" def answer question: str - str: """Retrieve relevant chunks and generate a grounded answer.""" Retrieve the most relevant document chunks for this question retrieved docs = retriever.invoke question Combine the retrieved chunks into a single context block context = "\n\n".join doc.page content for doc in retrieved docs Build and invoke the prompt with the question and retrieved context prompt = rag prompt.invoke {"context": context, "question": question} response = llm.invoke prompt return response.content ── DEMO ────────────────────────────────────────────────────────────────────── if name == " main ": q = "Which vector database should I use if I already run PostgreSQL?" print f"Q: {q}\nA: {answer q }" 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100 rag pipeline.py Minimal RAG pipeline: ingest documents → embed → store in Chroma → retrieve and answer Python 3.10+ | ChromaDB 0.5+ | LangChain 0.3+ import osfrom dotenv import load dotenvfrom langchain openai import ChatOpenAI, OpenAIEmbeddingsfrom langchain chroma import Chromafrom langchain text splitters import RecursiveCharacterTextSplitterfrom langchain core.documents import Documentfrom langchain core.prompts import ChatPromptTemplate load dotenv ── STEP 1: SAMPLE DOCUMENTS ────────────────────────────────────────────────── Replace this list with real documents from your knowledge base. In production, load from PDFs, databases, APIs, or file systems.documents = Document page content="Pinecone is a managed vector database optimized for fast, " "low-latency similarity search at scale. It handles infrastructure automatically " "and is best for production RAG when you don't want to manage servers.", metadata={"source": "vector db guide", "topic": "pinecone"} , Document page content="Weaviate is an open-source vector database with native hybrid search " "support, combining BM25 keyword search with dense vector search in a single query. " "It can be self-hosted or used via Weaviate Cloud.", metadata={"source": "vector db guide", "topic": "weaviate"} , Document page content="Chroma is a developer-friendly, local-first vector database ideal for " "prototyping. The 2025 Rust rewrite significantly improved performance. " "Best for small-to-medium production workloads and local development.", metadata={"source": "vector db guide", "topic": "chroma"} , Document page content="pgvector is a PostgreSQL extension that adds vector similarity search " "to an existing Postgres database. With HNSW indexing, it handles millions of vectors " "at low latency. Best choice if your team already runs PostgreSQL in production.", metadata={"source": "vector db guide", "topic": "pgvector"} , ── STEP 2: CHUNK THE DOCUMENTS ─────────────────────────────────────────────── Large documents are split into smaller chunks before embedding. chunk size=500 characters; chunk overlap=50 preserves context across chunk boundaries.splitter = RecursiveCharacterTextSplitter chunk size=500, chunk overlap=50 chunks = splitter.split documents documents ── STEP 3: EMBED AND STORE IN CHROMA ──────────────────────────────────────── OpenAIEmbeddings converts each chunk into a high-dimensional vector. Chroma stores those vectors locally in the ./chroma db directory. On subsequent runs, the existing store is loaded rather than rebuilt.embeddings = OpenAIEmbeddings model="text-embedding-3-small", Fast and cost-effective for most RAG tasks api key=os.getenv "OPENAI API KEY" vectorstore = Chroma.from documents documents=chunks, embedding=embeddings, persist directory="./chroma db" Persist to disk so you don't re-embed on every run ── STEP 4: RETRIEVAL ────────────────────────────────────────────────────────── Converts the query into an embedding and finds the most similar chunks. k=3 returns the top 3 most relevant chunks.retriever = vectorstore.as retriever search kwargs={"k": 3} ── STEP 5: GENERATION ────────────────────────────────────────────────────────llm = ChatOpenAI model="gpt-5.5", temperature=0, api key=os.getenv "OPENAI API KEY" The prompt tells the model to use only the retrieved context. This prevents the model from hallucinating facts not in your knowledge base.rag prompt = ChatPromptTemplate.from messages "system", "Answer the question using only the provided context. " "If the answer isn't in the context, say so clearly.\n\n" "Context:\n{context}" , "human", "{question}" def answer question: str - str: """Retrieve relevant chunks and generate a grounded answer.""" Retrieve the most relevant document chunks for this question retrieved docs = retriever.invoke question Combine the retrieved chunks into a single context block context = "\n\n".join doc.page content for doc in retrieved docs Build and invoke the prompt with the question and retrieved context prompt = rag prompt.invoke {"context": context, "question": question} response = llm.invoke prompt return response.content ── DEMO ──────────────────────────────────────────────────────────────────────if name == " main ": q = "Which vector database should I use if I already run PostgreSQL?" print f"Q: {q}\nA: {answer q }" What this does : The pipeline has two phases. During indexing, documents are chunked, converted to embeddings via OpenAI’s text-embedding-3-small model, and stored in a local Chroma database. During retrieval, the query is embedded using the same model, the three most similar chunks are pulled from Chroma, and the LLM uses those chunks and only those chunks to answer. The persist directory parameter means Chroma saves the vectors to disk, so you do not pay to re-embed your documents on every run. Layer 5: Tools and External Integrations An agent without tools is a very expensive text predictor. Tools are what give agents the ability to act on the world rather than just talk about it. In technical terms, a tool is a function that the model can choose to call. You describe what the function does in natural language, define its input parameters with a schema, and the model decides when calling that function would help it answer the question. The model does not execute the function; your code does. The model just decides when and with what arguments. The categories of tools that matter most in production agents are: web search for current information , code execution for calculation and data processing , file I/O for reading and writing documents , API calls for connecting to external services , and browser use for interacting with web interfaces that do not have APIs . One development worth understanding is the Model Context Protocol MCP https://modelcontextprotocol.io/ , introduced by Anthropic in late 2024. MCP is a standardized way for models to communicate with external tools and data sources. Rather than every team writing custom integration code for every tool, MCP provides a shared protocol. Amazon Bedrock Agents added native MCP support in 2025, and adoption across the ecosystem is growing fast. The single most important thing about tool design is the schema. The model decides whether to use a tool based on its description and decides what arguments to pass based on the parameter schema. A vague description produces wrong tool calls. A well-typed schema with clear parameter descriptions produces reliable ones. Prerequisites : pip install langchain langchain-openai langchain-community python-dotenv 1 pip install langchain langchain-openai langchain-community python-dotenv How to run : Save as tools.py , add OPENAI API KEY to your .env , then run python tools.py tools.py Defining, registering, and using tools with a LangChain agent Python 3.10+ | LangChain 0.3+ import os import json import requests from dotenv import load dotenv from langchain openai import ChatOpenAI from langchain.tools import tool from langchain community.tools import DuckDuckGoSearchRun from langchain core.messages import HumanMessage from langgraph.prebuilt import create react agent load dotenv llm = ChatOpenAI model="gpt-5.5", temperature=0, api key=os.getenv "OPENAI API KEY" ── TOOL 1: WEB SEARCH ──────────────────────────────────────────────────────── Built-in DuckDuckGo tool -- no API key needed. search = DuckDuckGoSearchRun ── TOOL 2: WEATHER LOOKUP ──────────────────────────────────────────────────── The @tool decorator does three things: 1. Registers the function as a callable tool 2. Uses the function name as the tool name 3. Uses the docstring as the tool description this is what the model reads The description is critical -- vague descriptions cause wrong tool calls. @tool def get weather city: str - str: """ Fetch the current weather for a given city. Use this when the user asks about weather conditions, temperature, or forecasts. Input: city name as a string e.g., 'London', 'Tokyo', 'New York' . """ try: Using open-meteo free, no API key for geocoding and weather geo url = f"https://geocoding-api.open-meteo.com/v1/search?name={city}&count=1" geo = requests.get geo url, timeout=5 .json if not geo.get "results" : return f"Could not find location data for '{city}'." lat = geo "results" 0 "latitude" lon = geo "results" 0 "longitude" weather url = f"https://api.open-meteo.com/v1/forecast" f"?latitude={lat}&longitude={lon}" f"&current weather=true" weather = requests.get weather url, timeout=5 .json current = weather.get "current weather", {} return f"Weather in {city}: " f"{current.get 'temperature', 'N/A' }°C, " f"wind speed {current.get 'windspeed', 'N/A' } km/h." except Exception as e: Always return a string from tools, even on failure. Raising exceptions from tools can crash the agent loop. return f"Weather lookup failed for '{city}': {str e }" ── TOOL 3: JSON CALCULATOR ─────────────────────────────────────────────────── @tool def calculate expression: str - str: """ Evaluate a mathematical expression and return the result. Use this for arithmetic, percentage calculations, or any numerical computation. Input: a valid Python mathematical expression as a string e.g., ' 150 1.08 / 12' . Do NOT use for complex code execution -- only simple math expressions. """ try: eval is scoped to only allow math -- no builtins, no imports result = eval expression, {" builtins ": {}}, {} return f"Result: {result}" except Exception as e: return f"Calculation error: {str e }" ── REGISTER TOOLS AND BUILD AGENT ──────────────────────────────────────────── tools = search, get weather, calculate agent = create react agent llm, tools ── DEMO ────────────────────────────────────────────────────────────────────── if name == " main ": queries = "What is the weather in Lagos right now?", "If I earn $85,000 a year, what is my monthly gross salary?", "Who won the most recent FIFA World Cup?" for query in queries: print f"\nQuery: {query}" result = agent.invoke {"messages": HumanMessage content=query } print f"Answer: {result 'messages' -1 .content}" 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798 tools.py Defining, registering, and using tools with a LangChain agent Python 3.10+ | LangChain 0.3+ import osimport jsonimport requestsfrom dotenv import load dotenvfrom langchain openai import ChatOpenAIfrom langchain.tools import toolfrom langchain community.tools import DuckDuckGoSearchRunfrom langchain core.messages import HumanMessagefrom langgraph.prebuilt import create react agent load dotenv llm = ChatOpenAI model="gpt-5.5", temperature=0, api key=os.getenv "OPENAI API KEY" ── TOOL 1: WEB SEARCH ──────────────────────────────────────────────────────── Built-in DuckDuckGo tool -- no API key needed.search = DuckDuckGoSearchRun ── TOOL 2: WEATHER LOOKUP ──────────────────────────────────────────────────── The @tool decorator does three things: 1. Registers the function as a callable tool 2. Uses the function name as the tool name 3. Uses the docstring as the tool description this is what the model reads The description is critical -- vague descriptions cause wrong tool calls.@tooldef get weather city: str - str: """ Fetch the current weather for a given city. Use this when the user asks about weather conditions, temperature, or forecasts. Input: city name as a string e.g., 'London', 'Tokyo', 'New York' . """ try: Using open-meteo free, no API key for geocoding and weather geo url = f"https://geocoding-api.open-meteo.com/v1/search?name={city}&count=1" geo = requests.get geo url, timeout=5 .json if not geo.get "results" : return f"Could not find location data for '{city}'." lat = geo "results" 0 "latitude" lon = geo "results" 0 "longitude" weather url = f"https://api.open-meteo.com/v1/forecast" f"?latitude={lat}&longitude={lon}" f"&current weather=true" weather = requests.get weather url, timeout=5 .json current = weather.get "current weather", {} return f"Weather in {city}: " f"{current.get 'temperature', 'N/A' }°C, " f"wind speed {current.get 'windspeed', 'N/A' } km/h." except Exception as e: Always return a string from tools, even on failure. Raising exceptions from tools can crash the agent loop. return f"Weather lookup failed for '{city}': {str e }" ── TOOL 3: JSON CALCULATOR ───────────────────────────────────────────────────@tooldef calculate expression: str - str: """ Evaluate a mathematical expression and return the result. Use this for arithmetic, percentage calculations, or any numerical computation. Input: a valid Python mathematical expression as a string e.g., ' 150 1.08 / 12' . Do NOT use for complex code execution -- only simple math expressions. """ try: eval is scoped to only allow math -- no builtins, no imports result = eval expression, {" builtins ": {}}, {} return f"Result: {result}" except Exception as e: return f"Calculation error: {str e }" ── REGISTER TOOLS AND BUILD AGENT ────────────────────────────────────────────tools = search, get weather, calculate agent = create react agent llm, tools ── DEMO ──────────────────────────────────────────────────────────────────────if name == " main ": queries = "What is the weather in Lagos right now?", "If I earn $85,000 a year, what is my monthly gross salary?", "Who won the most recent FIFA World Cup?" for query in queries: print f"\nQuery: {query}" result = agent.invoke {"messages": HumanMessage content=query } print f"Answer: {result 'messages' -1 .content}" What this does: Three tools are registered: a web search tool for current events, a weather tool that calls a free API with no key required, and a calculator that safely evaluates mathematical expressions. The agent receives each query, reasons about which tool to use, calls it, and synthesizes an answer from the result. The key design detail to notice is in the docstrings; each tool description is precise about what the tool does, when to use it, and what format the input should take. Layer 6: Observability and Evaluation Here is a production truth that does not get said enough: LLMs fail silently. As the team at Kanerika https://medium.com/@kanerika/llmops-observability-langsmith-vs-arize-vs-langfuse-vs-w-b-f1baeabd1bbf put it, a hallucinated answer still returns HTTP 200. A standard infrastructure monitoring tool sees a successful request. You see nothing unusual. Meanwhile, your agent has been confidently giving wrong answers for three days. Traditional monitoring was built for a world where “ correct ” is binary: the function returned the right type, the API returned 200, the query completed in under 100ms . LLM correctness is semantic. The response can be structurally valid, grammatically fluent, and completely wrong. That requires a different observability layer entirely. There are three things a good LLM observability setup tracks. Tracing follows every step of the agent’s execution: the LLM calls, the tool invocations, the retrieval queries, the intermediate reasoning steps, and how long each one took. Evaluation scores the output against metrics that matter: faithfulness did it stay grounded in the retrieved context? , relevance did it answer the question asked? , and hallucination rate. Monitoring tracks behavioral drift over time, whether the agent’s performance on a given class of inputs is getting better or worse as the model and prompts evolve. The leading platforms each have a different strength. LangSmith https://www.langchain.com/langsmith provides the deepest integration with LangChain and LangGraph. If you are already in that ecosystem, it is the fastest path to working traces. Langfuse https://langfuse.com/ is open-source with over 19,000 GitHub stars and an MIT license, self-hostable, and works with any framework. Arize Phoenix https://phoenix.arize.com/ brings ML-grade evaluation rigor and ships with over 50 research-backed metrics covering faithfulness, relevance, safety, and hallucination detection. According to MLflow’s analysis of observability platforms https://mlflow.org/top-5-agent-observability-tools/ , the right choice often comes down to your framework: LangChain teams get the most from LangSmith, while teams on LlamaIndex or raw API calls are better served by Phoenix or Langfuse. Here is how to add Langfuse tracing to an existing agent with minimal changes. Prerequisites : pip install langfuse langchain langchain-openai python-dotenv 1 pip install langfuse langchain langchain-openai python-dotenv Sign up at langfuse.com https://langfuse.com/ for a free account and add LANGFUSE PUBLIC KEY and LANGFUSE SECRET KEY to your .env . Self-hosting is also available if you prefer to keep data on your own infrastructure. How to run : Save as observability.py and run python observability.py . Open your Langfuse dashboard to see the trace. observability.py Adding Langfuse tracing to a LangChain agent Langfuse captures every LLM call, tool invocation, and token count automatically. import os from dotenv import load dotenv from langchain openai import ChatOpenAI from langchain community.tools import DuckDuckGoSearchRun from langchain core.messages import HumanMessage from langgraph.prebuilt import create react agent Langfuse integrates via the CallbackHandler pattern. It intercepts every LangChain event and sends it to your Langfuse dashboard. from langfuse.langchain import CallbackHandler load dotenv ── LANGFUSE SETUP ───────────────────────────────────────────────────────────── CallbackHandler reads LANGFUSE PUBLIC KEY and LANGFUSE SECRET KEY from the environment. session id groups all related traces into one session -- useful for debugging conversations. user id ties traces to a specific user for per-user performance analysis. langfuse handler = CallbackHandler session id="demo session 001", user id="demo user" ── AGENT SETUP ──────────────────────────────────────────────────────────────── llm = ChatOpenAI model="gpt-5.5", temperature=0, api key=os.getenv "OPENAI API KEY" , callbacks= langfuse handler Attach the handler here -- this is the only change tools = DuckDuckGoSearchRun agent = create react agent llm, tools ── RUN WITH TRACING ────────────────────────────────────────────────────────── Pass the handler in config so it traces tool calls as well as LLM calls. Without this, only the LLM calls are traced -- tool invocations are invisible. result = agent.invoke {"messages": HumanMessage content="What is the latest version of Python?" }, config={"callbacks": langfuse handler } print result "messages" -1 .content Flush ensures all traces are sent to Langfuse before the script exits. In a long-running server, this is handled automatically. langfuse handler.flush print "\nTrace sent to Langfuse. Check your dashboard at https://cloud.langfuse.com" 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152 observability.py Adding Langfuse tracing to a LangChain agent Langfuse captures every LLM call, tool invocation, and token count automatically. import osfrom dotenv import load dotenvfrom langchain openai import ChatOpenAIfrom langchain community.tools import DuckDuckGoSearchRunfrom langchain core.messages import HumanMessagefrom langgraph.prebuilt import create react agent Langfuse integrates via the CallbackHandler pattern. It intercepts every LangChain event and sends it to your Langfuse dashboard.from langfuse.langchain import CallbackHandler load dotenv ── LANGFUSE SETUP ───────────────────────────────────────────────────────────── CallbackHandler reads LANGFUSE PUBLIC KEY and LANGFUSE SECRET KEY from the environment. session id groups all related traces into one session -- useful for debugging conversations. user id ties traces to a specific user for per-user performance analysis.langfuse handler = CallbackHandler session id="demo session 001", user id="demo user" ── AGENT SETUP ────────────────────────────────────────────────────────────────llm = ChatOpenAI model="gpt-5.5", temperature=0, api key=os.getenv "OPENAI API KEY" , callbacks= langfuse handler Attach the handler here -- this is the only change tools = DuckDuckGoSearchRun agent = create react agent llm, tools ── RUN WITH TRACING ────────────────────────────────────────────────────────── Pass the handler in config so it traces tool calls as well as LLM calls. Without this, only the LLM calls are traced -- tool invocations are invisible.result = agent.invoke {"messages": HumanMessage content="What is the latest version of Python?" }, config={"callbacks": langfuse handler } print result "messages" -1 .content Flush ensures all traces are sent to Langfuse before the script exits. In a long-running server, this is handled automatically.langfuse handler.flush print "\nTrace sent to Langfuse. Check your dashboard at https://cloud.langfuse.com" What this does : Two changes from a standard agent setup: the CallbackHandler is initialized with a session and user ID, and it is attached to both the LLM and the agent.invoke config. That is enough for Langfuse to capture the full trace of every LLM call, every tool invocation, token counts, latency, and the complete input/output at each step. Everything you need to debug a production failure or track quality drift over time. Layer 7: Deployment Infrastructure You can have a flawless agent in development that turns into a maintenance problem in production. The infrastructure layer is where that gap lives. At a minimum, your agent should be containerized with Docker https://www.docker.com/ . Containers give you consistent behavior across environments, straightforward dependency management, and a clean path to any cloud deployment target. The alternative — shipping Python scripts with a requirements.txt and hoping the environment matches — creates a class of bugs that wastes engineering time disproportionate to the effort containerization would have taken. For most production agents, you have two architectural options for the serving layer: a synchronous API or an async queue. A synchronous API Flask or FastAPI works when your agent completes in under a few seconds, and you can afford to hold the HTTP connection open. When your agent involves multiple tool calls, long retrieval pipelines, or document processing that might take 30 to 60 seconds, an async queue Celery, AWS SQS, or Google Pub/Sub is the better choice. The client submits a job, gets a task ID back immediately, and polls for the result. On the cloud side, all three major platforms now have managed agent infrastructure. Amazon’s AgentCore https://aws.amazon.com/bedrock/agentcore/ , which became generally available in October 2025, provides dedicated agentic infrastructure on AWS for memory management, tool execution, and session handling without provisioning servers. Google Vertex AI Agent Builder https://cloud.google.com/products/agent-builder is the natural choice for teams already in the GCP ecosystem, with native Gemini integration and built-in observability. Azure OpenAI Service https://azure.microsoft.com/en-us/products/ai-services/openai-service with Semantic Kernel is the enterprise default for Microsoft shops. For cost management, three practices make the biggest difference: caching returning stored responses for repeated identical queries rather than calling the model again , request batching grouping non-urgent tasks to reduce per-call overhead , and setting max iterations in your agent executor to prevent runaway loops from consuming tokens without bound. Putting It All Together The right choices at each layer depend on where you are in the project lifecycle. Here is a practical reference that reflects the research and trade-offs discussed above. Prototype move fast, minimal infrastructure : Layer | Choice | Reason | |---|---|---| | Foundation Model | GPT-5.5 | Reliable tool-calling, mature ecosystem | | Orchestration | LangGraph | Fast setup, good documentation | | Memory | In-context only | No infrastructure needed | | Vector DB | Chroma | Local, no ops, good developer experience | | Tools | DuckDuckGo + custom @tool functions | Zero API keys required | | Observability | Langfuse cloud free tier | One-line setup | | Deployment | Local / Docker | Ship fast | Production Startup scale with control : Layer | Choice | Reason | |---|---|---| | Foundation Model | GPT-5.5 + Claude Sonnet 4.6 fallback | Reliability with redundancy | | Orchestration | LangGraph or CrewAI | State management and multi-agent support | | Memory | Episodic Postgres + Semantic RAG | Full persistent context | | Vector DB | Weaviate or Pinecone | Scale and hybrid search | | Tools | Full tool suite with MCP | Standardized integrations | | Observability | Langfuse self-hosted or Arize Phoenix | Data control + ML-grade evals | | Deployment | Docker + Kubernetes + async queue | Production-grade, cost-controlled | Enterprise: Layer | Choice | Reason | |---|---|---| | Foundation Model | Azure OpenAI or AWS Bedrock | Compliance, data residency, SLA | | Orchestration | Semantic Kernel or LangGraph | Enterprise language support, governance | | Memory | Managed memory with audit trail | Regulatory requirements | | Vector DB | Weaviate or pgvector | Self-hostable, compliance-ready | | Tools | MCP-based, internally approved | Security review and access control | | Observability | Langfuse self-hosted or Datadog LLM module | Existing infrastructure integration | | Deployment | AWS AgentCore / Vertex AI Agent Builder | Fully managed, governed, auditable | Conclusion The foundation model is the part of this stack that gets written about. The other six layers are the parts that determine whether what you built actually works in production. An agent fails at the orchestration layer when the ReAct loop gets stuck. It fails at the memory layer when it forgets the context it needs. It fails at the retrieval layer when the wrong chunks are returned, and the model hallucinates a grounded-sounding answer. It fails at the tools layer when a schema is too vague, and the model calls the wrong function. It fails at the observability layer when you have no way to know that any of this is happening. And it fails at the deployment layer when the infrastructure cannot handle the latency or cost requirements of real traffic. Gartner estimates that over 40% of agentic AI projects are at risk of cancellation by 2027 https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025 due to unclear value, rising costs, and weak governance. Most of those failures will trace back not to a bad model choice but to a stack that was built layer by layer without a clear picture of how the layers connect. Understanding the full stack does not mean you have to build all of it. It means you know what decisions you are making and what you are trading off when you make them. That is the difference between an agent that works in a demo and one that ships.