{"slug": "the-ai-agent-tech-stack-explained", "title": "The AI Agent Tech Stack Explained", "summary": "Gartner predicts 40% of enterprise apps will feature AI agents by 2026, up from under 5% in 2025. A seven-layer tech stack—from foundation models like GPT-5.5 and Claude Sonnet 4.6 to deployment infrastructure—enables agents to perform complex tasks such as researching competitors and generating reports. Understanding each layer is critical for engineers building production systems.", "body_md": "In this article, you will learn how the seven layers of a production AI agent stack fit together, from the foundation model down to deployment infrastructure.\n\nTopics we will cover include:\n\n- What each layer of the stack does, from the foundation model and orchestration framework through memory, retrieval, tools, observability, and deployment.\n- How to implement each layer with working code, including a stateful agent, a memory system, a RAG pipeline, custom tools, and tracing.\n- Which combination of technologies to use at each layer depending on whether you are prototyping, scaling a startup, or operating in an enterprise environment.\n\n## Introduction\n\nPicture this: you ask an AI agent to research three competitors, pull the pricing data from each of their websites, summarize the findings into a structured report, and drop it in a Slack channel by 9am. You hit enter. Thirty seconds later, the report is there.\n\nWhat just happened under the hood is not magic, and it is not one thing. It is seven distinct layers of technology working in sequence, each one handling a specific job, each one capable of breaking in its own specific way. The model at the top gets all the attention. The six layers beneath it are what determine whether the agent actually works.\n\n[According to Gartner](https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025), 40% of enterprise applications will be integrated with task-specific AI agents by the end of 2026, up from less than 5% in 2025. That is not a gradual curve. That is a near-vertical adoption line, and the engineers and technical leads responsible for those deployments need to understand the full stack, not just the layer they happen to own.\n\nThis article goes through each layer in order, from the foundation model down to deployment infrastructure. By the end, you will know what every piece is, why it exists, how the layers connect to each other, and what to actually use at each level.\n\n## Layer 1: The Foundation Model\n\nThe foundation model is the cognitive core of an agent. It is where reasoning happens, language is understood, and decisions about what to do next are made. Everything else in the stack is either feeding context into it or acting on what it produces.\n\nIn practical terms, your main options in 2026 are [OpenAI’s GPT-5.5](https://openai.com/index/introducing-gpt-5-5/), [Anthropic’s Claude Sonnet 4.6](https://www.anthropic.com/news/claude-sonnet-4-6) (or [Claude Opus 4.8](https://www.anthropic.com/news/claude-opus-4-8) for harder reasoning), [Google’s Gemini 3.1 Pro](https://deepmind.google/models/gemini/pro/), and open-weight models like [Meta’s Llama 4](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) and [Mistral Large 3](https://huggingface.co/collections/mistralai/mistral-large-3). Each has trade-offs worth understanding before you commit.\n\n**GPT-5.5** is fast for everyday calls and reliable at tool-calling, and it has the most mature ecosystem of integrations and the widest community of developers who have already run into and solved the edge cases you will encounter. **Claude Sonnet 4.6** handles long documents and nuanced instruction-following well at a lower price point than Anthropic’s Opus tier, which matters in document-heavy workflows; reach for **Claude Opus 4.8** when a task needs deeper, longer-horizon reasoning. **Gemini 3.1 Pro** has a 1 million token context window, which is relevant if your agent needs to process large codebases or lengthy knowledge bases in a single pass. Open-weight models like **Llama 4** give you full control over deployment and data residency, at the cost of the infrastructure overhead of running them yourself.\n\nThere is no longer a hard split between “**standard**” and “** reasoning**” model families, the way there was in 2025; [OpenAI](https://openai.com/), [Anthropic](https://www.anthropic.com/), and [Google](https://www.google.com/) have each folded reasoning into a single model that decides how long to think. GPT-5.5 ships with adjustable reasoning effort levels (*from none up to xhigh*), and the same applies to Claude’s effort parameter and Gemini’s thinking levels. For most agent workflows, the default or low-effort setting is the right choice: fast and cheap. For tasks that require careful planning or mathematical reasoning, dialling the effort level up earns back its cost in correctness.\n\n## Layer 2: The Orchestration Framework\n\nIf the foundation model is the brain, the orchestration framework is the nervous system. It handles the control flow: deciding what the agent should do next, when it should call a tool, how it should handle the result, and how the whole reasoning loop stays coherent across multiple steps.\n\nThe pattern that most frameworks implement is called ReAct (Reasoning and Acting). The agent produces a thought, decides on an action, executes the action through a tool, observes the result, and then thinks again. This loop repeats until the agent produces a final answer. It sounds simple. In practice, it is where most production failures occur: the agent calls the wrong tool, gets stuck in a loop, or fails to recognise when it has enough information to stop.\n\n[LangChain](https://www.langchain.com/)is the most widely adopted framework. It offers a large ecosystem of integrations and good documentation. The criticism that it adds too much abstraction is fair at the prototype stage, but less relevant once you need the features that abstraction provides.[LangGraph](https://langchain-ai.github.io/langgraph/), built by the same team, is better suited for stateful multi-agent workflows where you need fine-grained control over the execution graph. If your agent involves multiple specialists coordinating on a task, LangGraph is the cleaner choice.[CrewAI](https://www.crewai.com/)is designed specifically for multi-agent coordination. It lets you define agents with roles, assign them tasks, and have them collaborate within a structured workflow. It is higher-level than LangGraph and faster to get running, but gives you less control over the execution details.[AutoGen](https://microsoft.github.io/autogen/), from Microsoft, takes a conversational approach to multi-agent systems. Agents interact with each other through a message-passing interface, which makes the interaction logic very readable.[Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/overview/)is Microsoft’s enterprise-focused option, with production-ready support for C#, Python, and Java. If you are operating in an enterprise environment already running on the Microsoft stack, it fits naturally.[LlamaIndex](https://www.llamaindex.ai/)started as a document ingestion and retrieval framework and has since grown into a full agent framework, with particularly strong support for RAG-heavy workflows.\n\nThe right choice depends on what your agent needs to do. For a single-agent task runner: LangGraph or LangChain. For a coordinated team of specialized agents: [CrewAI](https://crewai.com/) or [AutoGen](https://microsoft.github.io/autogen/stable//index.html). For enterprise environments: Semantic Kernel. For document-heavy retrieval workflows: [LlamaIndex](https://www.llamaindex.ai/).\n\nHere is a minimal working agent in LangGraph that handles tool use and maintains state.\n\n**Prerequisites**:\n\n```\npip install langgraph langchain-openai langchain-community python-dotenv\n\n1\n\npip install langgraph langchain-openai langchain-community python-dotenv\n```\n\nHow to run: Save as **agent.py**, add your **OPENAI_API_KEY** to a **.env** file, then run **python agent.py**\n\n```\n# agent.py\n# Minimal stateful agent with tool use built on LangGraph\n# Python 3.10+ | LangGraph 0.2+ | LangChain 0.3+\n\nimport os\nfrom dotenv import load_dotenv\nfrom langchain_openai import ChatOpenAI\nfrom langchain_community.tools import DuckDuckGoSearchRun\nfrom langchain_core.messages import HumanMessage\nfrom langgraph.prebuilt import create_react_agent\n\n# Load API key from .env file\nload_dotenv()\n\n# Initialize the language model\n# temperature=0 for deterministic, focused responses in agentic tasks\nllm = ChatOpenAI(\n    model=\"gpt-5.5\",\n    temperature=0,\n    api_key=os.getenv(\"OPENAI_API_KEY\")\n)\n\n# Register the tools the agent can use\n# DuckDuckGoSearchRun requires no API key -- good for development\ntools = [DuckDuckGoSearchRun()]\n\n# create_react_agent from LangGraph wires together the LLM,\n# tools, and a built-in ReAct loop -- no boilerplate required\nagent = create_react_agent(llm, tools)\n\n# Run the agent with a sample query\n# The agent will decide whether to use a tool based on the question\nresult = agent.invoke({\n    \"messages\": [HumanMessage(content=\"What is the current market cap of Nvidia?\")]\n})\n\n# The final response is the last message in the messages list\nprint(result[\"messages\"][-1].content)\n\n1234567891011121314151617181920212223242526272829303132333435363738\n\n# agent.py# Minimal stateful agent with tool use built on LangGraph# Python 3.10+ | LangGraph 0.2+ | LangChain 0.3+ import osfrom dotenv import load_dotenvfrom langchain_openai import ChatOpenAIfrom langchain_community.tools import DuckDuckGoSearchRunfrom langchain_core.messages import HumanMessagefrom langgraph.prebuilt import create_react_agent # Load API key from .env fileload_dotenv() # Initialize the language model# temperature=0 for deterministic, focused responses in agentic tasksllm = ChatOpenAI(    model=\"gpt-5.5\",    temperature=0,    api_key=os.getenv(\"OPENAI_API_KEY\")) # Register the tools the agent can use# DuckDuckGoSearchRun requires no API key -- good for developmenttools = [DuckDuckGoSearchRun()] # create_react_agent from LangGraph wires together the LLM,# tools, and a built-in ReAct loop -- no boilerplate requiredagent = create_react_agent(llm, tools) # Run the agent with a sample query# The agent will decide whether to use a tool based on the questionresult = agent.invoke({    \"messages\": [HumanMessage(content=\"What is the current market cap of Nvidia?\")]}) # The final response is the last message in the messages listprint(result[\"messages\"][-1].content)\n```\n\n**What this does:** **create_react_agent** handles the full ReAct loop automatically. The agent receives the question, decides it needs current data, calls the DuckDuckGo search tool, reads the result, and synthesizes a final answer. The **messages** list in the output contains the full trace of that reasoning process.\n\n## Layer 3: Memory Systems\n\nStatelessness is the default behavior of any LLM. Every call starts from scratch, with no knowledge of what came before unless you explicitly pass that context in. For a one-shot question, that is fine. For an agent that needs to track a conversation, remember a user’s preferences, or build on work it did yesterday, it is a fundamental problem.\n\nAccording to [Atlan’s research](https://atlan.com/know/memory-layer-for-ai-agents/) on AI agent memory, 95% of enterprise generative AI pilots delivered zero measurable ROI in 2025, with failure attributed to context readiness rather than model quality. Agents are failing not because the model is wrong, but because the memory layer is not there.\n\nThere are four types of memory in a production agent, and each one handles a different job:\n\n**Working memory**(in-context) is the active context window. It holds the current conversation, any documents you have passed in, and the results of recent tool calls. It is fast and requires no infrastructure, but it is session-bound. When the session ends, it is gone.**Episodic memory** is a log of prior interactions. As described in the[research on memory types](https://atlan.com/know/ai-memory-system/), episodic memory stores what happened: timestamp, task, actions taken, outcome. This is what allows an agent to answer “*What did we work on last Tuesday?*” or “*What did the user say about this project three sessions ago?*“**Semantic memory** is factual knowledge stored externally, including definitions, entity relationships, and domain-specific facts that the model was not trained on. This is where your RAG pipeline feeds in (more on that in the next layer).**Procedural memory** encodes workflows and tool-use patterns, repeatable behaviors the agent should always follow. This lives in the system prompt or a version-controlled instruction file, and it shapes every response the agent produces.\n\nHere is how to implement working and episodic memory together using LangChain’s recommended pattern for **LangChain 0.3+**:\n\n**Prerequisites:**\n\n```\npip install langchain langchain-openai python-dotenv\n\n1\n\npip install langchain langchain-openai python-dotenv\n```\n\nHow to run: Save as **memory.py**, ensure your **.env** has **OPENAI_API_KEY**, then run **python memory.py**\n\n```\n# memory.py\n# Working memory + episodic memory for persistent agent context\n# Uses the current LangChain 0.3+ pattern (legacy ConversationBufferMemory is deprecated)\n\nimport os\nimport json\nfrom datetime import datetime\nfrom dotenv import load_dotenv\nfrom langchain_openai import ChatOpenAI\nfrom langchain_core.messages import HumanMessage, AIMessage, SystemMessage, trim_messages\n\nload_dotenv()\n\nllm = ChatOpenAI(\n    model=\"gpt-5.5\",\n    temperature=0.2,\n    api_key=os.getenv(\"OPENAI_API_KEY\")\n)\n\n# ── EPISODIC MEMORY STORE ─────────────────────────────────────────────────────\n# In production, replace this list with a database (SQLite, Postgres, Redis).\n# The structure here: each episode is a dict with timestamp, user input, and agent response.\nepisodic_store: list[dict] = []\n\ndef save_episode(user_input: str, agent_response: str) -> None:\n    \"\"\"Save a conversation turn to the episodic store.\"\"\"\n    episodic_store.append({\n        \"timestamp\": datetime.now().isoformat(),\n        \"user\": user_input,\n        \"agent\": agent_response\n    })\n\ndef load_recent_episodes(n: int = 5) -> str:\n    \"\"\"Retrieve the last N episodes as a formatted string for injection into context.\"\"\"\n    if not episodic_store:\n        return \"No prior conversation history.\"\n    recent = episodic_store[-n:]\n    return \"\\n\".join(\n        f\"[{ep['timestamp']}] User: {ep['user']} | Agent: {ep['agent']}\"\n        for ep in recent\n    )\n\n# ── WORKING MEMORY (IN-CONTEXT) ───────────────────────────────────────────────\n# We manage the message list ourselves and pass it through trim_messages\n# before each LLM call to stay within the model's context limit.\n# max_tokens=4000 leaves headroom for the model's response.\nworking_memory: list = []\n\ndef chat(user_input: str) -> str:\n    \"\"\"\n    Send a message to the agent.\n    Episodic history is injected into the system prompt.\n    Working memory is trimmed before each call to prevent context overflow.\n    \"\"\"\n    # Inject episodic memory into the system prompt so the model has long-term context\n    system = SystemMessage(content=(\n        \"You are a helpful, context-aware assistant.\\n\\n\"\n        \"Recent conversation history:\\n\"\n        f\"{load_recent_episodes()}\"\n    ))\n\n    # Add the new user message to working memory\n    working_memory.append(HumanMessage(content=user_input))\n\n    # Trim working memory to stay within the context window\n    # This compresses older messages rather than dropping them entirely\n    trimmed = trim_messages(\n        working_memory,\n        max_tokens=4000,\n        strategy=\"last\",              # Keep the most recent messages\n        token_counter=llm,            # Use the model's tokenizer for accurate counts\n        include_system=True,\n        allow_partial=False\n    )\n\n    # Call the model with system context + trimmed working memory\n    response = llm.invoke([system] + trimmed)\n    reply = response.content\n\n    # Save the exchange to episodic memory and add the reply to working memory\n    save_episode(user_input, reply)\n    working_memory.append(AIMessage(content=reply))\n\n    return reply\n\n# ── DEMO ──────────────────────────────────────────────────────────────────────\nif __name__ == \"__main__\":\n    print(chat(\"My name is Alex and I'm building a RAG pipeline for legal documents.\"))\n    print(chat(\"What's the best vector database for my use case?\"))\n    print(chat(\"What did I tell you I was building?\"))  # Tests episodic recall\n\n12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091\n\n# memory.py# Working memory + episodic memory for persistent agent context# Uses the current LangChain 0.3+ pattern (legacy ConversationBufferMemory is deprecated) import osimport jsonfrom datetime import datetimefrom dotenv import load_dotenvfrom langchain_openai import ChatOpenAIfrom langchain_core.messages import HumanMessage, AIMessage, SystemMessage, trim_messages load_dotenv() llm = ChatOpenAI(    model=\"gpt-5.5\",    temperature=0.2,    api_key=os.getenv(\"OPENAI_API_KEY\")) # ── EPISODIC MEMORY STORE ─────────────────────────────────────────────────────# In production, replace this list with a database (SQLite, Postgres, Redis).# The structure here: each episode is a dict with timestamp, user input, and agent response.episodic_store: list[dict] = [] def save_episode(user_input: str, agent_response: str) -> None:    \"\"\"Save a conversation turn to the episodic store.\"\"\"    episodic_store.append({        \"timestamp\": datetime.now().isoformat(),        \"user\": user_input,        \"agent\": agent_response    }) def load_recent_episodes(n: int = 5) -> str:    \"\"\"Retrieve the last N episodes as a formatted string for injection into context.\"\"\"    if not episodic_store:        return \"No prior conversation history.\"    recent = episodic_store[-n:]    return \"\\n\".join(        f\"[{ep['timestamp']}] User: {ep['user']} | Agent: {ep['agent']}\"        for ep in recent    ) # ── WORKING MEMORY (IN-CONTEXT) ───────────────────────────────────────────────# We manage the message list ourselves and pass it through trim_messages# before each LLM call to stay within the model's context limit.# max_tokens=4000 leaves headroom for the model's response.working_memory: list = [] def chat(user_input: str) -> str:    \"\"\"    Send a message to the agent.    Episodic history is injected into the system prompt.    Working memory is trimmed before each call to prevent context overflow.    \"\"\"    # Inject episodic memory into the system prompt so the model has long-term context    system = SystemMessage(content=(        \"You are a helpful, context-aware assistant.\\n\\n\"        \"Recent conversation history:\\n\"        f\"{load_recent_episodes()}\"    ))     # Add the new user message to working memory    working_memory.append(HumanMessage(content=user_input))     # Trim working memory to stay within the context window    # This compresses older messages rather than dropping them entirely    trimmed = trim_messages(        working_memory,        max_tokens=4000,        strategy=\"last\",              # Keep the most recent messages        token_counter=llm,            # Use the model's tokenizer for accurate counts        include_system=True,        allow_partial=False    )     # Call the model with system context + trimmed working memory    response = llm.invoke([system] + trimmed)    reply = response.content     # Save the exchange to episodic memory and add the reply to working memory    save_episode(user_input, reply)    working_memory.append(AIMessage(content=reply))     return reply  # ── DEMO ──────────────────────────────────────────────────────────────────────if __name__ == \"__main__\":    print(chat(\"My name is Alex and I'm building a RAG pipeline for legal documents.\"))    print(chat(\"What's the best vector database for my use case?\"))    print(chat(\"What did I tell you I was building?\"))  # Tests episodic recall\n```\n\n**What this does**: The **episodic_store** acts as a lightweight persistent log that gets summarized into the system prompt on every call. The **working_memory** list holds the in-session message history and gets trimmed by **trim_messages** before each LLM call to prevent token overflow. The final test question, “*What did I tell you I was building?*” verifies that episodic recall is working correctly even after the context window has moved on.\n\n## Layer 4: Vector Databases and Retrieval (RAG)\n\nFoundation models know a lot, but they do not know your documents. They were not trained on your internal knowledge base, your customer support history, your proprietary research, or anything that has happened since their training cutoff. Retrieval-Augmented Generation (RAG) is how you fix that.\n\n**The concept is straightforward**: instead of trying to fit an entire knowledge base into the context window, you convert your documents into numerical representations (embeddings), store them in a vector database, and retrieve only the most relevant chunks at query time. The agent gets a context window full of precisely the right information rather than everything you have ever written.\n\nThe global vector database market reached [\\$3.2 billion in 2025 and is growing at 24% annually](https://www.groovyweb.co/blog/vector-database-comparison-2026), which reflects how central retrieval has become to production AI systems.\n\nThe leading options each serve a different use case:\n\n[Pinecone](https://www.pinecone.io/)is fully managed with zero infrastructure overhead. You pay for it, push vectors to it, and query it. At 100 million vectors, it maintains recall without tuning. The right choice when you want to ship and not think about infrastructure.[Weaviate](https://weaviate.io/)is open-source with a managed cloud option, and it leads the field on hybrid search combining vector similarity, keyword matching (BM25), and metadata filtering in a single query. If your retrieval needs require more than pure semantic search, Weaviate handles it natively.[Chroma](https://www.trychroma.com/)is developer-first and runs locally with no infrastructure. The 2025 Rust rewrite made it significantly faster. It is the right choice for prototyping and small-to-medium production workloads where developer experience matters more than scale.[pgvector](https://github.com/pgvector/pgvector)is a PostgreSQL extension that adds vector search to a database you may already be running. If your team already runs Postgres, pgvector is the lowest-friction path to production RAG. It handles millions of vectors with HNSW indexing and stays within single-node PostgreSQL limits for most production workloads.\n\nHere is a working RAG pipeline using Chroma and OpenAI embeddings.\n\n**Prerequisites**:\n\n```\npip install langchain langchain-openai langchain-chroma langchain-text-splitters chromadb python-dotenv\n\n1\n\npip install langchain langchain-openai langchain-chroma langchain-text-splitters chromadb python-dotenv\n```\n\nHow to run: Save as **rag_pipeline.py**, add **OPENAI_API_KEY** to your **.env**, then run python **rag_pipeline.py**.\n\n```\n# rag_pipeline.py\n# Minimal RAG pipeline: ingest documents → embed → store in Chroma → retrieve and answer\n# Python 3.10+ | ChromaDB 0.5+ | LangChain 0.3+\n\nimport os\nfrom dotenv import load_dotenv\nfrom langchain_openai import ChatOpenAI, OpenAIEmbeddings\nfrom langchain_chroma import Chroma\nfrom langchain_text_splitters import RecursiveCharacterTextSplitter\nfrom langchain_core.documents import Document\nfrom langchain_core.prompts import ChatPromptTemplate\n\nload_dotenv()\n\n# ── STEP 1: SAMPLE DOCUMENTS ──────────────────────────────────────────────────\n# Replace this list with real documents from your knowledge base.\n# In production, load from PDFs, databases, APIs, or file systems.\ndocuments = [\n    Document(page_content=\"Pinecone is a managed vector database optimized for fast, \"\n             \"low-latency similarity search at scale. It handles infrastructure automatically \"\n             \"and is best for production RAG when you don't want to manage servers.\",\n             metadata={\"source\": \"vector_db_guide\", \"topic\": \"pinecone\"}),\n\n    Document(page_content=\"Weaviate is an open-source vector database with native hybrid search \"\n             \"support, combining BM25 keyword search with dense vector search in a single query. \"\n             \"It can be self-hosted or used via Weaviate Cloud.\",\n             metadata={\"source\": \"vector_db_guide\", \"topic\": \"weaviate\"}),\n\n    Document(page_content=\"Chroma is a developer-friendly, local-first vector database ideal for \"\n             \"prototyping. The 2025 Rust rewrite significantly improved performance. \"\n             \"Best for small-to-medium production workloads and local development.\",\n             metadata={\"source\": \"vector_db_guide\", \"topic\": \"chroma\"}),\n\n    Document(page_content=\"pgvector is a PostgreSQL extension that adds vector similarity search \"\n             \"to an existing Postgres database. With HNSW indexing, it handles millions of vectors \"\n             \"at low latency. Best choice if your team already runs PostgreSQL in production.\",\n             metadata={\"source\": \"vector_db_guide\", \"topic\": \"pgvector\"}),\n]\n\n# ── STEP 2: CHUNK THE DOCUMENTS ───────────────────────────────────────────────\n# Large documents are split into smaller chunks before embedding.\n# chunk_size=500 characters; chunk_overlap=50 preserves context across chunk boundaries.\nsplitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)\nchunks = splitter.split_documents(documents)\n\n# ── STEP 3: EMBED AND STORE IN CHROMA ────────────────────────────────────────\n# OpenAIEmbeddings converts each chunk into a high-dimensional vector.\n# Chroma stores those vectors locally in the ./chroma_db directory.\n# On subsequent runs, the existing store is loaded rather than rebuilt.\nembeddings = OpenAIEmbeddings(\n    model=\"text-embedding-3-small\",   # Fast and cost-effective for most RAG tasks\n    api_key=os.getenv(\"OPENAI_API_KEY\")\n)\n\nvectorstore = Chroma.from_documents(\n    documents=chunks,\n    embedding=embeddings,\n    persist_directory=\"./chroma_db\"   # Persist to disk so you don't re-embed on every run\n)\n\n# ── STEP 4: RETRIEVAL ──────────────────────────────────────────────────────────\n# Converts the query into an embedding and finds the most similar chunks.\n# k=3 returns the top 3 most relevant chunks.\nretriever = vectorstore.as_retriever(search_kwargs={\"k\": 3})\n\n# ── STEP 5: GENERATION ────────────────────────────────────────────────────────\nllm = ChatOpenAI(\n    model=\"gpt-5.5\",\n    temperature=0,\n    api_key=os.getenv(\"OPENAI_API_KEY\")\n)\n\n# The prompt tells the model to use only the retrieved context.\n# This prevents the model from hallucinating facts not in your knowledge base.\nrag_prompt = ChatPromptTemplate.from_messages([\n    (\"system\",\n     \"Answer the question using only the provided context. \"\n     \"If the answer isn't in the context, say so clearly.\\n\\n\"\n     \"Context:\\n{context}\"),\n    (\"human\", \"{question}\")\n])\n\ndef answer(question: str) -> str:\n    \"\"\"Retrieve relevant chunks and generate a grounded answer.\"\"\"\n    # Retrieve the most relevant document chunks for this question\n    retrieved_docs = retriever.invoke(question)\n\n    # Combine the retrieved chunks into a single context block\n    context = \"\\n\\n\".join(doc.page_content for doc in retrieved_docs)\n\n    # Build and invoke the prompt with the question and retrieved context\n    prompt = rag_prompt.invoke({\"context\": context, \"question\": question})\n    response = llm.invoke(prompt)\n    return response.content\n\n# ── DEMO ──────────────────────────────────────────────────────────────────────\nif __name__ == \"__main__\":\n    q = \"Which vector database should I use if I already run PostgreSQL?\"\n    print(f\"Q: {q}\\nA: {answer(q)}\")\n\n123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100\n\n# rag_pipeline.py# Minimal RAG pipeline: ingest documents → embed → store in Chroma → retrieve and answer# Python 3.10+ | ChromaDB 0.5+ | LangChain 0.3+ import osfrom dotenv import load_dotenvfrom langchain_openai import ChatOpenAI, OpenAIEmbeddingsfrom langchain_chroma import Chromafrom langchain_text_splitters import RecursiveCharacterTextSplitterfrom langchain_core.documents import Documentfrom langchain_core.prompts import ChatPromptTemplate load_dotenv() # ── STEP 1: SAMPLE DOCUMENTS ──────────────────────────────────────────────────# Replace this list with real documents from your knowledge base.# In production, load from PDFs, databases, APIs, or file systems.documents = [    Document(page_content=\"Pinecone is a managed vector database optimized for fast, \"             \"low-latency similarity search at scale. It handles infrastructure automatically \"             \"and is best for production RAG when you don't want to manage servers.\",             metadata={\"source\": \"vector_db_guide\", \"topic\": \"pinecone\"}),     Document(page_content=\"Weaviate is an open-source vector database with native hybrid search \"             \"support, combining BM25 keyword search with dense vector search in a single query. \"             \"It can be self-hosted or used via Weaviate Cloud.\",             metadata={\"source\": \"vector_db_guide\", \"topic\": \"weaviate\"}),     Document(page_content=\"Chroma is a developer-friendly, local-first vector database ideal for \"             \"prototyping. The 2025 Rust rewrite significantly improved performance. \"             \"Best for small-to-medium production workloads and local development.\",             metadata={\"source\": \"vector_db_guide\", \"topic\": \"chroma\"}),     Document(page_content=\"pgvector is a PostgreSQL extension that adds vector similarity search \"             \"to an existing Postgres database. With HNSW indexing, it handles millions of vectors \"             \"at low latency. Best choice if your team already runs PostgreSQL in production.\",             metadata={\"source\": \"vector_db_guide\", \"topic\": \"pgvector\"}),] # ── STEP 2: CHUNK THE DOCUMENTS ───────────────────────────────────────────────# Large documents are split into smaller chunks before embedding.# chunk_size=500 characters; chunk_overlap=50 preserves context across chunk boundaries.splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)chunks = splitter.split_documents(documents) # ── STEP 3: EMBED AND STORE IN CHROMA ────────────────────────────────────────# OpenAIEmbeddings converts each chunk into a high-dimensional vector.# Chroma stores those vectors locally in the ./chroma_db directory.# On subsequent runs, the existing store is loaded rather than rebuilt.embeddings = OpenAIEmbeddings(    model=\"text-embedding-3-small\",   # Fast and cost-effective for most RAG tasks    api_key=os.getenv(\"OPENAI_API_KEY\")) vectorstore = Chroma.from_documents(    documents=chunks,    embedding=embeddings,    persist_directory=\"./chroma_db\"   # Persist to disk so you don't re-embed on every run) # ── STEP 4: RETRIEVAL ──────────────────────────────────────────────────────────# Converts the query into an embedding and finds the most similar chunks.# k=3 returns the top 3 most relevant chunks.retriever = vectorstore.as_retriever(search_kwargs={\"k\": 3}) # ── STEP 5: GENERATION ────────────────────────────────────────────────────────llm = ChatOpenAI(    model=\"gpt-5.5\",    temperature=0,    api_key=os.getenv(\"OPENAI_API_KEY\")) # The prompt tells the model to use only the retrieved context.# This prevents the model from hallucinating facts not in your knowledge base.rag_prompt = ChatPromptTemplate.from_messages([    (\"system\",     \"Answer the question using only the provided context. \"     \"If the answer isn't in the context, say so clearly.\\n\\n\"     \"Context:\\n{context}\"),    (\"human\", \"{question}\")]) def answer(question: str) -> str:    \"\"\"Retrieve relevant chunks and generate a grounded answer.\"\"\"    # Retrieve the most relevant document chunks for this question    retrieved_docs = retriever.invoke(question)     # Combine the retrieved chunks into a single context block    context = \"\\n\\n\".join(doc.page_content for doc in retrieved_docs)     # Build and invoke the prompt with the question and retrieved context    prompt = rag_prompt.invoke({\"context\": context, \"question\": question})    response = llm.invoke(prompt)    return response.content  # ── DEMO ──────────────────────────────────────────────────────────────────────if __name__ == \"__main__\":    q = \"Which vector database should I use if I already run PostgreSQL?\"    print(f\"Q: {q}\\nA: {answer(q)}\")\n```\n\n**What this does**: The pipeline has two phases. During indexing, documents are chunked, converted to embeddings via OpenAI’s **text-embedding-3-small** model, and stored in a local Chroma database. During retrieval, the query is embedded using the same model, the three most similar chunks are pulled from Chroma, and the LLM uses those chunks and only those chunks to answer. The **persist_directory** parameter means Chroma saves the vectors to disk, so you do not pay to re-embed your documents on every run.\n\n## Layer 5: Tools and External Integrations\n\nAn agent without tools is a very expensive text predictor. Tools are what give agents the ability to act on the world rather than just talk about it.\n\nIn technical terms, a tool is a function that the model can choose to call. You describe what the function does in natural language, define its input parameters with a schema, and the model decides when calling that function would help it answer the question. The model does not execute the function; your code does. The model just decides when and with what arguments.\n\nThe categories of tools that matter most in production agents are: web search (for current information), code execution (for calculation and data processing), file I/O (for reading and writing documents), API calls (for connecting to external services), and browser use (for interacting with web interfaces that do not have APIs).\n\nOne development worth understanding is the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/), introduced by Anthropic in late 2024. MCP is a standardized way for models to communicate with external tools and data sources. Rather than every team writing custom integration code for every tool, MCP provides a shared protocol. Amazon Bedrock Agents added native MCP support in 2025, and adoption across the ecosystem is growing fast.\n\nThe single most important thing about tool design is the schema. The model decides whether to use a tool based on its description and decides what arguments to pass based on the parameter schema. A vague description produces wrong tool calls. A well-typed schema with clear parameter descriptions produces reliable ones.\n\n**Prerequisites**:\n\n```\npip install langchain langchain-openai langchain-community python-dotenv\n\n1\n\npip install langchain langchain-openai langchain-community python-dotenv\n```\n\n**How to run**: Save as **tools.py**, add **OPENAI_API_KEY** to your **.env**, then run python **tools.py**\n\n```\n# tools.py\n# Defining, registering, and using tools with a LangChain agent\n# Python 3.10+ | LangChain 0.3+\n\nimport os\nimport json\nimport requests\nfrom dotenv import load_dotenv\nfrom langchain_openai import ChatOpenAI\nfrom langchain.tools import tool\nfrom langchain_community.tools import DuckDuckGoSearchRun\nfrom langchain_core.messages import HumanMessage\nfrom langgraph.prebuilt import create_react_agent\n\nload_dotenv()\n\nllm = ChatOpenAI(model=\"gpt-5.5\", temperature=0, api_key=os.getenv(\"OPENAI_API_KEY\"))\n\n# ── TOOL 1: WEB SEARCH ────────────────────────────────────────────────────────\n# Built-in DuckDuckGo tool -- no API key needed.\nsearch = DuckDuckGoSearchRun()\n\n# ── TOOL 2: WEATHER LOOKUP ────────────────────────────────────────────────────\n# The @tool decorator does three things:\n# 1. Registers the function as a callable tool\n# 2. Uses the function name as the tool name\n# 3. Uses the docstring as the tool description (this is what the model reads)\n# The description is critical -- vague descriptions cause wrong tool calls.\n@tool\ndef get_weather(city: str) -> str:\n    \"\"\"\n    Fetch the current weather for a given city.\n    Use this when the user asks about weather conditions, temperature, or forecasts.\n    Input: city name as a string (e.g., 'London', 'Tokyo', 'New York').\n    \"\"\"\n    try:\n        # Using open-meteo (free, no API key) for geocoding and weather\n        geo_url = f\"https://geocoding-api.open-meteo.com/v1/search?name={city}&count=1\"\n        geo = requests.get(geo_url, timeout=5).json()\n\n        if not geo.get(\"results\"):\n            return f\"Could not find location data for '{city}'.\"\n\n        lat = geo[\"results\"][0][\"latitude\"]\n        lon = geo[\"results\"][0][\"longitude\"]\n\n        weather_url = (\n            f\"https://api.open-meteo.com/v1/forecast\"\n            f\"?latitude={lat}&longitude={lon}\"\n            f\"&current_weather=true\"\n        )\n        weather = requests.get(weather_url, timeout=5).json()\n        current = weather.get(\"current_weather\", {})\n\n        return (\n            f\"Weather in {city}: \"\n            f\"{current.get('temperature', 'N/A')}°C, \"\n            f\"wind speed {current.get('windspeed', 'N/A')} km/h.\"\n        )\n    except Exception as e:\n        # Always return a string from tools, even on failure.\n        # Raising exceptions from tools can crash the agent loop.\n        return f\"Weather lookup failed for '{city}': {str(e)}\"\n\n# ── TOOL 3: JSON CALCULATOR ───────────────────────────────────────────────────\n@tool\ndef calculate(expression: str) -> str:\n    \"\"\"\n    Evaluate a mathematical expression and return the result.\n    Use this for arithmetic, percentage calculations, or any numerical computation.\n    Input: a valid Python mathematical expression as a string (e.g., '(150 * 1.08) / 12').\n    Do NOT use for complex code execution -- only simple math expressions.\n    \"\"\"\n    try:\n        # eval is scoped to only allow math -- no builtins, no imports\n        result = eval(expression, {\"__builtins__\": {}}, {})\n        return f\"Result: {result}\"\n    except Exception as e:\n        return f\"Calculation error: {str(e)}\"\n\n# ── REGISTER TOOLS AND BUILD AGENT ────────────────────────────────────────────\ntools = [search, get_weather, calculate]\nagent = create_react_agent(llm, tools)\n\n# ── DEMO ──────────────────────────────────────────────────────────────────────\nif __name__ == \"__main__\":\n    queries = [\n        \"What is the weather in Lagos right now?\",\n        \"If I earn $85,000 a year, what is my monthly gross salary?\",\n        \"Who won the most recent FIFA World Cup?\"\n    ]\n\n    for query in queries:\n        print(f\"\\nQuery: {query}\")\n        result = agent.invoke({\"messages\": [HumanMessage(content=query)]})\n        print(f\"Answer: {result['messages'][-1].content}\")\n\n1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798\n\n# tools.py# Defining, registering, and using tools with a LangChain agent# Python 3.10+ | LangChain 0.3+ import osimport jsonimport requestsfrom dotenv import load_dotenvfrom langchain_openai import ChatOpenAIfrom langchain.tools import toolfrom langchain_community.tools import DuckDuckGoSearchRunfrom langchain_core.messages import HumanMessagefrom langgraph.prebuilt import create_react_agent load_dotenv() llm = ChatOpenAI(model=\"gpt-5.5\", temperature=0, api_key=os.getenv(\"OPENAI_API_KEY\")) # ── TOOL 1: WEB SEARCH ────────────────────────────────────────────────────────# Built-in DuckDuckGo tool -- no API key needed.search = DuckDuckGoSearchRun() # ── TOOL 2: WEATHER LOOKUP ────────────────────────────────────────────────────# The @tool decorator does three things:# 1. Registers the function as a callable tool# 2. Uses the function name as the tool name# 3. Uses the docstring as the tool description (this is what the model reads)# The description is critical -- vague descriptions cause wrong tool calls.@tooldef get_weather(city: str) -> str:    \"\"\"    Fetch the current weather for a given city.    Use this when the user asks about weather conditions, temperature, or forecasts.    Input: city name as a string (e.g., 'London', 'Tokyo', 'New York').    \"\"\"    try:        # Using open-meteo (free, no API key) for geocoding and weather        geo_url = f\"https://geocoding-api.open-meteo.com/v1/search?name={city}&count=1\"        geo = requests.get(geo_url, timeout=5).json()         if not geo.get(\"results\"):            return f\"Could not find location data for '{city}'.\"         lat = geo[\"results\"][0][\"latitude\"]        lon = geo[\"results\"][0][\"longitude\"]         weather_url = (            f\"https://api.open-meteo.com/v1/forecast\"            f\"?latitude={lat}&longitude={lon}\"            f\"&current_weather=true\"        )        weather = requests.get(weather_url, timeout=5).json()        current = weather.get(\"current_weather\", {})         return (            f\"Weather in {city}: \"            f\"{current.get('temperature', 'N/A')}°C, \"            f\"wind speed {current.get('windspeed', 'N/A')} km/h.\"        )    except Exception as e:        # Always return a string from tools, even on failure.        # Raising exceptions from tools can crash the agent loop.        return f\"Weather lookup failed for '{city}': {str(e)}\"  # ── TOOL 3: JSON CALCULATOR ───────────────────────────────────────────────────@tooldef calculate(expression: str) -> str:    \"\"\"    Evaluate a mathematical expression and return the result.    Use this for arithmetic, percentage calculations, or any numerical computation.    Input: a valid Python mathematical expression as a string (e.g., '(150 * 1.08) / 12').    Do NOT use for complex code execution -- only simple math expressions.    \"\"\"    try:        # eval is scoped to only allow math -- no builtins, no imports        result = eval(expression, {\"__builtins__\": {}}, {})        return f\"Result: {result}\"    except Exception as e:        return f\"Calculation error: {str(e)}\"  # ── REGISTER TOOLS AND BUILD AGENT ────────────────────────────────────────────tools = [search, get_weather, calculate]agent = create_react_agent(llm, tools) # ── DEMO ──────────────────────────────────────────────────────────────────────if __name__ == \"__main__\":    queries = [        \"What is the weather in Lagos right now?\",        \"If I earn $85,000 a year, what is my monthly gross salary?\",        \"Who won the most recent FIFA World Cup?\"    ]     for query in queries:        print(f\"\\nQuery: {query}\")        result = agent.invoke({\"messages\": [HumanMessage(content=query)]})        print(f\"Answer: {result['messages'][-1].content}\")\n```\n\n**What this does:** Three tools are registered: a web search tool for current events, a weather tool that calls a free API with no key required, and a calculator that safely evaluates mathematical expressions. The agent receives each query, reasons about which tool to use, calls it, and synthesizes an answer from the result. The key design detail to notice is in the docstrings; each tool description is precise about what the tool does, when to use it, and what format the input should take.\n\n## Layer 6: Observability and Evaluation\n\nHere is a production truth that does not get said enough: LLMs fail silently. As the [team at Kanerika](https://medium.com/@kanerika/llmops-observability-langsmith-vs-arize-vs-langfuse-vs-w-b-f1baeabd1bbf) put it, a hallucinated answer still returns HTTP 200. A standard infrastructure monitoring tool sees a successful request. You see nothing unusual. Meanwhile, your agent has been confidently giving wrong answers for three days.\n\nTraditional monitoring was built for a world where “*correct*” is binary: the function returned the right type, the API returned 200, the query completed in under **100ms**. LLM correctness is semantic. The response can be structurally valid, grammatically fluent, and completely wrong. That requires a different observability layer entirely.\n\nThere are three things a good LLM observability setup tracks. Tracing follows every step of the agent’s execution: the LLM calls, the tool invocations, the retrieval queries, the intermediate reasoning steps, and how long each one took. Evaluation scores the output against metrics that matter: faithfulness (*did it stay grounded in the retrieved context?*), relevance (*did it answer the question asked?*), and hallucination rate. Monitoring tracks behavioral drift over time, whether the agent’s performance on a given class of inputs is getting better or worse as the model and prompts evolve.\n\nThe leading platforms each have a different strength. [LangSmith](https://www.langchain.com/langsmith) provides the deepest integration with LangChain and LangGraph. If you are already in that ecosystem, it is the fastest path to working traces. [Langfuse](https://langfuse.com/) is open-source with over 19,000 GitHub stars and an MIT license, self-hostable, and works with any framework. [Arize Phoenix](https://phoenix.arize.com/) brings ML-grade evaluation rigor and ships with over 50 research-backed metrics covering faithfulness, relevance, safety, and hallucination detection.\n\nAccording to [MLflow’s analysis of observability platforms](https://mlflow.org/top-5-agent-observability-tools/), the right choice often comes down to your framework: LangChain teams get the most from LangSmith, while teams on LlamaIndex or raw API calls are better served by Phoenix or Langfuse.\n\nHere is how to add Langfuse tracing to an existing agent with minimal changes.\n\n**Prerequisites**:\n\n```\npip install langfuse langchain langchain-openai python-dotenv\n\n1\n\npip install langfuse langchain langchain-openai python-dotenv\n```\n\nSign up at [langfuse.com](https://langfuse.com/) for a free account and add **LANGFUSE_PUBLIC_KEY** and **LANGFUSE_SECRET_KEY** to your **.env**. Self-hosting is also available if you prefer to keep data on your own infrastructure.\n\n**How to run**: Save as **observability.py** and run **python observability.py**. Open your Langfuse dashboard to see the trace.\n\n```\n# observability.py\n# Adding Langfuse tracing to a LangChain agent\n# Langfuse captures every LLM call, tool invocation, and token count automatically.\n\nimport os\nfrom dotenv import load_dotenv\nfrom langchain_openai import ChatOpenAI\nfrom langchain_community.tools import DuckDuckGoSearchRun\nfrom langchain_core.messages import HumanMessage\nfrom langgraph.prebuilt import create_react_agent\n\n# Langfuse integrates via the CallbackHandler pattern.\n# It intercepts every LangChain event and sends it to your Langfuse dashboard.\nfrom langfuse.langchain import CallbackHandler\n\nload_dotenv()\n\n# ── LANGFUSE SETUP ─────────────────────────────────────────────────────────────\n# CallbackHandler reads LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY from the environment.\n# session_id groups all related traces into one session -- useful for debugging conversations.\n# user_id ties traces to a specific user for per-user performance analysis.\nlangfuse_handler = CallbackHandler(\n    session_id=\"demo_session_001\",\n    user_id=\"demo_user\"\n)\n\n# ── AGENT SETUP ────────────────────────────────────────────────────────────────\nllm = ChatOpenAI(\n    model=\"gpt-5.5\",\n    temperature=0,\n    api_key=os.getenv(\"OPENAI_API_KEY\"),\n    callbacks=[langfuse_handler]   # Attach the handler here -- this is the only change\n)\n\ntools = [DuckDuckGoSearchRun()]\nagent = create_react_agent(llm, tools)\n\n# ── RUN WITH TRACING ──────────────────────────────────────────────────────────\n# Pass the handler in config so it traces tool calls as well as LLM calls.\n# Without this, only the LLM calls are traced -- tool invocations are invisible.\nresult = agent.invoke(\n    {\"messages\": [HumanMessage(content=\"What is the latest version of Python?\")]},\n    config={\"callbacks\": [langfuse_handler]}\n)\n\nprint(result[\"messages\"][-1].content)\n\n# Flush ensures all traces are sent to Langfuse before the script exits.\n# In a long-running server, this is handled automatically.\nlangfuse_handler.flush()\n\nprint(\"\\nTrace sent to Langfuse. Check your dashboard at https://cloud.langfuse.com\")\n\n12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152\n\n# observability.py# Adding Langfuse tracing to a LangChain agent# Langfuse captures every LLM call, tool invocation, and token count automatically. import osfrom dotenv import load_dotenvfrom langchain_openai import ChatOpenAIfrom langchain_community.tools import DuckDuckGoSearchRunfrom langchain_core.messages import HumanMessagefrom langgraph.prebuilt import create_react_agent # Langfuse integrates via the CallbackHandler pattern.# It intercepts every LangChain event and sends it to your Langfuse dashboard.from langfuse.langchain import CallbackHandler load_dotenv() # ── LANGFUSE SETUP ─────────────────────────────────────────────────────────────# CallbackHandler reads LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY from the environment.# session_id groups all related traces into one session -- useful for debugging conversations.# user_id ties traces to a specific user for per-user performance analysis.langfuse_handler = CallbackHandler(    session_id=\"demo_session_001\",    user_id=\"demo_user\") # ── AGENT SETUP ────────────────────────────────────────────────────────────────llm = ChatOpenAI(    model=\"gpt-5.5\",    temperature=0,    api_key=os.getenv(\"OPENAI_API_KEY\"),    callbacks=[langfuse_handler]   # Attach the handler here -- this is the only change) tools = [DuckDuckGoSearchRun()]agent = create_react_agent(llm, tools) # ── RUN WITH TRACING ──────────────────────────────────────────────────────────# Pass the handler in config so it traces tool calls as well as LLM calls.# Without this, only the LLM calls are traced -- tool invocations are invisible.result = agent.invoke(    {\"messages\": [HumanMessage(content=\"What is the latest version of Python?\")]},    config={\"callbacks\": [langfuse_handler]}) print(result[\"messages\"][-1].content) # Flush ensures all traces are sent to Langfuse before the script exits.# In a long-running server, this is handled automatically.langfuse_handler.flush() print(\"\\nTrace sent to Langfuse. Check your dashboard at https://cloud.langfuse.com\")\n```\n\n**What this does**: Two changes from a standard agent setup: the **CallbackHandler** is initialized with a session and user ID, and it is attached to both the LLM and the **agent.invoke** config. That is enough for Langfuse to capture the full trace of every LLM call, every tool invocation, token counts, latency, and the complete input/output at each step. Everything you need to debug a production failure or track quality drift over time.\n\n## Layer 7: Deployment Infrastructure\n\nYou can have a flawless agent in development that turns into a maintenance problem in production. The infrastructure layer is where that gap lives.\n\nAt a minimum, your agent should be containerized with [Docker](https://www.docker.com/). Containers give you consistent behavior across environments, straightforward dependency management, and a clean path to any cloud deployment target. The alternative — shipping Python scripts with a **requirements.txt** and hoping the environment matches — creates a class of bugs that wastes engineering time disproportionate to the effort containerization would have taken.\n\nFor most production agents, you have two architectural options for the serving layer: a synchronous API or an async queue. A synchronous API (Flask or FastAPI) works when your agent completes in under a few seconds, and you can afford to hold the HTTP connection open.\n\nWhen your agent involves multiple tool calls, long retrieval pipelines, or document processing that might take 30 to 60 seconds, an async queue (Celery, AWS SQS, or Google Pub/Sub) is the better choice. The client submits a job, gets a task ID back immediately, and polls for the result.\n\nOn the cloud side, all three major platforms now have managed agent infrastructure. [Amazon’s AgentCore](https://aws.amazon.com/bedrock/agentcore/), which became generally available in October 2025, provides dedicated agentic infrastructure on AWS for memory management, tool execution, and session handling without provisioning servers. [Google Vertex AI Agent Builder](https://cloud.google.com/products/agent-builder) is the natural choice for teams already in the GCP ecosystem, with native Gemini integration and built-in observability. [Azure OpenAI Service](https://azure.microsoft.com/en-us/products/ai-services/openai-service) with Semantic Kernel is the enterprise default for Microsoft shops.\n\nFor cost management, three practices make the biggest difference: caching (returning stored responses for repeated identical queries rather than calling the model again), request batching (grouping non-urgent tasks to reduce per-call overhead), and setting **max_iterations** in your agent executor to prevent runaway loops from consuming tokens without bound.\n\n## Putting It All Together\n\nThe right choices at each layer depend on where you are in the project lifecycle. Here is a practical reference that reflects the research and trade-offs discussed above.\n\n**Prototype (move fast, minimal infrastructure):**\n\nLayer |\nChoice |\nReason |\n|---|---|---|\n| Foundation Model | GPT-5.5 | Reliable tool-calling, mature ecosystem |\n| Orchestration | LangGraph | Fast setup, good documentation |\n| Memory | In-context only | No infrastructure needed |\n| Vector DB | Chroma | Local, no ops, good developer experience |\n| Tools | DuckDuckGo + custom @tool functions | Zero API keys required |\n| Observability | Langfuse (cloud free tier) | One-line setup |\n| Deployment | Local / Docker | Ship fast |\n\n**Production Startup (scale with control):**\n\nLayer |\nChoice |\nReason |\n|---|---|---|\n| Foundation Model | GPT-5.5 + Claude Sonnet 4.6 fallback | Reliability with redundancy |\n| Orchestration | LangGraph or CrewAI | State management and multi-agent support |\n| Memory | Episodic (Postgres) + Semantic (RAG) | Full persistent context |\n| Vector DB | Weaviate or Pinecone | Scale and hybrid search |\n| Tools | Full tool suite with MCP | Standardized integrations |\n| Observability | Langfuse self-hosted or Arize Phoenix | Data control + ML-grade evals |\n| Deployment | Docker + Kubernetes + async queue | Production-grade, cost-controlled |\n\n**Enterprise:**\n\nLayer |\nChoice |\nReason |\n|---|---|---|\n| Foundation Model | Azure OpenAI or AWS Bedrock | Compliance, data residency, SLA |\n| Orchestration | Semantic Kernel or LangGraph | Enterprise language support, governance |\n| Memory | Managed memory with audit trail | Regulatory requirements |\n| Vector DB | Weaviate or pgvector | Self-hostable, compliance-ready |\n| Tools | MCP-based, internally approved | Security review and access control |\n| Observability | Langfuse self-hosted or Datadog LLM module | Existing infrastructure integration |\n| Deployment | AWS AgentCore / Vertex AI Agent Builder | Fully managed, governed, auditable |\n\n## Conclusion\n\nThe foundation model is the part of this stack that gets written about. The other six layers are the parts that determine whether what you built actually works in production.\n\nAn agent fails at the orchestration layer when the ReAct loop gets stuck. It fails at the memory layer when it forgets the context it needs. It fails at the retrieval layer when the wrong chunks are returned, and the model hallucinates a grounded-sounding answer. It fails at the tools layer when a schema is too vague, and the model calls the wrong function. It fails at the observability layer when you have no way to know that any of this is happening. And it fails at the deployment layer when the infrastructure cannot handle the latency or cost requirements of real traffic.\n\nGartner estimates that over [40% of agentic AI projects are at risk of cancellation by 2027](https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025) due to unclear value, rising costs, and weak governance. Most of those failures will trace back not to a bad model choice but to a stack that was built layer by layer without a clear picture of how the layers connect.\n\nUnderstanding the full stack does not mean you have to build all of it. It means you know what decisions you are making and what you are trading off when you make them. That is the difference between an agent that works in a demo and one that ships.", "url": "https://wpnews.pro/news/the-ai-agent-tech-stack-explained", "canonical_source": "https://machinelearningmastery.com/the-ai-agent-tech-stack-explained/", "published_at": "2026-06-26 14:01:03+00:00", "updated_at": "2026-07-07 18:02:21.606141+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-infrastructure", "ai-tools", "ai-research"], "entities": ["OpenAI", "Anthropic", "Google", "Meta", "Mistral", "Gartner", "GPT-5.5", "Claude Sonnet 4.6"], "alternates": {"html": "https://wpnews.pro/news/the-ai-agent-tech-stack-explained", "markdown": "https://wpnews.pro/news/the-ai-agent-tech-stack-explained.md", "text": "https://wpnews.pro/news/the-ai-agent-tech-stack-explained.txt", "jsonld": "https://wpnews.pro/news/the-ai-agent-tech-stack-explained.jsonld"}}