Demystifying AI Agents: Building an Agentic Pipeline From Scratch in Pure Python

To build an AI agent pipeline from scratch using pure Python, stripping away the abstractions of popular frameworks like LangChain and CrewAI. It describes the core mechanics of an agent as a continuous execution cycle of thinking, acting, and observing, rather than a single LLM transaction. The tutorial provides a modular code structure with components like a configuration file, HTTP client, memory manager, and orchestration engine to demonstrate the underlying runtime architecture.

Most AI demos look impressive until you ask a simple question: What is actually happening under the hood? Frameworks like LangChain, CrewAI, and Microsoft AutoGen make it incredibly easy to spin up an “AI agent” in a few lines of code. But abstractions come with a cost. Many developers can build agents using frameworks without fully understanding the runtime architecture behind them. But what happens when something goes wrong and, because of the layers of abstraction, you don’t know exactly how to fix it? Or when you can’t even use those libraries and frameworks in your workplace and need to build an agentic application from scratch… At their core, most agent frameworks are built around surprisingly simple primitives: - Prompt orchestration - Stateful memory - Tool execution - Control loops - Structured outputs This week I was talking with a friend who wanted to understand how AI agents actually work under the hood. During that conversation, I realized something: most tutorials make AI agents feel far more mysterious than they really are. Frameworks are great for moving fast, but they also hide many of the core mechanics behind layers of abstractions. You import a library, initialize an “agent,” attach a tool, and suddenly everything looks autonomous and intelligent. But underneath those abstractions, most agent systems are built on a surprisingly small set of concepts: - Prompts - Memory - Tool execution - Structured outputs - Control loops So I decided to write the article I wish I had found when I first started exploring agentic systems. No heavy frameworks. No orchestration libraries. No hidden runtime magic. Just the core ideas, built step-by-step from scratch in pure Python. In this article, we will strip away the abstractions and build a production-inspired agentic pipeline entirely from scratch using: - Pure Python - The standard library only - Native HTTP requests - No SDKs - No orchestration frameworks By the end, you will understand the core mechanics behind modern AI agents and why most frameworks are essentially layered convenience abstractions over a deterministic execution loop. What Is an Agentic Pipeline? A standard LLM interaction is usually a single-shot transaction: User Prompt ── Model Response The model receives context once and generates a static response. An agent, however, behaves differently. Instead of generating a single response, it operates inside a continuous execution cycle: ┌───────────────────────────────────────┐ │ │ ▼ │ THINK ─── Decision ─── ACT ─── OBSERVE Tool Call Tool Result Think The model evaluates the user objective, available tools, prior observations, and current memory state. It then decides what to do next. Act The agent executes an action. This could be calling a function, querying a database, searching the web, reading files, or returning a final answer. Observe The system captures the result of the action and feeds it back into the context window. The cycle repeats until the objective is complete. A Helpful Mental Model Think of an agent like a developer debugging a production issue: Observe error logs │ ▼ Form a hypothesis │ ▼ Run a command │ ▼ Inspect output │ ▼ Repeat That iterative feedback loop is exactly how agentic systems operate. Project Structure We will organize the codebase into small, focused modules. agentic-pipeline/ ├── config.json Runtime configuration ├── llm client.py Low-level HTTP client ├── memory.py Context/state manager ├── agent.py Agent orchestration engine └── main.py Runtime execution loop This separation mirrors how production systems are commonly structured. Step 1 — Configuration Management Avoid hardcoding runtime variables directly in code. For this demo we’re going to Create a config.json file just for demonstration purposes: { "llm": { "provider": "openai", "model": "gpt-4o", "api key": "sk-your-api-key", "temperature": 0.2, "max tokens": 1024 } } ⚠️ Note:In production systems, credentials should come from environment variables or a secrets manager rather than static configuration files. Step 2 — Building the Infrastructure Layer Most SDKs hide the reality that every LLM interaction is just an HTTP request. Underneath the abstraction, the process is straightforward: Serialize payload ── Send HTTPS POST request ── Receive JSON response ── Parse output Let’s implement that manually in llm client.py . python import json import urllib.request import urllib.error from typing import Dict, List class LLMClient: def init self, config: Dict : self.config = config "llm" self.api key = self.config "api key" def chat completion self, messages: List Dict , temperature: float = None - str: payload = { "model": self.config "model" , "messages": messages, "temperature": temperature or self.config.get "temperature", 0.2 , "max tokens": self.config.get "max tokens", 1024 } data = json.dumps payload .encode "utf-8" req = urllib.request.Request "https://api.openai.com/v1/chat/completions", data=data, method="POST" req.add header "Content-Type", "application/json" req.add header "Authorization", f"Bearer {self.api key}" try: with urllib.request.urlopen req as response: result = json.loads response.read .decode return result "choices" 0 "message" "content" .strip except urllib.error.HTTPError as e: error body = e.read .decode raise Exception f"LLM API error: {e.code} - {error body}" To understand what LLMClient is doing here, it helps to think of it like an old-school telegraph operator. This layer has no concept of reasoning, planning, or executing tools. It doesn't even manage memory. Its only job is to package up a stack of text, send it down the wire to the model, and hand you back the raw response. It moves the messages back and forth reliably without needing to understand a single word written inside them. Step 3 — Managing Agent Memory LLMs are stateless. They do not remember previous interactions unless the entire history is resent with every request. As the execution loop progresses, the context window continuously grows. We therefore need a lightweight memory manager in memory.py . python from typing import List, Dict class AgentMemory: def init self, max messages: int = 20 : self.messages: List Dict = self.max messages = max messages def add self, role: str, content: str : self.messages.append { "role": role, "content": content } if len self.messages self.max messages: Preserve system prompt system prompt = self.messages 0 Slide conversation window active history = self.messages 1: self.messages = system prompt + active history - self.max messages - 1 : def get messages self - List Dict : return self.messages.copy def clear self : self.messages.clear If the LLM client is our telegraph operator, you can picture this memory manager like a detective's notebook. As the agent investigates a task, every tiny detail gets written down: the original user request, internal reasoning, tool choices, and the clues discovered along the way. Because the notebook can't hold infinite pages, the detective eventually has to archive old details while keeping the core investigation context front and center. That sliding window logic is exactly how we keep the context manageable. Step 4 — Building the Agent Engine This is where the orchestration logic lives. The agent must understand available tools, decide when to use them, parse structured outputs, execute functions, and feed observations back into memory. Let's write agent.py : python from llm client import LLMClient from memory import AgentMemory from typing import Dict, Callable import json class Agent: def init self, system prompt: str, config path: str = "config.json" : with open config path as f: self.config = json.load f self.llm = LLMClient self.config self.memory = AgentMemory self.system prompt = system prompt self.tools: Dict str, dict = {} self.memory.add "system", system prompt def register tool self, name: str, func: Callable, description: str : self.tools name = { "func": func, "description": description } def get tool descriptions self - str: if not self.tools: return "No tools available." return "\n".join f"- {name}: {info 'description' }" for name, info in self.tools.items def think self, user input: str - str: self.memory.add "user", user input messages = self.memory.get messages tool info = self. get tool descriptions if self.tools: messages = messages.copy enhanced content = f"{user input}\n\n" f"AVAILABLE TOOLS:\n" f"{tool info}\n\n" f"If you need a tool, respond ONLY with JSON:\n" f'{{"tool":"tool name","args":{{}}}}\n\n' f"If the task is complete, respond naturally and include 'FINAL ANSWER'." messages -1 "content" = enhanced content response = self.llm.chat completion messages self.memory.add "assistant", response return response def act self, response: str : if "{" in response and "}" in response: try: start = response.find "{" end = response.rfind "}" + 1 tool json = json.loads response start:end tool name = tool json.get "tool" args = tool json.get "args", {} if tool name in self.tools: result = self.tools tool name .get “func” args self.memory.add "system", f"Observation from '{tool name}': {result}" return result except Exception as e: error msg = f"Tool execution failed: {str e }" self.memory.add "system", error msg return error msg return None This structural handoff brings up one of the most misunderstood parts of modern AI agents: the model does not execute your Python functions directly. Instead, you are providing plain text descriptions of your local code inside the prompt. When the model reads these descriptions and decides it needs help, it simply formats its text output into a raw JSON block specifying a tool name and parameters. Your host application then catches that JSON, reads it, runs the native Python code locally, and passes the results back into the text history. The LLM itself remains entirely isolated, your local application serves as the actual execution environment. Step 5 — The Runtime Control Loop Without a runtime loop, the agent cannot perform multi-step reasoning. The host application must continuously drive execution forward. Let's look at main.py : php from agent import Agent import time def web search query: str - str: print f"🔍 Searching index for: '{query}'" time.sleep 1 if "agentic ai" in query.lower : return "Found: Modern agentic systems are moving away from rigid chains " "toward lightweight control loops and modular tools." return "Found: Building agents from scratch reveals implementation details " "often hidden by frameworks." if name == " main ": system prompt = "You are an autonomous operations assistant. " "Reason step-by-step. " "Use tools when necessary. " "When the task is fully complete, include the phrase FINAL ANSWER." agent = Agent system prompt agent.register tool name="search", func=web search, description="Queries an index database. Input schema: {'query': str}" task = "Research trends in agentic AI and explain why building from scratch is valuable." print f"🎯 Objective: {task}" max steps = 5 for step in range max steps : print f"\n Cycle {step + 1} " prompt = task if step == 0 else "Analyze previous observations and continue." response = agent.think prompt print f"\n🤖 Agent:\n{response}" tool output = agent.act response if tool output: print f"\n🛠 Observation:\n{tool output}" if "final answer" in response.lower : print "\n✅ Objective completed." break Tracing the Runtime Execution Here is a look at what happens internally during execution over two separate cycles: Cycle 1 - Think: The model receives the task, tool descriptions, and the initial system memory state. It realizes it lacks direct information about current trends. - Act: The model emits structured JSON: { "tool": "search", "args": { "query": "latest trends in agentic AI" } } The runtime parses this block and executes the local Python web search function. - Observe: The tool output gets appended back into memory. The model now has additional context to continue reasoning. Cycle 2 The model reviews the original objective, prior observations, and tool outputs. It synthesizes a complete response and emits: plaintext FINAL ANSWER The control loop detects this completion keyword and exits gracefully. What You Actually Built Underneath all the abstractions, you implemented a fully working pipeline: - Stateful memory - Tool registration - Structured tool calling - Runtime orchestration - Multi-step execution - Context management - Deterministic control flow That is the foundation of nearly every modern agent framework. Production Considerations This implementation is intentionally minimal. Real production systems typically add: | Domain | Operational Mechanics | |---|---| Resilience & Tracking | Retry policies, Token accounting, Observability & tracing | Data & Run Management | Parallel tool execution, Sandboxed runtimes, Rate limiting | Architecture Scaling | Distributed orchestration, Long-term memory persistence layers | Security & Safety | Guardrails and validation, Human approval checkpoints | Frameworks become valuable once these operational concerns grow large enough. But understanding the core loop first changes how you design AI systems. Final Thoughts AI agents can appear magical when hidden behind high-level abstractions. But once you strip away the layers, most systems reduce to a small set of deterministic building blocks: prompts, memory, tools, parsing, and loops. Understanding those primitives gives you far more architectural control than blindly composing frameworks. Before introducing another dependency into your stack, it is worth asking: “Do I actually need a framework here, or do I just need a well-designed control loop?” If you can answer that question confidently, you already understand more about agentic systems than most developers using them today.