Most AI demos look impressive until you ask a simple question: What is actually happening under the hood?
Frameworks like LangChain, CrewAI, and Microsoft AutoGen make it incredibly easy to spin up an “AI agent” in a few lines of code. But abstractions come with a cost.
Many developers can build agents using frameworks without fully understanding the runtime architecture behind them. But what happens when something goes wrong and, because of the layers of abstraction, you don’t know exactly how to fix it? Or when you can’t even use those libraries and frameworks in your workplace and need to build an agentic application from scratch…
At their core, most agent frameworks are built around surprisingly simple primitives:
- Prompt orchestration
- Stateful memory
- Tool execution
- Control loops
- Structured outputs
This week I was talking with a friend who wanted to understand how AI agents actually work under the hood. During that conversation, I realized something: most tutorials make AI agents feel far more mysterious than they really are.
Frameworks are great for moving fast, but they also hide many of the core mechanics behind layers of abstractions. You import a library, initialize an “agent,” attach a tool, and suddenly everything looks autonomous and intelligent. But underneath those abstractions, most agent systems are built on a surprisingly small set of concepts:
- Prompts
- Memory
- Tool execution
- Structured outputs
- Control loops
So I decided to write the article I wish I had found when I first started exploring agentic systems. No heavy frameworks. No orchestration libraries. No hidden runtime magic. Just the core ideas, built step-by-step from scratch in pure Python.
In this article, we will strip away the abstractions and build a production-inspired agentic pipeline entirely from scratch using:
- Pure Python
- The standard library only
- Native HTTP requests
- No SDKs
- No orchestration frameworks
By the end, you will understand the core mechanics behind modern AI agents and why most frameworks are essentially layered convenience abstractions over a deterministic execution loop.
What Is an Agentic Pipeline? #
A standard LLM interaction is usually a single-shot transaction:
User Prompt ──> Model Response
The model receives context once and generates a static response. An agent, however, behaves differently. Instead of generating a single response, it operates inside a continuous execution cycle:
┌───────────────────────────────────────┐
│ │
▼ │
[ THINK ] ───> (Decision) ───> [ ACT ] ───> [ OBSERVE ]
(Tool Call) (Tool Result)
Think
The model evaluates the user objective, available tools, prior observations, and current memory state. It then decides what to do next.
Act
The agent executes an action. This could be calling a function, querying a database, searching the web, reading files, or returning a final answer.
Observe
The system captures the result of the action and feeds it back into the context window. The cycle repeats until the objective is complete.
A Helpful Mental Model #
Think of an agent like a developer debugging a production issue:
Observe error logs
│
▼
Form a hypothesis
│
▼
Run a command
│
▼
Inspect output
│
▼
Repeat
That iterative feedback loop is exactly how agentic systems operate.
Project Structure #
We will organize the codebase into small, focused modules.
agentic-pipeline/
├── config.json # Runtime configuration
├── llm_client.py # Low-level HTTP client
├── memory.py # Context/state manager
├── agent.py # Agent orchestration engine
└── main.py # Runtime execution loop
This separation mirrors how production systems are commonly structured.
Step 1 — Configuration Management #
Avoid hardcoding runtime variables directly in code. For this demo we’re going to Create a config.json
file just for demonstration purposes:
{
"llm": {
"provider": "openai",
"model": "gpt-4o",
"api_key": "sk-your-api-key",
"temperature": 0.2,
"max_tokens": 1024
}
}
⚠️
Note:In production systems, credentials should come from environment variables or a secrets manager rather than static configuration files.
Step 2 — Building the Infrastructure Layer #
Most SDKs hide the reality that every LLM interaction is just an HTTP request. Underneath the abstraction, the process is straightforward:
Serialize payload ──> Send HTTPS POST request ──> Receive JSON response ──> Parse output
Let’s implement that manually in llm_client.py
.
import json
import urllib.request
import urllib.error
from typing import Dict, List
class LLMClient:
def __init__(self, config: Dict):
self.config = config["llm"]
self.api_key = self.config["api_key"]
def chat_completion(
self,
messages: List[Dict],
temperature: float = None
) -> str:
payload = {
"model": self.config["model"],
"messages": messages,
"temperature": temperature or self.config.get("temperature", 0.2),
"max_tokens": self.config.get("max_tokens", 1024)
}
data = json.dumps(payload).encode("utf-8")
req = urllib.request.Request(
"https://api.openai.com/v1/chat/completions",
data=data,
method="POST"
)
req.add_header("Content-Type", "application/json")
req.add_header("Authorization", f"Bearer {self.api_key}")
try:
with urllib.request.urlopen(req) as response:
result = json.loads(response.read().decode())
return result["choices"][0]["message"]["content"].strip()
except urllib.error.HTTPError as e:
error_body = e.read().decode()
raise Exception(f"LLM API error: {e.code} - {error_body}")
To understand what LLMClient
is doing here, it helps to think of it like an old-school telegraph operator. This layer has no concept of reasoning, planning, or executing tools. It doesn't even manage memory. Its only job is to package up a stack of text, send it down the wire to the model, and hand you back the raw response. It moves the messages back and forth reliably without needing to understand a single word written inside them.
Step 3 — Managing Agent Memory #
LLMs are stateless. They do not remember previous interactions unless the entire history is resent with every request. As the execution loop progresses, the context window continuously grows. We therefore need a lightweight memory manager in memory.py
.
from typing import List, Dict
class AgentMemory:
def __init__(self, max_messages: int = 20):
self.messages: List[Dict] = []
self.max_messages = max_messages
def add(self, role: str, content: str):
self.messages.append({
"role": role,
"content": content
})
if len(self.messages) > self.max_messages:
system_prompt = self.messages[0]
active_history = self.messages[1:]
self.messages = (
[system_prompt] +
active_history[-(self.max_messages - 1):]
)
def get_messages(self) -> List[Dict]:
return self.messages.copy()
def clear(self):
self.messages.clear()
If the LLM client is our telegraph operator, you can picture this memory manager like a detective's notebook. As the agent investigates a task, every tiny detail gets written down: the original user request, internal reasoning, tool choices, and the clues discovered along the way. Because the notebook can't hold infinite pages, the detective eventually has to archive old details while keeping the core investigation context front and center. That sliding window logic is exactly how we keep the context manageable.
Step 4 — Building the Agent Engine #
This is where the orchestration logic lives. The agent must understand available tools, decide when to use them, parse structured outputs, execute functions, and feed observations back into memory. Let's write agent.py
:
from llm_client import LLMClient
from memory import AgentMemory
from typing import Dict, Callable
import json
class Agent:
def __init__(self, system_prompt: str, config_path: str = "config.json"):
with open(config_path) as f:
self.config = json.load(f)
self.llm = LLMClient(self.config)
self.memory = AgentMemory()
self.system_prompt = system_prompt
self.tools: Dict[str, dict] = {}
self.memory.add("system", system_prompt)
def register_tool(self, name: str, func: Callable, description: str):
self.tools[name] = {
"func": func,
"description": description
}
def _get_tool_descriptions(self) -> str:
if not self.tools:
return "No tools available."
return "\n".join([
f"- {name}: {info['description']}"
for name, info in self.tools.items()
])
def think(self, user_input: str) -> str:
self.memory.add("user", user_input)
messages = self.memory.get_messages()
tool_info = self._get_tool_descriptions()
if self.tools:
messages = messages.copy()
enhanced_content = (
f"{user_input}\n\n"
f"AVAILABLE TOOLS:\n"
f"{tool_info}\n\n"
f"If you need a tool, respond ONLY with JSON:\n"
f'{{"tool":"tool_name","args":{{}}}}\n\n'
f"If the task is complete, respond naturally and include 'FINAL ANSWER'."
)
messages[-1]["content"] = enhanced_content
response = self.llm.chat_completion(messages)
self.memory.add("assistant", response)
return response
def act(self, response: str):
if "{" in response and "}" in response:
try:
start = response.find("{")
end = response.rfind("}") + 1
tool_json = json.loads(response[start:end])
tool_name = tool_json.get("tool")
args = tool_json.get("args", {})
if tool_name in self.tools:
result = self.tools[tool_name].get(“func”)(**args)
self.memory.add(
"system",
f"Observation from '{tool_name}': {result}"
)
return result
except Exception as e:
error_msg = f"Tool execution failed: {str(e)}"
self.memory.add("system", error_msg)
return error_msg
return None
This structural handoff brings up one of the most misunderstood parts of modern AI agents: the model does not execute your Python functions directly.
Instead, you are providing plain text descriptions of your local code inside the prompt. When the model reads these descriptions and decides it needs help, it simply formats its text output into a raw JSON block specifying a tool name and parameters. Your host application then catches that JSON, reads it, runs the native Python code locally, and passes the results back into the text history. The LLM itself remains entirely isolated, your local application serves as the actual execution environment.
Step 5 — The Runtime Control Loop #
Without a runtime loop, the agent cannot perform multi-step reasoning. The host application must continuously drive execution forward. Let's look at main.py
:
from agent import Agent
import time
def web_search(query: str) -> str:
print(f"🔍 Searching index for: '{query}'")
time.sleep(1)
if "agentic ai" in query.lower():
return (
"Found: Modern agentic systems are moving away from rigid chains "
"toward lightweight control loops and modular tools."
)
return (
"Found: Building agents from scratch reveals implementation details "
"often hidden by frameworks."
)
if __name__ == "__main__":
system_prompt = (
"You are an autonomous operations assistant. "
"Reason step-by-step. "
"Use tools when necessary. "
"When the task is fully complete, include the phrase FINAL ANSWER."
)
agent = Agent(system_prompt)
agent.register_tool(
name="search",
func=web_search,
description="Queries an index database. Input schema: {'query': str}"
)
task = "Research trends in agentic AI and explain why building from scratch is valuable."
print(f"🎯 Objective: {task}")
max_steps = 5
for step in range(max_steps):
print(f"\n[Cycle {step + 1}]")
prompt = task if step == 0 else "Analyze previous observations and continue."
response = agent.think(prompt)
print(f"\n🤖 Agent:\n{response}")
tool_output = agent.act(response)
if tool_output:
print(f"\n🛠 Observation:\n{tool_output}")
if "final answer" in response.lower():
print("\n✅ Objective completed.")
break
Tracing the Runtime Execution #
Here is a look at what happens internally during execution over two separate cycles:
Cycle 1
Think: The model receives the task, tool descriptions, and the initial system memory state. It realizes it lacks direct information about current trends. - Act: The model emits structured JSON:
{
"tool": "search",
"args": {
"query": "latest trends in agentic AI"
}
}
The runtime parses this block and executes the local Python web_search
function.
Observe: The tool output gets appended back into memory. The model now has additional context to continue reasoning.
Cycle 2
The model reviews the original objective, prior observations, and tool outputs. It synthesizes a complete response and emits:
plaintext
FINAL ANSWER
The control loop detects this completion keyword and exits gracefully.
What You Actually Built #
Underneath all the abstractions, you implemented a fully working pipeline:
- Stateful memory
- Tool registration
- Structured tool calling
- Runtime orchestration
- Multi-step execution
- Context management
- Deterministic control flow
That is the foundation of nearly every modern agent framework.
Production Considerations #
This implementation is intentionally minimal. Real production systems typically add:
| Domain | Operational Mechanics |
|---|---|
| Resilience & Tracking | |
| Retry policies, Token accounting, Observability & tracing | |
| Data & Run Management | |
| Parallel tool execution, Sandboxed runtimes, Rate limiting | |
| Architecture Scaling | |
| Distributed orchestration, Long-term memory persistence layers | |
| Security & Safety | |
| Guardrails and validation, Human approval checkpoints |
Frameworks become valuable once these operational concerns grow large enough. But understanding the core loop first changes how you design AI systems.
Final Thoughts #
AI agents can appear magical when hidden behind high-level abstractions. But once you strip away the layers, most systems reduce to a small set of deterministic building blocks: prompts, memory, tools, parsing, and loops.
Understanding those primitives gives you far more architectural control than blindly composing frameworks. Before introducing another dependency into your stack, it is worth asking:
“Do I actually need a framework here, or do I just need a well-designed control loop?”
If you can answer that question confidently, you already understand more about agentic systems than most developers using them today.