From Attention to Agency: The Progressive Evolution of AI-Assisted Programming

A developer's field guide traces the progressive evolution of AI-assisted programming, from the Transformer architecture to agentic AI, explaining how each layer emerged from the limitations of the previous one. The guide covers key concepts such as pre-training, post-training, ChatGPT, and agentic architectures, emphasizing the causal chain that drives the AI stack.

A Developer's Field Guide to the AI Stack in 2026 If you've been following the AI tooling space over the past few years, you've probably noticed something strange: the concepts arrive in waves, but they don't arrive randomly. Transformer architectures gave us the foundation. Pre-training gave us raw capability. Post-training gave us alignment. ChatGPT gave us a product. And then, almost inevitably, we got agents. But here's what most articles miss — each layer of this stack didn't just appear . It emerged because the layer below it hit a wall. And understanding that chain of causation is what separates developers who use AI tools from developers who understand them. This article traces that chain: from the math that started it all, through the engineering choices that made it practical, to the agentic architectures that are reshaping what "programming" even means. We'll cover every concept in your toolkit — Transformer, Pre-training, Post-training, ChatGPT, React, Agentic AI, AI Agents, Harness, Tools, Hooks, Permission, Skills, Compact, Memory, Sub-agents, and MCP — and show how they fit into a coherent progressive evolution. In 2017, a team at Google published a paper with a provocatively simple title: "Attention Is All You Need" 1 . The paper introduced the Transformer architecture, which replaced recurrent neural networks RNNs with a mechanism called Self-Attention . RNNs processed sequences step by step — reading token 1, then token 2, then token 3. This made them painfully slow for long sequences and caused them to "forget" early tokens by the time they reached the end. Self-Attention solves this differently. Instead of processing sequentially, it lets every token in a sequence "look at" every other token simultaneously and compute a weighted relevance score. This is the attention mechanism — the model learns which parts of the input matter most relative to each other, regardless of distance. The key insight: parallelization . Transformers can process entire sequences at once, making them dramatically faster to train and capable of capturing long-range dependencies that RNNs struggled with. A Transformer consists of: Modern LLMs like GPT-4, Claude, and Llama are mostly decoder-only Transformers following the GPT architecture from OpenAI's 2018 paper . The encoder is often omitted because generation tasks writing code, answering questions are inherently sequential. What made Transformers truly revolutionary wasn't just the architecture — it was what happened when you scaled them. The 2020 scaling laws paper from OpenAI 2 showed that model performance follows a smooth power law with respect to compute, data, and parameters. More of everything meant better results, predictably. This is the engine that drove everything that followed: the race to larger models, the emergence of emergent capabilities, and ultimately, the ability to write coherent code. Key takeaway: The Transformer is the engine. Attention is the fuel. Without this foundation, none of the rest of this stack exists. Once you have a Transformer, you need to teach it something. That's what pre-training does. The core pre-training objective is deceptively simple: predict the next token . Feed the model a sequence of text, mask the last token, and ask it to predict what comes next. Repeat on billions of tokens. This sounds trivial, but it's extraordinarily powerful. To predict the next token well, the model must implicitly learn: Pre-training data is typically a massive corpus mixing: The quality and diversity of this data directly determines what the model can do. Code-specific pre-training training on large amounts of programming code is a major reason why models like GPT-4, Claude, and Codex developed strong coding abilities. Here's the critical point: a pre-trained model is not yet useful . It can predict text, but it doesn't know how to behave . It might complete a sentence rudely, helpfully, incorrectly, or dangerously — with equal probability. The model has "knowledge" but no "judgment." This is why pre-training alone isn't enough, and it's exactly why post-training exists. Key takeaway: Pre-training gives the model knowledge and capability. It's a powerful but undirected force — like having a vast encyclopedia memorized but no common sense about when to share what. Post-training is the umbrella term for the techniques that take a raw pre-trained model and make it actually useful and safe. This is where the magic really happens. The first step is often Supervised Fine-Tuning — training the model on high-quality examples of desired behavior. Human annotators write or curate prompt-response pairs that demonstrate good answers. For code tasks, this might mean: SFT teaches the model what good responses look like in specific domains. It's relatively simple and efficient, but it has a ceiling: the model can only be as good as the examples it sees. To push past that ceiling, most modern models use Reinforcement Learning from Human Feedback RLHF . The process: This is what makes models like ChatGPT feel natural and helpful — they learned not just what to say, but how to say it in ways that humans prefer. More recent approaches like DPO Direct Preference Optimization simplify this by directly optimizing against preference data without needing a separate reward model. Code models undergo specialized post-training: The result is a model that doesn't just complete code — it engages with code tasks the way a thoughtful developer would. Key takeaway: Pre-training is education. Post-training is etiquette training. The model knows things; post-training teaches it when and how to share that knowledge appropriately. Everything before this point was invisible infrastructure. Then OpenAI shipped ChatGPT in November 2022, and AI became visible. ChatGPT didn't introduce new technology — it packaged existing technology into a product that was: The developer community's reaction was immediate. Within weeks, developers were integrating GPT into their tools. Within months, every software company had an "AI strategy." For developers specifically, ChatGPT and its API opened the door to: But ChatGPT as a chat interface had real limits for coding work: no file system access, no terminal, no persistent context across sessions. It was a brilliant assistant for thinking through problems, but not yet a capable agent that could act in your codebase. Key takeaway: ChatGPT proved that LLMs were genuinely useful. But it was a starting point — the foundation on which agentic tooling would later be built. The term React in this context goes beyond the JavaScript library — it describes a pattern of AI-assisted development where the AI reactively assists the developer rather than driving the interaction. Modern AI coding assistants Claude Code, GitHub Copilot, Cursor embody the React pattern: This is a human-in-the-loop model where AI augments human intent rather than replacing it. The human remains the conductor; the AI is a powerful instrument. Anthropic's Claude Code exemplifies the React pattern. It: The React pattern is fundamental because it acknowledges a crucial truth: for complex, high-stakes development work, human judgment is irreplaceable. AI accelerates and assists, but doesn't autonomously decide. Key takeaway: The React pattern keeps humans in control while letting AI handle the mechanical work. It's the practical intersection of AI capability and human oversight. This is where the stack takes its most significant leap. Agentic AI refers to AI systems that can autonomously plan, reason, and act toward goals — not just respond to prompts. An AI agent is characterized by: The canonical agent loop looks like this: Observe → Think → Plan → Act → Evaluate → Repeat This is often implemented with a ReAct Reasoning + Acting pattern, where the model interleaves natural language reasoning with tool-calling actions. The agentic AI space has exploded: As agents become more capable, the question of permission becomes critical. What should an agent be allowed to do autonomously? The spectrum ranges from: Modern agent frameworks like Claude Code implement granular permission systems where developers can scope what the agent can access and do. This is both a security concern and a practical necessity — you want your agent helpful, but not dangerously so. Key takeaway: Agentic AI shifts the paradigm from "AI answers questions" to "AI solves problems." The challenge is building agents that are helpful, safe, and reliable — which requires solving permission, memory, and tool integration problems. An agent harness is the software framework that orchestrates an AI agent's behavior — its tools, memory, reasoning patterns, and permissions. Think of it as the operating system for an AI agent. A well-designed harness handles: DeepSeek's Harness open-sourced under MIT in August 2026 provides four operating modes 3 : The plugin architecture in DeepSeek Harness is noteworthy — every capability is a plugin, making the system extensible without modifying core logic. Claude Code takes a different approach, optimizing for developer ergonomics: CLAUDE.md and AGENTS.md Key takeaway: The harness is the architecture that turns a language model into an agent. Good harnesses are opinionated about what's allowed and provide clean abstractions for extensibility. Tools are the interface between an AI agent and the external world. Without tools, an agent is just a very sophisticated text generator. File System Tools Search & Retrieval Tools API Tools Development Tools Every tool needs a schema — a machine-readable description of what the tool does, what inputs it expects, and what outputs it produces. This is harder than it sounds: This is where MCP Model Context Protocol comes in — it's a standardized way to define and discover tools. A key challenge is compactness — tool outputs especially file system reads can be enormous and quickly fill up the context window. Effective agents use strategies like: Key takeaway: Tools are how agents act. Designing good tool schemas — concise, accurate, and appropriately scoped — is one of the most practical skills in agent engineering. Hooks are lifecycle callbacks that run at specific points in an agent's execution. They enable developers to intercept, validate, or modify agent behavior without changing the agent's core logic. pre think → Before the agent starts reasoning post think → After reasoning, before acting pre tool → Before a tool is called post tool → After a tool returns pre response → Before the agent's response is delivered post response→ After the response is delivered on error → When an error occurs Compliance & Safety rm -rf , database drops Quality Gates Observability Custom Routing Hooks are powerful because they're composable — you can add, remove, and chain them without breaking the agent. But they work best for soft constraints. For hard safety guarantees, you need permission systems see below that can't be bypassed by a hook. Key takeaway: Hooks are the nervous system of an agent framework. They let you inject custom logic at precisely the right moments — making agents behave intelligently without monolithic rewrites. If hooks are the agent's reflexes, permissions are its immune system. Permissions define what an agent cannot do, regardless of what it's asked to do. Path-Based Permissions .env files, credentials, private keys Action-Based Permissions Capability-Based Permissions The best practice is to grant only the permissions needed for the specific task: This is the same principle that applies to human access control, and for good reason: agents, like humans, make mistakes. Permissions are the last line of defense. Claude Code implements a thoughtful permission model: --verbose flag exposes what the agent is doing, enabling human oversight Key takeaway: Permissions are not about distrust — they're about building reliable systems. An agent with appropriate permissions is an agent that can be trusted to be helpful. Skills are packaged capabilities that extend what an agent can do. Think of them as plugins or skill packs that add domain-specific expertise. A skill typically includes: Code Analysis Skills Domain Skills Integration Skills As the number of skills grows, skill discovery becomes important. The agent needs to know which skill applies to a given task. This is typically solved through: Clawhub.ai and similar registries are emerging as the ecosystem for sharing and discovering agent skills. Key takeaway: Skills are how you specialize a general-purpose agent into an expert in your specific domain. A well-designed skill is self-contained, well-documented, and composable with other skills. Compact refers to the practice of keeping the agent's context lean and efficient. It's not a feature — it's a discipline. Every LLM has a finite context window — the total amount of text it can "see" at once measured in tokens . Modern models offer 128K to 200K+ tokens, which sounds like a lot but fills up quickly: Research from Liu et al. "Lost in the Middle" shows that models perform 20-30% worse when relevant information is buried in the middle of a long context 4 . The model literally "forgets" things in the middle. Context Engineering as it was named in 2026 involves: A useful mental model: the context window is a cache, not a memory . It's fast to access but limited in size. Long-term information should be stored in a proper memory system, not kept in context 5 . Key takeaway: Compactness is not about being stingy — it's about being effective. The best agents are not the ones with the most context; they're the ones with therightcontext. This is where most AI agents fail. Memory is the system that allows an agent to persist information across sessions, learn from past interactions, and maintain continuity. Most AI agents don't actually have memory — they have context windows . Each session starts fresh. The "memory" is just whatever you put in the prompt this time. True memory requires: Explicit Memory Files CLAUDE.md , AGENTS.md in Claude Code Vector Memory Semantic Memory Systems The most insightful framing comes from the "Learn, Don't Store" methodology: most agents today remember everything and learn nothing . A better approach is to actively extract patterns and principles rather than storing raw transcripts. Example: Key takeaway: Memory is the hardest unsolved problem in AI agents. The difference between a tool that remembers and one that truly learns is whether it can extract actionable patterns from experience. When a task is too complex for a single agent, sub-agents also called delegates or child agents divide the work. Fanout Pattern Pipeline Pattern Hierarchical Pattern The biggest challenge with sub-agents is context explosion . Each sub-agent needs relevant context to do its job, but providing too much context: Effective sub-agent design uses context isolation — each sub-agent gets only the information it needs, not the full project context. A delegation runtime handles the mechanics of sub-agent management: spawning agents, routing tasks, collecting results, handling errors, and managing concurrency. Claude Code's architecture, for example, allows sub-agents to be spawned with isolated contexts or forked contexts depending on whether they need the parent's conversation history. Key takeaway: Sub-agents are about leverage — one human can orchestrate many specialized agents. The skill is knowing when to split work and how to keep each sub-agent focused. MCP Model Context Protocol is the emerging standard for connecting AI models to external tools and data sources. Developed by Anthropic and now adopted broadly, MCP provides a standardized interface for tool discovery, invocation, and data exchange. Before MCP, every agent framework defined tools differently: MCP solves this by providing: ┌─────────────┐ MCP ┌──────────────┐ │ AI Agent │◄────────────►│ MCP Host │ └─────────────┘ │ your app │ └──────┬───────┘ │ MCP ┌─────▼─────┐ │ MCP Server│ │ plugin │ └─────┬─────┘ │ native ┌─────▼─────┐ │File/DB/API│ └───────────┘ The MCP Host your application runs MCP Servers plugins that connect to actual resources files, databases, APIs . The AI agent talks to the host via MCP, without needing to know how the underlying systems work. Common MCP servers: The power of MCP is that it's composable — you can run multiple MCP servers simultaneously, giving your agent access to a diverse set of capabilities through a unified interface. Key takeaway: MCP is the USB-C of AI tooling. It standardizes how agents connect to the outside world, making the entire ecosystem more interoperable and extensible. Here's the complete evolutionary chain: Transformer + Attention ↓ Pre-Training raw knowledge ↓ Post-Training alignment, behavior ↓ ChatGPT proof of concept, consumer product ↓ React Pattern human-in-the-loop assistance ↓ Agentic AI autonomous goal pursuit ↓ ┌─────────────────────────────────┐ │ Harness orchestration layer │ │ ├── Tools capabilities │ │ ├── Hooks lifecycle events │ │ ├── Permission security │ │ └── Skills extensibility │ ├─────────────────────────────────┤ │ Compact context efficiency │ │ Memory persistence & learning │ │ Sub-Agents parallelization │ │ MCP standardized integration │ └─────────────────────────────────┘ Each layer exists because the layer above it hit a limitation. You can't have useful agents without good harnesses. You can't have effective harnesses without well-defined tools. You can't have reliable tools without compact, well-managed context. And you can't have any of this without the Transformer foundation. The trajectory is clear: The developers who thrive in this new paradigm won't be the ones who resist AI tooling. They'll be the ones who understand the stack deeply enough to build on it, extend it, and debug it when it breaks. 1 Vaswani, A., et al. "Attention Is All You Need." NeurIPS 2017 . https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762 2 Kaplan, J., et al. "Scaling Laws for Neural Language Models." arXiv 2020 . https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361 3 Rohit Raj. "DeepSeek Harness vs Claude Code vs Codex CLI: The v0.1 Developer Preview, Honestly — 2026." DEV Community . https://dev.to/rohit raj 8c7902b7d37cf21/deepseek-harness-vs-claude-code-vs-codex-cli-the-v01-developer-preview-honestly-2026-433e https://dev.to/rohit raj 8c7902b7d37cf21/deepseek-harness-vs-claude-code-vs-codex-cli-the-v01-developer-preview-honestly-2026-433e 4 Liu, N. F., et al. "Lost in the Middle: How Language Models Use Long Contexts." arXiv 2024 . https://arxiv.org/abs/2407.01073 https://arxiv.org/abs/2407.01073 5 Loop & Retry. "The context window is a cache, not a memory." DEV Community . https://dev.to/loopandretry/the-context-window-is-a-cache-not-a-memory-29f8 https://dev.to/loopandretry/the-context-window-is-a-cache-not-a-memory-29f8 This article is written for developers who want to understand the full AI-assisted programming stack, not just the parts they already use. Bookmark it, share it, and come back when you need to understand how the next piece fits in.