{"slug": "separating-the-idea-state-and-artifact-planes-in-agent-pipelines", "title": "Separating the Idea, State, and Artifact Planes in Agent Pipelines", "summary": "An engineer built a personal automation pipeline where an AI agent scouts trending content and logs ideas, but killed it because the chat-based substrate made retrieval and state tracking impossible. The developer advocates separating agent pipelines into three data planes—idea intake, state tracking, and artifact storage—to avoid the failure mode of append-only chat logs with no schema or query interface.", "body_md": "This failure mode won't show up in your eval suite.\n\nAn AI architect builds a personal automation pipeline: an agent scouts trending content, pipes ideas into a chat thread, he reacts, things get logged. The agent did its job. It found the trends, wrote them down, and responded when he asked it to.\n\nHe killed it anyway, and not over hallucination or token cost. He killed it because everything lived in \"a flat scroll of messages. No structure, no views, no way to see what is in scripting versus what is scheduled.\" He was spending his entire one-hour creation window scrolling backwards through a Telegram thread, looking for something he'd written the day before.\n\nThe model wasn't the problem. The substrate was, and almost everyone shipping agents today has the same problem, because the default agent UI is a chat window and the default system of record is the transcript.\n\nIf your agent's output lands in a conversation, you've built an append-only log with no schema and no query interface. Retrieval is O(n), and a human eye does the scanning. That works fine in a demo and falls apart somewhere around week three.\n\nBreak a chat log down to its data properties and the problem is obvious:\n\n```\nCHAT TRANSCRIPT AS DATABASE\n---------------------------\nSchema            none (freeform text)\nPrimary key       timestamp (not semantic)\nIndexes           none\nQuery interface   Ctrl+F, human eyes\nMutation          impossible (append-only, no UPDATE)\nAggregation       none\nViews/filters     none\nState transitions untracked\n```\n\nThe row that matters is `Mutation: impossible`\n\n. Real work has state that *changes*: an idea becomes a draft, the draft gets scheduled, the scheduled item ships. In a chat log none of those transitions exist as data. They exist as later messages contradicting earlier ones, and the reader has to reconstruct the current truth by replaying the whole thread.\n\nThat's event sourcing without a projection. You get the event stream and no materialized view.\n\nIt also compounds. Every session starts with context reconstruction: *what did I decide? where did I leave off? which of these forty messages is still live?* The human pays that cost every time, and it grows with the length of the thread while the thread's value stays flat.\n\nYou'll know you have the problem when you catch yourself using the agent to search the agent's own output.\n\nThe working version of the pipeline separates concerns into three data planes, each with a different write pattern and a different owner.\n\n```\n        ┌──────────────────────────────────────────┐\n        │  PLANE 1: IDEA / INTAKE                  │\n        │  Google Sheet · append-only              │\n        │  Writer: agent (scheduled, unattended)   │\n        │  Volume: high · Precision: low           │\n        │  Never edited in place                   │\n        └───────────────────┬──────────────────────┘\n                            │  human selects (the gate)\n                            ▼\n        ┌──────────────────────────────────────────┐\n        │  PLANE 2: STATE / PIPELINE               │\n        │  Notion DB · typed records               │\n        │  Writer: agent on human instruction      │\n        │  Volume: low · Precision: high           │\n        │  status ∈ {Idea, Scripting, Filming,     │\n        │            Scheduled, Published}         │\n        └───────────────────┬──────────────────────┘\n                            │  references by ID\n                            ▼\n        ┌──────────────────────────────────────────┐\n        │  PLANE 3: ARTIFACT                       │\n        │  Google Docs · Drive · rendered media    │\n        │  Writer: human (scripts) + agent (renders)│\n        │  Addressed by link from Plane 2          │\n        └──────────────────────────────────────────┘\n```\n\nEach plane has a job the other two are bad at.\n\nPlane 1 optimizes for recall rather than precision. The scheduled scan writes 20+ ideas a day across TikTok, Reels, X and YouTube, each row carrying the trend, a hook, an angle, a caption, hashtags, format notes, and a suggested post time. Most of those rows are garbage, which is fine. It's an intake buffer, and the rule is *never curate in place*.\n\nThat rule earns its keep. Agent writes are non-deterministic, so if your agent mutates rows in the same table it writes to, you lose the ability to tell \"the agent changed its mind\" from \"I changed my mind\" from \"the agent silently dropped something.\" Append-only intake gives you an audit trail for free, and it makes human selection the only path from noise to signal.\n\nPlane 2 optimizes for queryable state: one typed record per unit of work, with a status enum. A single glance answers what's in scripting, what's waiting to film, and what ships next, which is the question the chat thread could never answer.\n\nPlane 3 holds the payload. Documents and media are referenced by ID from Plane 2, never inlined into it. That's ordinary normalization, and agent systems break it all the time by pasting entire drafts into chat messages.\n\nThe state machine itself is simple enough to copy:\n\n```\n  Idea ──► Scripting ──► Filming ──► Scheduled ──► Published\n   │           │             │\n   └───────────┴─────────────┴──► Dropped\n```\n\nFive states and one escape hatch. That's the whole orchestration model, and both the human and the agent can read it, which matters: an agent that sees `status = \"Scripting\"`\n\ncan act on it directly. An agent reading a chat thread has to *infer* status from prose, and inference is where reliability starts slipping.\n\nThe second architectural idea in the piece is a clean three-way split of what most people mash into a single \"agent\":\n\n| Component | Answers | Analogue |\n|---|---|---|\nWorkflow |\nHow the job is done |\nDAG definition / playbook |\nRunner |\nWho owns and executes it |\nWorker identity + permissions + memory |\nSchedule |\nWhen it fires |\nCron / event trigger |\n\nIf you've ever built data infrastructure, this is Airflow's DAG-vs-worker-vs-scheduler split arriving in the agent world, and the fact that it keeps getting rediscovered independently is a decent signal that it's correct.\n\nThe practical consequence is that each axis varies independently. Same workflow, different runner (dev vs prod credentials). Same runner, different schedule (daily scan vs on-demand). Same schedule, swapped workflow (v1 to v2 of your edit pipeline) without touching the trigger or the identity.\n\nThe monolithic alternative is a chat agent where the how, the who, and the when are tangled together in a prompt you retype every session. You can't version that, and you certainly can't hand it to anyone else.\n\n```\nENTANGLED (chat agent)\n  \"hey can you scan trends and put them in my sheet like\n   you did last time, you know the format\" ──► ???\n\nDECOMPOSED\n  Workflow:  trend_scan.v3   (deterministic playbook)\n  Runner:    content_bot     (Drive+Notion scopes, memory: voice/topics)\n  Schedule:  0 6 * * *       (daily 06:00)\n```\n\nThe decomposed version is a config artifact. You can diff it, review it, and roll it back.\n\nThis is the detail with the longest shelf life. The video editing logic (cut to vertical 1080×1920, word-synced captions, motion graphics, brand logos, phone-legible thumbnail) was a Claude Skill wrapping Remotion. When the pipeline moved to a different orchestrator, that logic wasn't rebuilt from scratch. It was ported into a saved Workflow and invoked by a Runner.\n\nThe value was never in the runtime.\n\nIt was in the encoded decision rules: caption style, safe zones, font sizes, brand colors, render steps. Those took work to get right and would be expensive to rediscover. The execution environment around them is commodity infrastructure that churns every eighteen months or so.\n\n```\n       ┌─────────────────────────────────────┐\n       │  DURABLE  (you author this once)    │\n       │  • decision rules & checklists       │\n       │  • constraints, safe zones, brand    │\n       │  • step ordering & failure handling  │\n       └──────────────┬──────────────────────┘\n                      │ portable\n       ┌──────────────▼──────────────────────┐\n       │  COMMODITY (swap every ~18 months)  │\n       │  • agent runtime / orchestrator      │\n       │  • model provider · tool bindings    │\n       └─────────────────────────────────────┘\n```\n\nSo write skills as instruction artifacts rather than runtime code. A skill expressed as a markdown playbook with explicit steps, constraints, and decision rules is a text file you can carry anywhere. Bind it hard to one vendor's SDK and you've scheduled a rewrite.\n\nIf you already have Claude Skills, you already have portable assets. They're the durable layer, not vendor lock-in, and every orchestrator (including the one you're using today) is replaceable.\n\nThe pipeline has five stages. Two of them are deliberately not automated:\n\n```\n1. Trend scan   AGENT   (scheduled, unattended)\n2. Curation     HUMAN   → agent files the selection\n3. Scripting    HUMAN   ← agent assists on request\n4. Filming      HUMAN   (fully manual, by design)\n5. Editing      AGENT   (workflow-invoked)\n```\n\nAutomation sits at the ends, intake and post-production, while the human occupies the middle where judgment lives. The stated reason for keeping scripting human is sharp: *\"the point of the system is not to remove me from the work that is mine.\"*\n\nThe engineering framing is Amdahl's law. Total time is critical path plus serial overhead. In a creative or knowledge pipeline the critical path is judgment, and it's irreducible because *you* are the value being produced. The serial overhead is everything around it: noticing, logging, filing, linking, formatting, rendering.\n\nSpeed up the overhead and the whole system gets faster. Try to speed up the critical path by automating judgment and you don't get faster output, you get more output that nobody wanted.\n\nThere's a second constraint the piece names explicitly, and it deserves a term: the friction budget. As he puts it: *\"anything that is not seamless does not get done. If an idea requires me to open six tabs, copy something from one place to another, and remember where I put the draft, it dies.\"*\n\nTreat friction as a hard budget. For a part-time operator with one hour, the budget is near zero: six tabs exceeds it, and the workflow gets abandoned no matter how good the agent is. It's the same reason an accurate internal tool that takes six clicks to reach loses to a rougher one sitting behind a single button. Adoption follows friction more than capability. Most agent projects that \"failed\" cleared the capability bar and blew the friction budget.\n\nIf you're standing up an agent pipeline, personal or production, the port is mechanical:\n\nEvery major agent product is converging on the same interface: a chat box. But the moment an agent's work has state that outlives a session, the chat box stops being a UI and starts being a bad database.\n\nSo which is it? Is chat a transitional interface we'll look back on the way we look back on command-line-only databases, with agents eventually shipping real structured front ends? Or is the transcript fine, and the fix is just better memory and retrieval bolted onto the thread?\n\nIf you've killed an agent that technically worked, what made you stop using it? I'd bet more of those stories are about state and friction than about the model.", "url": "https://wpnews.pro/news/separating-the-idea-state-and-artifact-planes-in-agent-pipelines", "canonical_source": "https://dev.to/shakti_mishra_308e9f36b5d/separating-the-idea-state-and-artifact-planes-in-agent-pipelines-518g", "published_at": "2026-08-22 23:17:55+00:00", "updated_at": "2026-08-22 23:43:23.988247+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "artificial-intelligence"], "entities": ["Google Sheets", "Notion", "Google Docs", "Google Drive", "TikTok", "Reels", "X", "YouTube"], "alternates": {"html": "https://wpnews.pro/news/separating-the-idea-state-and-artifact-planes-in-agent-pipelines", "markdown": "https://wpnews.pro/news/separating-the-idea-state-and-artifact-planes-in-agent-pipelines.md", "text": "https://wpnews.pro/news/separating-the-idea-state-and-artifact-planes-in-agent-pipelines.txt", "jsonld": "https://wpnews.pro/news/separating-the-idea-state-and-artifact-planes-in-agent-pipelines.jsonld"}}