How We Built Charlie, Part 15: What We Learned Building Charlie
What we learned about making AI engineering work owned, bounded, verifiable, and useful to a team.
How follow-ups change active work safely without starting over or letting stale requirements continue.
How We Built Charlie continues from Chapter 8: the root Task now has to absorb a human follow-up while work is running.
Charlie is debugging a checkout failure reported in Slack. He is inspecting the repository and recent payment changes when the reporter adds two details: the bug only appears in Safari, and it only affects saved cards.
Those details should change the investigation. They should not open a second checkout task, disappear in a thread Charlie has already read, or interrupt a tool call halfway through a repository operation. The active owner needs to receive the clarification at a boundary where it can update its plan safely.
Charlie handles that with a durable mailbox attached to each Task. Routing can append a follow-up to the existing Task. The executor checks for later input between phases, adds each new message to the model-visible transcript once, and continues with the new context. Delivery and incorporation remain separate facts: a successful append means the Task owns the message, while later executor state shows whether a model call has actually seen it.
That distinction is what makes steering dependable. People can add scope through the collaboration surface they are already using, and the system can preserve the update without pretending that a running model or tool was interrupted in place.
The follow-up is durable when appended. It becomes model input at a later executor checkpoint.
Each Task provides a durable target for later messages. A mailbox message has its own identity, creation time, and content parts. Text can be accompanied by file references when the provider surface supports them. The scheduler stores the message against the target Task rather than relying on an in-memory model session to remain available.
This is the same ownership principle used throughout Charlie’s coordination layer. The Slack thread or GitHub comment is the human collaboration surface. The Task mailbox is the runtime address for input that should affect an active objective.
A mailbox is per Task, not a broadcast bus. A child worker does not automatically receive a follow-up sent to its parent. The parent may decide to pass the new detail into a later child contract, stop obsolete delegated work, or wait for the child and reinterpret its result. Automatic broadcast would make it difficult to know which instructions each worker actually used.
The mailbox also does not mutate prior transcript history. It adds a later user message. The original request remains the original request, and the correction remains a correction with its own identity and time.
Before a mailbox can help, the orchestrator has to identify the existing Task and decide that the new Signal belongs there. Chapter 3 described how provider events become enriched Signals. Routing then compares the Signal’s resource and thread context with active work and policy.
For a Slack follow-up in the same thread, routing may select the active checkout Task. For a GitHub issue comment, it may resolve the canonical issue and find a Task already handling that objective. If the new message asks for unrelated work, if no suitable Task exists, or if the prior owner is terminal, routing may create a fresh Task instead.
The choice is an ownership decision:
| Route | Meaning |
|---|---|
| Append to existing mailbox | This is later input for an objective that still has an active owner |
| Schedule a new Task | This is a new objective, or the prior owner can no longer accept the input |
| Drop or no-op | The Signal is duplicate, unsupported, self-generated, or non-actionable |
| Route to a daemon | A pre-scoped persistent role owns this class of event |
Thread proximity is useful evidence, but it is not sufficient by itself. A Slack thread can drift into a second request. A GitHub issue can collect multiple distinct objectives. Routing uses the available work-aware context and policy rather than treating every later message on the same surface as guaranteed continuation.
Providers and queues can redeliver. The orchestrator may retry after a timeout. A dependable append path needs repeated attempts for the same intended effect to converge.
The scheduler accepts an idempotency key with the mailbox write. Repeating the same key against the same Task returns success without inserting the message twice. The same key used against a different Task does not collide, because deduplication is scoped to the target.
Conceptually, the effect key is:
(target task ID, mailbox idempotency key) -> one stored message
This target scope matters during routing recovery. Suppose the orchestrator first tries Task A, discovers that A is terminal, and has enough proof to schedule fallback Task B. The same source Signal can participate in both attempts without a global mailbox key accidentally suppressing the valid write to B.
The guarantee is deliberately local. Mailbox idempotency does not make the provider delivery, Signal creation, routing decision, Task scheduling, and every later tool call one atomic exactly-once transaction. It protects one defined scheduler effect.
At the executor boundary, the message identity provides another guard. When mailbox messages are appended to the transcript, the executor uses a deterministic item ID derived from the mailbox message ID. If a checkpoint is replayed or the scheduler returns an already-seen message again, the transcript does not receive a second copy.
These two layers solve different repetition problems:
| Boundary | Dedupe identity | Prevents |
|---|---|---|
| Scheduler mailbox | Target Task plus append idempo | Duplicate stored message from repeated writes |
| Executor transcript | Mailbox message ID | Duplicate model-visible user item on replay |
An existing Task can become terminal between routing lookup and mailbox append. The scheduler reports that conflict instead of accepting input into completed work.
The orchestrator classifies mailbox failures. An explicit already-terminal response is strong evidence that the selected target cannot accept the message. A not-found response is more ambiguous, so routing can read the Task back and distinguish a terminal Task, a still-active Task, a genuinely missing Task, or a reconciliation failure.
Only bounded cases justify automatic fallback to a new Task. If the append failed because the target is proven terminal, a fresh Task can preserve the user’s request. If the target is proven missing, scheduling a new owner may also be appropriate. If read-back shows the Task is still active, or reconciliation itself fails, blindly scheduling another Task would risk duplicate ownership.
Fallback requires evidence about the target state. Ambiguity should not turn into a second owner automatically.
Transient failures are handled differently. Timeouts, selected retryable statuses, and server errors can be retried according to the scheduler client’s policy. Quota and non-retryable failures remain explicit outcomes. The routing plan records which attempt and idempotency key it used so retries can remain tied to the same intended effect.
This is a good example of why “send the message somewhere” is not enough. The product behavior users expect is one coherent owner. Recovery policy has to preserve that ownership even when the append races with completion.
The executor is a persisted phase machine. It calls the model, dispatches tool calls, persists results, checks scheduler state, and advances the run. Mailbox delivery is integrated into that loop at explicit scheduler-check phases.
At a check, the executor first asks whether the Task should stop. If cancellation or ancestor lifecycle policy requires a stop, it transitions to stopped reporting instead of pulling more work into a Task that no longer has authority to continue. Otherwise it asks for mailbox messages after the Task’s current mailbox position.
New messages are appended to the transcript, the mailbox position advances, and the next model call can see them as later user input. This ordering means a stop request has lifecycle authority over ordinary continuation input at that checkpoint.
The executor also checks stop state around tool dispatch. Before a batch begins, it can avoid starting more effects when a stop is already known. After a batch finishes and its results are persisted, it checks again. With a parallel tool batch, it cannot stop between tools that are already executing as part of that batch. The boundary is after the batch settles.
That is why we describe cancellation and follow-up incorporation as cooperative. A mailbox message does not interrupt a model call in progress. It does not splice itself into a tool request already sent to GitHub or a command already running in a devbox. A stop request cannot retract an external API call that a provider already accepted. The executor observes both at defined boundaries and changes what happens next.
Several facts can be true at different times:
Collapsing those facts into “Charlie got the update” makes status reporting imprecise. A successful mailbox API response proves durable delivery to the Task. It does not prove that the active run has reached a checkpoint, that the next model call included the message, or that the model interpreted it correctly.
When incorporation matters, evidence should move further down that chain. The transcript can show the deterministic mailbox item. A later plan update can mention the Safari constraint. The diff can add a WebKit-specific regression case. The final response can explain how saved-card scope changed the diagnosis. Those artifacts support a stronger claim than append success alone.
Mailbox content enters the model-visible history as a user-role transcript item. For agents with mailbox prompt rendering configured, Charlie wraps the new input in a structured follow-up prompt. The renderer receives the mailbox message, a Task-shaped view whose message is the follow-up, and the follow-up text. Non-text parts such as image references pass through alongside the rendered text.
A simplified shape looks like this:
<mailbox_update>
The reporter confirmed this only affects Safari with saved cards.
</mailbox_update>
The exact prompt is versioned with the agent specification. If configured rendering is unavailable or fails, the executor can fall back to appending the raw mailbox content rather than discarding the user’s update.
This rendering boundary lets the model distinguish the original task from later input without rewriting either. It can also carry instructions about how to treat the update, such as acknowledging a follow-up on the origin surface before continuing work. The durable mailbox remains transport and ownership; the agent prompt defines how that later message is presented to the model.
The sequence has two durable handoffs: scheduler storage and transcript insertion. Model incorporation happens after both.
Provider delivery, routing, mailbox storage, and transcript insertion each retain the identity available at that boundary:
This chapter’s concern is the human requirement: one follow-up should reach the active owner and enter the model-visible history once. Chapter 4 covers routing convergence, and Chapter 6 covers replay from persisted executor state.
The visible product experience varies by provider, while the runtime semantics remain consistent.
In Slack, an engineer may add a reply in the original thread: “This also reproduces with saved cards.” The thread identity helps routing find the active Task. Charlie acknowledges the follow-up according to communication policy, and the executor later incorporates it through the mailbox.
In GitHub, a maintainer may add a second comment on an issue after Charlie has started a branch: “Please keep the public API unchanged.” The canonical issue and comment context can route that Signal to the active owner. At the next checkpoint, the constraint appears as a later user message, and Charlie can revise the implementation or plan before the next effect.
The provider thread is not the execution transcript. Slack edits, GitHub comment order, and provider redelivery behavior remain provider concerns. The Task mailbox gives both surfaces a common continuation path without flattening their source identities.
This also supports a clear response when the follow-up arrives too late. If the active Task has already completed, routing can create a new Task only when the terminal state or missing target is established. The new owner can link back to the prior artifact instead of mutating history as if the first task never finished.
A human may follow up with “stop” rather than more scope. Cancellation is lifecycle authority, not an ordinary prose instruction that the model may choose to ignore.
The scheduler records the stop condition. The executor checks it before mailbox retrieval at the scheduler boundary and around tool dispatch. When observed, the run transitions to stopped reporting. Descendant work can observe ancestor stop state through scheduler policy at its own checks.
The limitation is explicit: a stop is not a hardware interrupt. If a shell command is already running, a model request is already in flight, or a provider write has already been accepted, the scheduler cannot rewind that effect. The executor stops starting new work at the supported boundary and reports what happened.
This is safer than injecting “stop now” as another user message and hoping the next model response handles it correctly. Lifecycle control belongs to the scheduler, while mailbox content belongs to continuation.
Suppose Slack retries a delivery and the orchestrator repeats the same mailbox append. The scheduler’s target-scoped idempotency returns success without storing a second message. If an executor checkpoint is replayed and the same stored message is returned, the deterministic transcript item ID prevents a second insertion.
The model therefore sees one follow-up, not two copies that could make the constraint look more important or lead to two acknowledgments. This is especially important for action-oriented updates. Repeated text such as “please also update the changelog” should not cause the same external effect twice merely because transport retried.
Idempotent input does not automatically make downstream effects idempotent. The agent still needs operation-specific safeguards for commits, comments, pull requests, or deployment actions. The mailbox ensures that repeated delivery does not manufacture repeated intent at the transcript boundary.
Follow-ups feel conversational to the user, but dependable continuation requires more than an open chat window. The system needs a stable Task address, routing that can distinguish continuation from new work, per-target idempotency, terminal-state reconciliation, checkpointed executor delivery, deterministic transcript insertion, and scheduler-owned cancellation.
Those pieces expose a useful progression: the Task stores the follow-up, the executor inserts it at a checkpoint, the next model turn incorporates it, and later artifacts show whether the requirement changed the work. Cancellation uses the same boundaries to end authority before further dispatch.
For the checkout investigation, the result is straightforward: the reporter adds Safari and saved-card scope in Slack, the update reaches the active Task, the executor incorporates it before the next reasoning step, and Charlie adjusts the investigation. The runtime can show how that happened without claiming the message appeared inside work that was already in flight.
Previous: How We Built Charlie, Part 8: Delegation Without Swarms. Next: How We Built Charlie, Part 10: Devboxes as the Agent’s Body. Browse the full How We Built Charlie series.