Creating an AI Streamer That Remembers Previous Visits — Designing Memory and Multi-Streaming States

A developer building an unattended AI avatar stream detailed two state-management challenges: remembering viewers across streams and maintaining state during renderer crashes. The project adopted per-character memory scoping and incremental extraction to handle interrupted streams, while fixing a bug where memories lacked viewer identifiers. A unique-constraint collision on guest email addresses was identified when same-named viewers appeared across different characters.

📝 Originally published in Japanese at forge.workstyle.tech . I'm building an AI avatar stream that runs unattended. No human broadcaster — the avatar reacts to comments, makes small talk, and carries the show through to the sign-off. Of all the walls you have to get over to make that work, this post covers two problems that both come down to state . The first is remembering viewers across streams. Whether the avatar can say "oh, you were here last time too" makes a surprisingly large difference to the experience. The second is where to keep the state during a stream. The renderer the headless browser that produces the video crashes. And if the avatar redoes its greeting every time it crashes and reconnects, viewers see a stream that introduces itself once a minute. Push on either one and you arrive at the same question: which state, at what granularity, held where? I'll go through them in order, and I'll be candid about both the design decisions and the bugs I hit along the way. Start with viewer memory. The first thing I had to decide was the unit at which memory is shared. | Option | Meaning | Problem | |---|---|---| | Per show | Only remembers people met on "Monday Chat" | Back to strangers on a different show with the same character. Unnatural | | Per stream run | Only remembers within that one run | Can't recognize returning viewers. No point implementing it | Per character | Remembers everyone that character has met | Adopted | I went with per character . The reason: the entity a viewer forms a relationship with isn't the "show," it's the character. If that same character is hosting a different show and remembers you from before, that matches how it intuitively ought to feel. In implementation terms, the scope for memory lookups is keyed by the character's identifier. scope = broadcast-char:{characterId} This scope design sets up a bug I hit later. I made the process that extracts "things worth remembering" from conversation a two-stage affair. My first attempt used only final extraction, and it failed. Streams get cut off. The renderer crashes, the server restarts, the viewer leaves without saying anything. A design that "processes everything when it's over" leaves you with nothing when the end never arrives. Accumulate incrementally, tidy up when the end comes. And if the end never comes, the incremental part is still there. This instinct — don't assume there will be an ending — shows up again in exactly the same shape in the state design below. In the first implementation, I passed the whole conversation log to the extraction step at once. The resulting memories came back with no viewer identifier attached , and got saved as memories shared across the character. Which means what Alice said could get referenced in a conversation with Bob. On a live stream, the avatar starts mixing in someone else's business — that's real harm. The fix was to run the extraction inside the utterance turn, in the context of the target viewer . "Who is this memory about" is carried as the execution context, not as an input to the extraction. Data that belongs to someone — like a memory — should be created in that person's context from the start, not associated with them after the fact. This was the cleanest bug of the bunch. Viewers are anonymous guests with no registered account. To create an internal record for them, I generated a placeholder email address. guest+{displayName}@guest.invalid Email addresses carry a unique constraint . And that string contains no scope — nothing indicating which character's viewer this is. Here's what happens. "Taro" shows up on Character A's stream → guest+Taro@guest.invalid is created ✓ "Taro" shows up on Character B's stream → tries to create guest+Taro@guest.invalid → unique constraint collision → error ✗ It breaks the moment a same-named viewer appears in a different scope. Display names are whatever viewers choose, so collisions should have been the design assumption. Worse, the error surfaced in an unhelpful way: "only viewer processing fails, and only on certain streams." It took me a while to trace it back to the cause. The fix is just to include the scope in the identifier. guest+{scope}+{displayName}@guest.invalid The lesson: if data has a scope, its unique key must include that scope. It sounds obvious, but this is exactly the obvious thing that slips when you're generating placeholder or dummy values — because the real data an email address genuinely is globally unique, and the stand-in isn't . Also: "same-named users in different scopes" will never, ever happen in your test data. It only showed up once real viewers arrived. Which is another way of saying unique constraints lie to you until real data arrives . Once viewer memory was working, the next thing that started standing out was state loss. Baseline reality: in an AI avatar livestream, the renderer a headless browser crashes. The GPU host gets flaky, the browser crashes, the video encoding process dies. Recovery — reload and reconnect — was in there from the beginning. And here's what that produced. On every recovery, the avatar redoes its greeting: "Good evening Let's get started." From the viewer's side, that's a stream that starts introducing itself once a minute. A wonderfully ill-timed failure mode: the better the recovery logic works, the more obvious the symptom. The cause was clear: the page owned the "have I greeted yet?" flag. page loads → connects → hasn't greeted yet the variable is at its initial value → greets State in the page's memory disappears on reload. Obviously. And yet that's where I'd put "what has happened during this stream." For the same reason, all of this was being lost too: The memory-extraction failure above was "assume there's an ending and you're left with nothing" — this is its twin. This time it's "put something you can't afford to lose in a place that isn't designed to break." I changed the approach. The page holds nothing. stream state Redis snapshot one per stream run ↑ updates dialogue server generates utterances, updates state ↓ WebSocket page display only. On connect, receives the current state and renders it The page became a "display-only client." On connect it receives a snapshot and builds the screen from it. The page itself remembers nothing, and doesn't need to. The greeting decision moved server-side too, naturally. a connection arrives → look at the snapshot → greeted flag is set → don't greet, resume from where we were → flag isn't set → greet, set the flag Now the page can crash and reconnect any number of times, and the greeting happens exactly once. Put in too much and it gets heavy; put in too little and you get inconsistencies after recovery. Here's roughly what's in there: That last one — the short per-viewer memory — is where this connects back to the character-scoped memory from the first half. The canonical memory lives in the persistent store, but the slice needed during the stream rides along in the snapshot, so the avatar can still say "you were here last time too" after a recovery. What's not in there: video frames, the audio itself, UI animation state. Those are all "just redo them after recovery" things, so there's no point holding them as state. The test is: is this needed to resume from where we left off? Appearance can be rebuilt; context can't. With this kind of "data that keeps being updated for the whole stream," the scary failure is unbounded growth . Naively appending the entire conversation history will eat all your memory on a long stream. I ran a two-hour continuous test to check. | Metric | Result | |---|---| | Snapshot size | Plateaus at 6.2KB stops growing | | Process memory | No increase | | Utterance turns | 53 turns, all measured | Because history is capped to the most recent entries, it stops at a ceiling. Decide at design time whether there's a bound, then confirm it by measurement. One without the other isn't enough — the design can have a bound while the implementation grows somewhere else. Pushing state into an external store produced a few advantages I hadn't planned on. 1. Recovery is much easier to test Just manually reload the page and you can watch the recovery behavior. Before, I had to kill the process and wait for it to come back up. 2. You can watch mid-stream Since display-only clients are unlimited, I can peek at the live stream state from another browser. Debugging got dramatically easier. 3. Renderers became disposable This is the big one. Because the page holds no state, a brutally simple recovery works: if the GPU host is flaky, throw the whole thing away and re-acquire on a different host. With state living in the page, that option didn't exist. Remembering what a viewer said and bringing it up next time is good as an experience, but it needs care. Here's where I've drawn the line: Even for remarks in a public setting, some people find being remembered uncomfortable. At minimum, being able to explain what is remembered is the builder's responsibility. Keeping only short summaries in the snapshot isn't just a size concern — it lines up with this line too. Viewer memory and stream state look like separate topics, but they converged on the same design principles. The thread running through all of it: don't put important state in a place that breaks, or a place that assumes an ending. Sometimes reconsidering where state lives is faster than hardening your recovery logic, and reconsidering "whose context is this created in" is faster than trying to re-associate memories after the fact. Both are things I learned from bugs that only surfaced once real viewers showed up. Test data doesn't lie to you like that.