{"slug": "companion-lab-an-open-local-lab-an-unscripted-companion-and-a-whole-novella", "title": "Companion Lab: an open local lab, an unscripted companion, and a whole novella", "summary": "A hands-on test of the open-source Companion Lab found that moving a prompt instruction between the start and later in a conversation changes more than placement: early instructions get joined to the leading system message, while late ones become a separate system message after earlier turns, altering message boundaries and potentially the model's rendering. The tester ran Qwen3-4B-Instruct-2507 Q4_K_M GGUF on a Colab T4 via a prebuilt JamePeng CUDA wheel and the Lab's OpenAI-compatible adapter, and reported that one follow-up probe produced only 144 output tokens and hit finish_reason=length, which the Lab correctly marked incomplete. The tester cautioned that the samples are far too small to rank placements or infer anything about Helena's private memory or long-term behavior, and recommended checking assembled messages, role handling by the selected model/backend, and whether generation completed before running a larger evaluation.", "body_md": "For now, I tried running it:\n\n**Of the three things you suggested comparing, I would start with prompt placement.** I tried the public Companion Lab with a real GGUF model served through an OpenAI-compatible endpoint. That gave me a small experiment I could actually reproduce with the included `turns` context plugin, two saved prompt versions, and no knowledge of the private context system behind Helena.\n\nMy first useful observation was about **what the Lab sent**, rather than which placement produced the better answer. Moving an instruction to the beginning also joined it to the leading system message; moving it late made it a separate system message after earlier turns. Both are legitimate arrangements, but the message boundaries and potentially the model’s rendering change too. The saved requests make this visible.\n\n**For “what would make the lab more useful,” my first small request would be a portable A/B comparison packet:** one easy download containing the *two saved requests, two replies, prompt-version IDs and available model/adapter/completion metadata*. This comes from having to assemble those pieces myself. The Lab already records the comparison requests and replies under `data/compares/`, and it has a separate **Save full transcript** function for a normal conversation. I am not suggesting it lacks logging. I would simply find it useful to take one complete comparison away, inspect it later or share a synthetic reproduction without joining the files by hand.\n\nMy default first experiment would be:\n\n`turns` plugin, and two saved prompt versions differing in the placement of one instruction. Keep the actual instruction text, final question, short conversation history and generation settings fixed as far as possible.`request-a.json` and `request-b.json`: message count, order, roles, grouping and content. Also note the adapter’s additions and what the backend can actually accept.`finish_reason`, output limits and errors when available. A correct but cut-off explanation is not a completed answer.\n**What I saw in brief:** I used Qwen3-4B-Instruct-2507 Q4_K_M GGUF on a Colab T4 through a prebuilt JamePeng CUDA wheel and the Lab’s OpenAI-compatible adapter. Both placement arrangements reached the model, and the native logs confirmed GPU inference; one attempted answer later ran into my own output limit. Three short, invented continuity situations were judged as expected by the completed responses. I also tested a third *manually rearranged* message sequence directly through the native worker. That third condition was **not** a hidden third lane in the public Side by side UI. The samples are far too small to rank placements or infer anything about Helena’s private memory, novella criticism or long-term behavior.\n\nOne useful negative result was mine, not the Lab’s: my follow-up probe gave a response only **144 output tokens**, so one explanation reached `finish_reason=length`. The Lab correctly marked it incomplete. A targeted native retry with a larger budget completed it. That reminded me to inspect completion status before attributing a short answer to prompt wording or placement.\n\nI would therefore put three inexpensive checks ahead of a larger evaluation: **the assembled messages, how the selected model/backend handles their roles, and whether generation completed**. The [Hugging Face chat-template guide](https://huggingface.co/docs/transformers/main/en/chat_templating) is useful context: a list of role/content objects is not necessarily the exact token sequence a model receives. And the [Companion Lab README](https://github.com/sheilastudios/companion-lab) already makes the public/private boundary explicit; the published 50-turn interaction is an unscripted example, not a controlled comparison. Keeping a missed detail in that record, rather than only the polished moments, is useful context for readers.\n\nBelow are the details and optional branches I would follow depending on whether the real question is placement, a memory selector, a different model, or useful criticism of a story. None of them requires turning the Lab into a full benchmark platform.\n\n1. Prompt placement: what actually moves\nSo my two practical answers remain **prompt placement first**, using the public `turns` plugin and the saved request diff, and **a portable complete A/B packet** as the improvement I’d most appreciate. The GGUF run gave me an actual reason for both choices. Memory selection, cross-model behavior and story criticism are interesting follow-ups if those become the specific questions someone wants to explore, not extra requirements for this public release. Thanks for sharing a workbench where those questions can be inspected rather than only guessed at.", "url": "https://wpnews.pro/news/companion-lab-an-open-local-lab-an-unscripted-companion-and-a-whole-novella", "canonical_source": "https://discuss.huggingface.co/t/companion-lab-an-open-local-lab-an-unscripted-companion-and-a-whole-novella/192115#post_2", "published_at": "2026-11-19 13:08:54+00:00", "updated_at": "2026-11-19 13:11:36.574626+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-research"], "entities": ["Companion Lab", "Qwen3-4B-Instruct-2507", "Colab T4", "JamePeng", "Helena", "Hugging Face", "sheilastudios"], "also_reported_by": [], "takeaway": "In a small Companion Lab test, moving an instruction to the start of a prompt merged it into the leading system message while moving it late created a separate system message after earlier turns, and a 144-token output limit caused finish_reason=length, so check message boundaries and completion sta", "recommendation": {"advice": "Before ranking prompt placements, inspect the assembled message roles and boundaries and confirm finish_reason is not length; a 144-token cap produced a truncated answer that the Lab flagged as incomplete.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/companion-lab-an-open-local-lab-an-unscripted-companion-and-a-whole-novella", "markdown": "https://wpnews.pro/news/companion-lab-an-open-local-lab-an-unscripted-companion-and-a-whole-novella.md", "text": "https://wpnews.pro/news/companion-lab-an-open-local-lab-an-unscripted-companion-and-a-whole-novella.txt", "jsonld": "https://wpnews.pro/news/companion-lab-an-open-local-lab-an-unscripted-companion-and-a-whole-novella.jsonld"}}