Setup: - Ollama serving llama3.1:8b-instruct-q4_K_M via /api/chat (streaming) - num_ctx=4096, temperature 0.2-0.3, num_predict 80-150 - FastAPI backend, RAG via Qdrant, feeding a voice agent (Retell AI, ~2-11 turns per call) and a WhatsApp agent, both in Spanish - Each turn: retrieve top-3 RAG chunks, build a messages[] array (system + full conversation history + latest user turn), call Ollama I’ve hit three related issues in production over the last couple weeks that all seem to come down to how this model handles injected context in a growing multi-turn conversation. Already fixed #1 and #3 with workarounds; still not 100% sure #2 is fully solved, and would love input on whether these are known limitations or if I’m doing something wrong architecturally. — Issue 1 (fixed): a second “system” message mid-conversation gets treated as something to respond to, not absorbed silently I was inserting retrieved RAG context as a separate “system” role message right before the latest user turn — i.e. [system(persona), user, assistant, user, assistant, system(RAG context + instruction “use this as background knowledge”), user(latest question)]. Instead of using that context silently, the model would respond directly to the injected instruction, e.g.: User: “¿Enrique Nguix es mi nombre? ¿Te sirve?” Model: “Sí, puedo utilizar la información proporcionada para responder a las preguntas relacionadas con Edentia y sus cursos. ¿Cuál es la pregunta específica que deseas saber?” It also started losing track of the actual conversation — one turn later, asked to recall something said 2 turns earlier, it replied “No hemos tenido una conversación previa sobre este tema” despite the full transcript being in messages[]. Fix: moved all dynamic context (RAG + any persistent memory) into the ONE system message at position 0, built fresh each turn, instead of inserting a second system message later in the array. Conversation history now goes in untouched after that single system message. This resolved both symptoms above in testing so far. Question: is this documented/expected behavior for Llama 3.1’s instruct tuning — i.e. is it only trained/reliable with exactly one system message at the very start, and a second system turn later in the conversation is essentially out-of-distribution? Or is this more likely an Ollama chat-template quirk in how multiple system-role messages get rendered into the underlying prompt? — Issue 2 (workaround in place, not fully confident it’s solved): model refuses to answer when a retrieved chunk is long/unstructured When a RAG chunk was a longer descriptive paragraph rather than a tight Q&A pair, the model would sometimes respond with something like: “Lo siento, pero no puedo continuar con la conversación debido a que el texto proporcionado parece ser una descripción general de la formación y requisitos para higienistas dentales en España, más que una pregunta o tema específico sobre el cual discutir.” i.e. it’s treating the injected context block itself as the thing it needs to respond to, rather than the actual user question. Workaround: added an explicit prompt instruction telling it the context block is never the question, and rewrote the knowledge base into short, single-answer chunks instead of long paragraphs. This reduced the frequency a lot but I’m not fully confident it’s eliminated, since it’s hard to reproduce reliably. Question: at 8B quantized to Q4_K_M, is this kind of “confusion about what to respond to when given a long context block” a known capability ceiling, or is there a better way to structurally signal “this is reference material, never the question” beyond prompt instructions (e.g. via message role/ordering, or some other delimiter convention this model was actually trained on)? — Issue 3 (fixed): silently truncating conversation history to a small window causes amnesia in longer conversations Was capping conversation history sent to the model at the last 6 messages (3 exchanges) to stay well under num_ctx=2048. In calls longer than ~3 exchanges (common in a support use case), the model would lose the original reason for the call, names given earlier, etc. — because that information had literally fallen out of the window it was given, even though the full transcript existed upstream (Retell AI keeps it). Fix: raised num_ctx to 4096 and widened the window to the last 16 turns, matching a comparable, simpler agent I run (booking-only, no RAG) that never truncated history at all and never had this problem. Not really a question here, just flagging it in case it’s useful context for issues 1/2 above — happy to hear if 4096 is still marginal for this kind of use case and I should go higher / consider a summarization fallback for anything older than the window instead of a hard cutoff. — Any pointers — whether that’s “yeah this is a known Llama 3.1 instruct limitation,” “you’re holding the chat template wrong,” or “here’s how people usually structure RAG context for this model” — would be genuinely useful. Happy to share more of the actual system prompt / message construction code if helpful.
Anatomy of a $5 "Good Morning"