Setup: Ollama serving llama3.1:8b-instruct-q4_K_M, chat completions API (/api/chat), num_ctx=4096, temperature 0.2-0.3

A developer reports three production issues with Ollama serving llama3.1:8b-instruct-q4_K_M for multi-turn conversations in Spanish, including the model treating a second system message as a response target, refusing to answer when RAG chunks are long, and truncating conversation history causing amnesia. The developer fixed issues #1 and #3 with workarounds but seeks input on whether these are known limitations or architectural problems.

Setup: - Ollama serving llama3.1:8b-instruct-q4 K M via /api/chat streaming - num ctx=4096, temperature 0.2-0.3, num predict 80-150 - FastAPI backend, RAG via Qdrant, feeding a voice agent Retell AI, ~2-11 turns per call and a WhatsApp agent, both in Spanish - Each turn: retrieve top-3 RAG chunks, build a messages array system + full conversation history + latest user turn , call Ollama I’ve hit three related issues in production over the last couple weeks that all seem to come down to how this model handles injected context in a growing multi-turn conversation. Already fixed 1 and 3 with workarounds; still not 100% sure 2 is fully solved, and would love input on whether these are known limitations or if I’m doing something wrong architecturally. — Issue 1 fixed : a second “system” message mid-conversation gets treated as something to respond to, not absorbed silently I was inserting retrieved RAG context as a separate “system” role message right before the latest user turn — i.e. system persona , user, assistant, user, assistant, system RAG context + instruction “use this as background knowledge” , user latest question . Instead of using that context silently, the model would respond directly to the injected instruction, e.g.: User: “¿Enrique Nguix es mi nombre? ¿Te sirve?” Model: “Sí, puedo utilizar la información proporcionada para responder a las preguntas relacionadas con Edentia y sus cursos. ¿Cuál es la pregunta específica que deseas saber?” It also started losing track of the actual conversation — one turn later, asked to recall something said 2 turns earlier, it replied “No hemos tenido una conversación previa sobre este tema” despite the full transcript being in messages . Fix: moved all dynamic context RAG + any persistent memory into the ONE system message at position 0, built fresh each turn, instead of inserting a second system message later in the array. Conversation history now goes in untouched after that single system message. This resolved both symptoms above in testing so far. Question: is this documented/expected behavior for Llama 3.1’s instruct tuning — i.e. is it only trained/reliable with exactly one system message at the very start, and a second system turn later in the conversation is essentially out-of-distribution? Or is this more likely an Ollama chat-template quirk in how multiple system-role messages get rendered into the underlying prompt? — Issue 2 workaround in place, not fully confident it’s solved : model refuses to answer when a retrieved chunk is long/unstructured When a RAG chunk was a longer descriptive paragraph rather than a tight Q&A pair, the model would sometimes respond with something like: “Lo siento, pero no puedo continuar con la conversación debido a que el texto proporcionado parece ser una descripción general de la formación y requisitos para higienistas dentales en España, más que una pregunta o tema específico sobre el cual discutir.” i.e. it’s treating the injected context block itself as the thing it needs to respond to, rather than the actual user question. Workaround: added an explicit prompt instruction telling it the context block is never the question, and rewrote the knowledge base into short, single-answer chunks instead of long paragraphs. This reduced the frequency a lot but I’m not fully confident it’s eliminated, since it’s hard to reproduce reliably. Question: at 8B quantized to Q4 K M, is this kind of “confusion about what to respond to when given a long context block” a known capability ceiling, or is there a better way to structurally signal “this is reference material, never the question” beyond prompt instructions e.g. via message role/ordering, or some other delimiter convention this model was actually trained on ? — Issue 3 fixed : silently truncating conversation history to a small window causes amnesia in longer conversations Was capping conversation history sent to the model at the last 6 messages 3 exchanges to stay well under num ctx=2048. In calls longer than ~3 exchanges common in a support use case , the model would lose the original reason for the call, names given earlier, etc. — because that information had literally fallen out of the window it was given, even though the full transcript existed upstream Retell AI keeps it . Fix: raised num ctx to 4096 and widened the window to the last 16 turns, matching a comparable, simpler agent I run booking-only, no RAG that never truncated history at all and never had this problem. Not really a question here, just flagging it in case it’s useful context for issues 1/2 above — happy to hear if 4096 is still marginal for this kind of use case and I should go higher / consider a summarization fallback for anything older than the window instead of a hard cutoff. — Any pointers — whether that’s “yeah this is a known Llama 3.1 instruct limitation,” “you’re holding the chat template wrong,” or “here’s how people usually structure RAG context for this model” — would be genuinely useful. Happy to share more of the actual system prompt / message construction code if helpful.