Most people think building a support chatbot is about generating human-like text. It isn’t. The real problem is memory. Or more specifically, the complete lack of it.
I got tired of prompt engineering hacks and started looking for something better. I didn’t want a bot that sounded smart for one message and then immediately forgot everything like it hit its head on a table.
That’s how IRIS (Intelligent Recall & Issue Support) started.
Instead of building yet another chatbot, I built a multi-tenant API service that plugs into existing e-commerce systems and turns them into context-aware support agents. Because apparently, remembering what a customer said five minutes ago is still considered advanced technology.
Most bots I tested had the same fatal flaw: every interaction felt like a first date. A customer complains three times about a delayed package, and the bot still asks for the order number again like it has goldfish-level cognition. Not ideal.
So the goal became simple: give the agent memory across sessions, not just within a single chat window.
What IRIS Does (And Why It’s Actually Useful)
IRIS is a stateful API layer built with FastAPI that sits between:
The customer chat interface
Order Management Systems (OMS)
A large language model
Its job is to route messages, manage multi-tenant authentication, and most importantly, maintain long-term state.
The system follows an integration-first approach. Businesses don’t want another dashboard. They want something that quietly works in the background without forcing their team to learn yet another UI.
So IRIS is headless. One API. Multiple tenants. Strict data isolation per brand.
The architecture is built around three core pillars:
We use Groq-hosted llama3-70b-8192 for fast inference. Speed matters because the system performs multiple internal steps before responding.
Connectors like Shopify and REST APIs fetch real-time order data. This prevents hallucinations like “your package is delivered” when it’s still sitting in a warehouse.
Instead of stuffing entire chat histories into a vector database or relying on massive context windows, I used Hindsight for structured agent memory.
How the System Actually Works
When a message comes in:
The system queries the memory layer for:
Customer history
Behavioral profile
It fetches live order data from OMS.
It checks a global incident stream:
Are multiple users reporting the same issue?
All of this context is compiled into a structured prompt.
The LLM generates a response.
If an action is required (like a refund), it outputs structured JSON.
The backend extracts that JSON, executes the action, and removes it from the user-visible response.
The interaction is stored back into memory.
Simple idea. Annoyingly complex execution.
The Real Challenge: Memory Architecture
The hardest part wasn’t calling an LLM API. That’s easy.
The real challenge was structuring memory so the system could be:
Personalized per user
Aware of system-wide issues
Initially, I tried dumping everything into one vector store per tenant. That went about as well as you’d expect.
The agent started mixing user contexts. User A’s complaint showed up in User B’s conversation. Not great unless you enjoy chaos.
So I switched to a segmented memory model using Hindsight.
Two-Layer Memory Design
Each user gets their own memory space:
Communication style
Past issues
Preferences
Tenant-wide memory that tracks trends:
Issue spikes
System outages
Common failures
This separation completely changed the system behavior. It stopped reacting and started anticipating.
Code Walkthrough
Message Processing
async def process_message(tenant_id: str, user_id: str, message: str):
issue_type = detect_issue_type(message)
if issue_type:
await report_to_global_memory(tenant_id, issue_type)
active_incident = await check_active_incidents(tenant_id, issue_type)
if active_incident:
context["incident_alert"] = f"Known issue: {active_incident.description}"
customer_history = await memory_client.recall(
bank_id=f"{tenant_id}_user_{user_id}",
query=message,
limit=5
)
profile = await memory_client.reflect(
bank_id=f"{tenant_id}_user_{user_id}"
)
return await generate_llm_response(message, customer_history, profile, context)
The key part here is reflect. Instead of dumping raw chat logs into the LLM, we generate a summarized profile. This reduces token usage and improves accuracy.
Storing Interactions
async def retain_interaction(tenant_id: str, user_id: str, user_msg: str, agent_response: str):
bank_id = f"{tenant_id}user{user_id}"
await memory_client.retain(
bank_id=bank_id,
content=f"Customer: {user_msg}\nAgent: {agent_response}",
metadata={"timestamp": get_current_time()}
)
Nothing fancy here. Just disciplined storage instead of hoping the LLM remembers things out of kindness.
Action Execution
def extract_and_execute_action(llm_response: str, order_data: dict):
match = re.search(r'
', llm_response, re.DOTALL)
if match:
try:
action_req = json.loads(match.group(1))
if action_req.get("action") == "initiate_refund":
execute_refund(action_req.get("order_id"))
clean_response = llm_response.replace(match.group(0), "").strip()
return clean_response, True
except json.JSONDecodeError:
pass
return llm_response, False
Instead of trusting the LLM to behave, we force it into a structured output format and clean things up afterward. Because optimism is not a system design strategy.
Results: Before vs After
Without memory:
User: Where is my replacement?
Bot: Please provide your order number.
With IRIS:
User: Where is my replacement?
IRIS: I see we initiated a replacement for order #12345 yesterday due to damage. It shipped this morning via UPS (Tracking: 1Z9999) and should arrive by Thursday.
That’s the difference between a chatbot and a support system.
Incident Detection in Action
During a simulated outage:
First few users → normal troubleshooting
After pattern detection → immediate acknowledgment
Response shifts to:
“We’re currently experiencing checkout issues. Our team is working on it. I’ll notify you once it’s resolved.”
This avoids hundreds of duplicate support tickets and reduces system load.
Lessons Learned
1. State > Intelligence
LLMs are smart, but without memory they’re basically polite amnesiacs. Memory must be a core architectural component, not an afterthought.
2. Summarize, Don’t Dump
Raw transcripts degrade performance. Reflection-based summaries improve both accuracy and efficiency.
3. Separate Responsibilities
Don’t ask the LLM to generate text and execute logic perfectly at the same time. Use structured outputs and backend validation.
4. Speed Matters
Fast inference (Groq) makes multi-step workflows viable. Slow systems can’t afford layered reasoning.
Final Thought
Building a stateful agent isn’t about crafting the perfect prompt. It’s about building the infrastructure that ensures the model always receives the right context at the right time.
Everything else is just clever wording wrapped around forgetfulness.