{"slug": "an-educational-lab-of-ai-agent-architectures", "title": "An educational lab of AI agent architectures", "summary": "An educational lab demonstrating various AI agent architectures built on LangChain and a local Ollama server has been released. The project includes multiple runnable CLI variants for studying chat with memory, tool calling, RAG, hybrid, and agentic RAG patterns, each configurable via environment variables.", "body_md": "An educational lab of AI agent architectures, built on **LangChain** and a local\n**Ollama** server. Each variant is a separate, runnable CLI so you can study one\nmechanism at a time and watch it work through the logs.\n\n- Model (generation):\n`gemma4:e4b`\n\n- Model (embeddings):\n`mxbai-embed-large:latest`\n\n- Host:\n`http://10.100.102.10:11434`\n\n(configurable) - Chat provider: local Ollama by default, or OpenRouter with\n`LLM_PROVIDER=openrouter`\n\n| Category | Variant | Command |\n|---|---|---|\n| Chat + memory | full buffer | `chat-buffer` |\n| Chat + memory | summary buffer | `chat-summary` |\n| Tool calling | ReAct text protocol | `tools-react` |\n| Tool calling | native `bind_tools` |\n`tools-native` |\n| RAG only | pure numpy cosine | `rag-numpy` |\n| RAG only | Chroma vector DB | `rag-chroma` |\n| RAG only | LlamaIndex (in-memory) | `rag-llamaindex` |\n| RAG only | LlamaIndex + Chroma store | `rag-llamaindex-chroma` |\n| RAG only | Haystack | `rag-haystack` |\n| RAG only | hybrid BM25+dense + cross-encoder rerank | `rag-rerank` |\n| Hybrid (chat + RAG) | semantic router | `hybrid-semantic` |\n| Hybrid (chat + RAG) | LLM classifier router | `hybrid-llm` |\n| Hybrid (chat + RAG) | adaptive RAG (LangGraph) | `hybrid-adaptive` |\n| Hybrid (chat + RAG) | corrective RAG (multi-grader + rewrite) | `hybrid-adaptive-plus` |\n| Agentic RAG | tool-calling + retrieval-as-a-tool | `agentic-rag` |\n\nThe chat commands accept `--persist`\n\nto save memory to SQLite across runs.\n\nInstall dependencies into a virtual environment:\n\n```\nuv sync\n```\n\nConfigure the connection (copy and edit if your host differs):\n\n```\ncopy .env.example .env\n```\n\nTo use OpenRouter for chat generation, set `LLM_PROVIDER=openrouter`\n\n,\n`OPENROUTER_API_KEY`\n\n, and `OPENROUTER_MODEL`\n\nin `.env`\n\n. Embeddings still use\nOllama via `EMBED_MODEL`\n\n, so RAG commands still need the local embedding model.\n\nEvery agent is its own console script. Run any of them with `uv run <command>`\n\n.\n\nChat + memory:\n\n```\nuv run chat-buffer\nuv run chat-summary --persist\n```\n\nTool calling:\n\n```\nuv run tools-react\nuv run tools-native\n```\n\nRAG (one backend each):\n\n```\nuv run rag-numpy\nuv run rag-chroma\nuv run rag-llamaindex\nuv run rag-llamaindex-chroma\nuv run rag-haystack\nuv run rag-rerank\n```\n\nHybrid (chat + RAG):\n\n```\nuv run hybrid-semantic\nuv run hybrid-llm\nuv run hybrid-adaptive\nuv run hybrid-adaptive-plus\n```\n\nAgentic RAG and evaluation:\n\n```\nuv run agentic-rag\nuv run rag-eval\n```\n\nCommon flags: `--no-soul`\n\n(all agents), `--persist`\n\n(chat), `--no-index`\n\nand `--drop`\n\n(RAG/hybrid/agentic).\n\n`chat-buffer`\n\n- a chatbot that remembers everything you said in this chat.`chat-summary`\n\n- a chatbot that remembers a short summary when the chat gets long.`tools-react`\n\n- a bot that can use tools (like a calculator) by writing them out as text.`tools-native`\n\n- the same idea, but it calls tools the \"official\" way the model supports.`rag-numpy`\n\n- answers questions by first reading your documents; the simplest version.`rag-chroma`\n\n- the same, but it saves what it read so it does not re-read every time.`rag-llamaindex`\n\n/`rag-haystack`\n\n- the same idea built with the LlamaIndex / Haystack libraries.`rag-llamaindex-chroma`\n\n- LlamaIndex doing the reading, with the saved store from Chroma.`rag-rerank`\n\n- a smarter search: it looks two ways and then re-sorts results to pick the best.`hybrid-semantic`\n\n/`hybrid-llm`\n\n- decides \"should I read the docs or just chat?\" before answering.`hybrid-adaptive`\n\n- reads the docs first, then checks \"is this useful?\" and chats if not.`hybrid-adaptive-plus`\n\n- the careful version: checks each document, the answer, and retries if needed.`agentic-rag`\n\n- the bot decides by itself when to search the documents, when to use a tool, or just answer. Searching the documents is given to it as one of its \"tools\".`rag-eval`\n\n- not a chat. It is a small test that scores how well each RAG version finds the right document, so you can compare them with numbers.\n\nInside the REPL: type a message, or use slash commands.\n\n`/skills`\n\nlists available skills (a numbered menu).`/<number>`\n\nor`/<name>`\n\nloads a skill into the active context.`/add [path]`\n\ningests a text file into the RAG store live, without restarting. With no path (or an invalid one), it scans`data/corpus/`\n\nand adds any new files not yet indexed. Works for RAG and hybrid agents; persists for Chroma.`/win`\n\nprints the full context window currently sent to the model.`/active`\n\n,`/clear`\n\n,`/rules`\n\n,`/help`\n\n,`/exit`\n\n.\n\nThe tool-calling agents (`tools-react`\n\n, `tools-native`\n\n) can use tools from any\nModel Context Protocol server. Copy the example and edit it:\n\n```\ncopy mcp.json.example mcp.json\n```\n\nEach entry is either a stdio server (`command`\n\n+ `args`\n\n) or an HTTP server\n(`url`\n\n+ `transport`\n\n). On startup the agents connect, load the servers' tools,\nand merge them with the built-ins (built-ins win on name clashes). If `mcp.json`\n\nis absent or a server fails, the agents simply run with the built-in tools.\n\nSet `MCP_CONFIG`\n\nto point at a different config file if needed.\n\n`SOUL.md`\n\n(project root) is a global persona/identity prepended to every agent's system prompt. Every agent accepts`--no-soul`\n\nto skip loading it.- Agents can write durable facts into the managed\n`## Memory`\n\nsection of`SOUL.md`\n\n(between`memory:start`\n\n/`memory:end`\n\nmarkers, append-only + deduped): use the`/remember <fact>`\n\ncommand in any agent, or the`remember`\n\ntool in the tool-calling agents. New memories apply on the next turn (no restart). `rules/`\n\nholds always-on markdown that is injected into every agent's system prompt automatically.`skills/`\n\nholds opt-in markdown skills loaded on demand via the`/`\n\nmenu. Drop a new`.md`\n\nfile in`skills/`\n\n(optionally with`name:`\n\n/`description:`\n\nfrontmatter) and it appears in the menu.\n\nAll four RAG backends implement one `Retriever`\n\ninterface (`index`\n\n/ `search`\n\n)\nand return the same `RetrievedChunk(text, source, score)`\n\n, so the agents are\nidentical regardless of the engine behind them. The flow:\n\n**At startup (index once):**\n\n- Load every file in\n`data/corpus/`\n\nand split it into chunks. - Send the chunks to Ollama\n`mxbai-embed-large`\n\nto get one vector per chunk. - Store the vectors in the backend (numpy matrix, Chroma, LlamaIndex, Haystack).\n\n**Per query (search):**\n\n- Embed the query with the same model.\n- Score every chunk by cosine similarity and take the top-k (default 4).\n\n**Answer (retrieve-then-read):**\n\n- If nothing is retrieved, the agent replies \"I don't know\".\n- Otherwise it builds\n`Context: [source] chunk... / Question: ...`\n\nand sends it to`gemma4:e4b`\n\nwith a system rule: answer**only** from the context and cite sources as`[source]`\n\n. - Retrieval hits (sources + scores) and the LLM call are logged.\n\nTwo defining traits: the RAG-only agents are **stateless** (each query is\nindependent, no conversation memory), and embeddings come from a different model\n(`mxbai-embed-large`\n\n) than generation (`gemma4:e4b`\n\n).\n\n**pure-numpy**(`rag-numpy`\n\n): the most transparent. Chunks by character window with overlap, L2-normalizes vectors, and does a brute-force dot product (`matrix @ query_vec`\n\n) for cosine similarity. No database, O(N) per query - great for understanding, poor for scale.**pure-chroma**(`rag-chroma`\n\n): same idea but vectors live in a**persistent** Chroma collection (`.chroma/`\n\n) that performs the nearest-neighbor search. It is the only backend that survives restarts: if the collection already holds data it is reused instead of re-indexed.**llamaindex**(`rag-llamaindex`\n\n): uses LlamaIndex's`VectorStoreIndex`\n\nwith a sentence splitter and Ollama embeddings/LLM, kept in memory.**llamaindex-chroma**(`rag-llamaindex-chroma`\n\n): same LlamaIndex orchestration, but with a**persistent Chroma** vector store behind it. This shows the common production layering - a framework owns the pipeline while an external, scalable store owns the vectors. Contrast it with`rag-chroma`\n\n, where our own code does the orchestration and Chroma is driven directly.**haystack**(`rag-haystack`\n\n): uses Haystack components (document splitter, in-memory store, embedding retriever) wired into the same interface.\n\nBecause the chunking differs (numpy/chroma split by characters; LlamaIndex and Haystack split by sentences/words), the same question can retrieve slightly different passages across backends - a good thing to compare.\n\n**rerank**(`rag-rerank`\n\n): a production-style retrieval stack on a small local scale. It runs**hybrid retrieval**- dense (Ollama embeddings, cosine) plus sparse (BM25 keyword) - fuses the two rankings with Reciprocal Rank Fusion, then**reranks** the candidates with a cross-encoder (`cross-encoder/ms-marco-MiniLM-L-6-v2`\n\nby default) and keeps the top-k. The cross-encoder gives much sharper relevance separation than raw cosine (clearly positive scores for the right chunk, strongly negative for the rest). The model downloads on first run; configure it via`RERANK_MODEL`\n\nand the candidate pool via`FUSION_CANDIDATES`\n\n.\n\nEvery RAG agent (and every hybrid agent) accepts:\n\n`--no-index`\n\n- start without indexing/uploading the corpus. The in-memory backends start empty; Chroma uses whatever is already persisted.`--drop`\n\n- drop persisted RAG storage before starting (Chroma only; a no-op for the in-memory backends), forcing a clean rebuild.\n\nBy default the in-memory backends index the corpus on every run, while Chroma reuses its persisted collection if present.\n\n```\nuv run rag-chroma --drop\nuv run rag-chroma --no-index\n```\n\n`agentic-rag`\n\nis a different take on combining tools and retrieval: instead of a\nrouter deciding RAG-vs-chat up front, retrieval is exposed to a native\ntool-calling agent as a `search_knowledge_base`\n\ntool (backed by the hybrid\nBM25+dense+rerank engine). The model decides per turn whether to search the\ncorpus, use another tool (calculator, shell, web_search, MCP), or answer\ndirectly. It is instructed to prefer the knowledge base for factual questions.\n\n```\nuv run agentic-rag\n```\n\n**Human-in-the-loop**: dangerous tools (currently`shell`\n\n) require confirmation before running. Controlled by`REQUIRE_TOOL_APPROVAL`\n\n(default true); the user is prompted`run this? [y/N]`\n\nand can deny.**Reliability**: every LLM call retries on transient errors/timeouts (`LLM_MAX_RETRIES`\n\n) and generation/tool calls fall back to a second model (`FALLBACK_MODEL`\n\n, default`llama3.1:8b`\n\n) if the primary fails.**Observability (LangSmith)**: set`LANGSMITH_TRACING=true`\n\nand`LANGSMITH_API_KEY`\n\nto trace every chain/LLM/tool step in LangSmith. No-op when disabled.**Evaluation**:`uv run rag-eval`\n\nruns a small labelled question set against the RAG backends and prints hit-rate and context-recall, for objective comparison and regression checks.\n\n```\nuv run rag-eval\n```\n\nEach agent writes two channels:\n\n- A readable, bordered console stream (no colors or emojis).\n- A structured JSON-lines file at\n`logs/<agent>.jsonl`\n\ncapturing LLM calls (prompt, response, latency, tokens), tool calls, retrieval hits with scores, routing decisions, and errors.\n\nTail a log to monitor an agent, for example:\n\n```\nGet-Content logs/hybrid-adaptive.jsonl -Wait\nsrc/localagent/\n  config.py          settings from .env (host, models, chunking, logging)\n  llm.py             ChatOllama / OllamaEmbeddings factories + token usage\n  logging_setup.py   dual console + JSON logging\n  memory.py          buffer and summary-buffer memory with SQLite persistence\n  skills.py          skill discovery for the / menu\n  rules.py           always-on rules loader\n  cli.py             REPL and slash-command engine\n  tools/             example tools: calculator, shell, read_file, web_search (DuckDuckGo)\n  mcp_tools.py       optional Model Context Protocol tool loading (mcp.json)\n  rag/               common Retriever interface + numpy, chroma, llamaindex, haystack\n  agents/            one module + console script per variant\n```\n\nLangChain orchestrates the agents, memory and routing. Each RAG backend\nimplements the same `Retriever`\n\ninterface (`index`\n\n/ `search`\n\n), so the agents do\nnot care which engine is behind it. The corpus in `data/corpus/`\n\nis indexed at\nstartup for the RAG and hybrid agents.", "url": "https://wpnews.pro/news/an-educational-lab-of-ai-agent-architectures", "canonical_source": "https://github.com/Rudnik-Ilia/Agents-Sandbox", "published_at": "2026-07-11 15:33:22+00:00", "updated_at": "2026-07-11 16:05:17.582891+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["LangChain", "Ollama", "Chroma", "LlamaIndex", "Haystack", "OpenRouter", "Gemma", "mxbai-embed-large"], "alternates": {"html": "https://wpnews.pro/news/an-educational-lab-of-ai-agent-architectures", "markdown": "https://wpnews.pro/news/an-educational-lab-of-ai-agent-architectures.md", "text": "https://wpnews.pro/news/an-educational-lab-of-ai-agent-architectures.txt", "jsonld": "https://wpnews.pro/news/an-educational-lab-of-ai-agent-architectures.jsonld"}}