{"slug": "building-a-streaming-local-ai-agent", "title": "Building a Streaming Local AI Agent", "summary": "A new tutorial by an unnamed author demonstrates building a streaming local AI agent that monitors Wikipedia's live edit feed for vandalism using Ollama, with a two-stage funnel to filter events before invoking a local LLM. The agent consumes Wikipedia's public EventStreams endpoint without API keys and streams output token by token, addressing both meanings of streaming in AI agents. The build uses Python 3.11+, FastAPI, and Ollama with a model like llama3.1:8b, and was fully tested before publication.", "body_md": "# Building a Streaming Local AI Agent\n\nStreaming gets used in two different ways when people talk about AI agents. Straighten out your understanding here.\n\n\"**Streaming**\" gets used in two different ways when people talk about AI agents, and most tutorials only build one of them. Sometimes it means the agent consumes a live stream of events instead of waiting for someone to type a message. Sometimes it means the agent's own output streams out token by token instead of appearing all at once after a long pause. This build does both, on purpose, because they solve two different problems, and a genuinely useful always-on agent needs both solved.\n\nThe framing worth borrowing here comes from what's usually called an ambient agent, one [LangChain describes as triggered by events rather than by a human message](https://www.langchain.com/blog/introducing-ambient-agents), and Google's Agent Development Kit describes from the infrastructure side the same way: agents woken by something arriving on a stream, not sitting behind a request-response call. The scenario for this build is concrete and genuinely real: a local agent that watches Wikipedia's live, public edit feed, no API key required, and reasons about which edits look like vandalism, running entirely on your own machine through Ollama. Every line of code below was written, then actually tested, before it went into this article.\n\nThese are your prerequisites:\n\n- Python 3.11 or newer\n[Ollama](https://ollama.com/)installed locally, with a model pulled (`ollama pull llama3.1:8b`\n\n, or any model that supports structured JSON output)`pip install fastapi uvicorn httpx pydantic ollama sse-starlette`\n\n- No API keys, no cloud account, and no cost beyond your own electricity. The only outbound network connection this service makes is to Wikipedia's public EventStreams endpoint, which requires no authentication\n\n## # The One Design Decision That Matters\n\nWikipedia's edit stream isn't a trickle. On an active day, it pushes several edits per second across every language edition combined. Hand every single one of those to a language model and two things happen at once: you burn through your machine's compute on edits that were never interesting in the first place, and the agent falls behind the live stream it's supposed to be watching, which defeats the entire point of building something \"**always on.**\"\n\nThe fix is a two-stage funnel, and it's the single most important idea in this build:\n\n- Stage one is cheap, plain Python math that runs on every event with no model involved at all: how many bytes did this edit remove, how many edits has this user made in the last couple of minutes? The overwhelming majority of edits are boring, and boring is free to detect\n- Stage two, the actual local LLM, only wakes up for the small fraction of events that trip a threshold in stage one. This is the same principle behind any good monitoring system: cheap filters up front, expensive reasoning reserved for the candidates that survive\n\n### // Folder Structure\n\n```\nstreaming-local-agent/\n├── src/\n│   ├── __init__.py\n│   ├── config.py\n│   ├── schemas.py\n│   ├── stream_source.py\n│   ├── filters.py\n│   ├── agent.py\n│   ├── broadcaster.py\n│   └── main.py\n├── tests/\n│   └── test_filters.py\n├── requirements.txt\n└── .env.example\n```\n\nEach file maps to exactly one stage of the pipeline described above, which makes the whole thing easy to reason about and easy to test in isolation, which is exactly how it was actually built for this article.\n\n## # Build Section 1: The Event Stream Consumer\n\nWikipedia's EventStreams service pushes edits as Server-Sent Events over plain HTTP. No key, no handshake beyond an ordinary **GET** request that stays open.\n\n``` python\n# src/stream_source.py\nimport asyncio\nimport json\nimport re\nimport time\nfrom typing import AsyncIterator, Optional\nimport httpx\n\nfrom .schemas import RecentChangeEvent\nfrom . import config\n\n# Wikipedia doesn't send an explicit \"is this user anonymous\" flag on this\n# stream; anonymous edits are attributed to the editor's IP address instead\n# of a username, so an IP-shaped username is how you detect one in practice.\n_IPV4_RE = re.compile(r\"^\\d{1,3}(\\.\\d{1,3}){3}$\")\n_IPV6_RE = re.compile(r\"^[0-9A-Fa-f:]+:[0-9A-Fa-f:]+$\")\n\ndef is_anonymous_user(username: str) -> bool:\n    return bool(_IPV4_RE.match(username) or _IPV6_RE.match(username))\n\ndef parse_sse_line(line: str) -> Optional[dict]:\n    \"\"\"SSE frames data as lines prefixed with 'data: '. Comment lines\n    (starting with ':') and blank keep-alive lines are common on this\n    feed and should be silently ignored, not treated as errors.\"\"\"\n    if not line or line.startswith(\":\"):\n        return None\n    if line.startswith(\"data:\"):\n        raw = line[len(\"data:\"):].strip()\n        if not raw:\n            return None\n        try:\n            return json.loads(raw)\n        except json.JSONDecodeError:\n            return None\n    return None\n\ndef to_event(raw: dict) -> Optional[RecentChangeEvent]:\n    \"\"\"Converts a raw Wikimedia payload into our normalized schema.\n    Returns None for event types we don't care about rather than\n    raising, since a stream this high-volume constantly includes shapes\n    we're not watching for.\"\"\"\n    if raw.get(\"type\") != \"edit\":\n        return None\n    length = raw.get(\"length\") or {}\n    if \"old\" not in length or \"new\" not in length:\n        return None\n    return RecentChangeEvent(\n        wiki=raw.get(\"wiki\", \"unknown\"),\n        user=raw.get(\"user\", \"unknown\"),\n        title=raw.get(\"title\", \"unknown\"),\n        is_anonymous=is_anonymous_user(raw.get(\"user\", \"\")),\n        is_bot=raw.get(\"bot\", False),\n        old_length=length[\"old\"],\n        new_length=length[\"new\"],\n        timestamp=raw.get(\"timestamp\", time.time()),\n        comment=raw.get(\"comment\", \"\") or \"\",\n    )\n\nasync def wikipedia_event_stream() -> AsyncIterator[RecentChangeEvent]:\n    \"\"\"The live async generator used by main.py. Reconnects automatically\n    on a dropped connection rather than letting the whole service die\n    because of one network hiccup, which matters a lot for something\n    meant to run unattended.\"\"\"\n    while True:\n        try:\n            async with httpx.AsyncClient(timeout=None) as client:\n                async with client.stream(\"GET\", config.WIKIPEDIA_STREAM_URL) as response:\n                    async for line in response.aiter_lines():\n                        raw = parse_sse_line(line)\n                        if raw is None:\n                            continue\n                        if raw.get(\"wiki\") not in config.WATCHED_WIKIS:\n                            continue\n                        event = to_event(raw)\n                        if event is not None:\n                            yield event\n        except httpx.HTTPError:\n            await asyncio.sleep(5)\n```\n\n**What this does:** anonymity detection here is worth calling out specifically, because the naive approach (checking for an explicit \"is anonymous\" field) doesn't actually exist on this feed.\n\nWikipedia attributes anonymous edits to the editor's IP address as their username, so `is_anonymous_user`\n\nchecks whether the username is shaped like an IPv4 or IPv6 address instead, which is how this detection genuinely works in production. `parse_sse_line`\n\nand to_event are both deliberately pure functions with no network dependency, which is what lets me test the parsing logic directly against realistic sample payloads before ever touching a live connection, catching a real bug in an earlier draft of the anonymity check in the process.\n\n`wikipedia_event_stream`\n\nwraps the actual connection in a `while True`\n\nwith a reconnect-and-sleep on any **HTTP error**, since an always-on service that dies on the first dropped connection isn't actually always-on.\n\n## # Build Section 2: The Cheap Filter, Stage One\n\n``` python\n# src/filters.py\nimport time\nfrom collections import defaultdict, deque\nfrom typing import Optional\n\nfrom .schemas import RecentChangeEvent, FilterSignal\nfrom . import config\n\nclass EditVelocityTracker:\n    \"\"\"Tracks recent edit timestamps per user in a sliding window, so the\n    filter can catch rapid-fire editing bursts, not just single large\n    deletions. Bounded memory: old users get evicted, not kept forever.\"\"\"\n\n    def __init__(self, window_seconds: int = config.EDIT_VELOCITY_WINDOW_SECONDS,\n                 max_tracked: int = config.MAX_TRACKED_WINDOWS):\n        self.window_seconds = window_seconds\n        self.max_tracked = max_tracked\n        self._history: dict[str, deque[float]] = defaultdict(deque)\n\n    def record_and_count(self, user: str, timestamp: float) -> int:\n        \"\"\"Records this edit and returns how many edits this user has\n        made within the trailing window, including this one.\"\"\"\n        history = self._history[user]\n        history.append(timestamp)\n\n        cutoff = timestamp - self.window_seconds\n        while history and history[0] < cutoff:\n            history.popleft()\n\n        if len(self._history) > self.max_tracked:\n            self._evict_oldest()\n\n        return len(history)\n\n    def _evict_oldest(self) -> None:\n        oldest_user = min(self._history, key=lambda u: self._history[u][-1] if self._history[u] else 0)\n        del self._history[oldest_user]\n\nclass Stage1Filter:\n    \"\"\"Wraps the velocity tracker and the byte-removal check into one\n    pass/fail decision per event.\"\"\"\n\n    def __init__(self, tracker: Optional[EditVelocityTracker] = None):\n        self.tracker = tracker or EditVelocityTracker()\n\n    def evaluate(self, event: RecentChangeEvent) -> Optional[FilterSignal]:\n        \"\"\"Returns a FilterSignal if this event is worth the LLM's time,\n        otherwise None, and None is the common case by a wide margin.\"\"\"\n        if event.is_bot:\n            return None  # bot edits have their own, separate review path\n\n        recent_count = self.tracker.record_and_count(event.user, event.timestamp)\n        bytes_removed = event.bytes_removed\n\n        reasons = []\n        if bytes_removed >= config.BYTES_REMOVED_THRESHOLD:\n            reasons.append(f\"removed {bytes_removed} bytes in one edit\")\n        if recent_count >= config.EDIT_VELOCITY_THRESHOLD:\n            reasons.append(f\"{recent_count} edits in {self.tracker.window_seconds}s\")\n\n        if not reasons:\n            return None\n\n        return FilterSignal(\n            event=event, bytes_removed=bytes_removed,\n            recent_edit_count=recent_count, reason=\"; \".join(reasons),\n        )\n```\n\n**What this does:** `EditVelocityTracker`\n\nkeeps a per-user deque of recent edit timestamps and trims anything outside the trailing window on every single call, which is what makes \"**5 edits in 2 minutes**\" a real, continuously accurate number rather than an approximation.\n\nThe `max_tracked`\n\neviction guard exists because this dictionary would otherwise grow forever on a stream that never stops, a detail that's easy to skip in a demo and expensive to discover in production. `Stage1Filter.evaluate`\n\nis the actual gate: it returns `None, meaning \"`\n\n**not interesting,**\" for the overwhelming majority of events, and only builds a `FilterSignal`\n\nobject when a real threshold is crossed.\n\n## # Build Section 3: The Local Reasoner, Stage Two\n\nOnly signals that survive Stage 1 reach here. This is where a strict schema and token streaming both matter.\n\n``` python\n# src/schemas.py\nfrom __future__ import annotations\nfrom pydantic import BaseModel, Field\n\nclass RecentChangeEvent(BaseModel):\n    wiki: str\n    user: str\n    title: str\n    is_anonymous: bool\n    is_bot: bool\n    old_length: int\n    new_length: int\n    timestamp: float\n    comment: str = \"\"\n\n    @property\n    def bytes_removed(self) -> int:\n        return max(0, self.old_length - self.new_length)\n\nclass FilterSignal(BaseModel):\n    event: RecentChangeEvent\n    bytes_removed: int\n    recent_edit_count: int\n    reason: str\n\nclass AgentVerdict(BaseModel):\n    \"\"\"The structured judgment we force the local model to return.\n    Constraining this with a schema is what makes the output usable in\n    code rather than just readable by a human.\"\"\"\n    is_likely_vandalism: bool\n    severity: int = Field(ge=1, le=5, description=\"1 = probably fine, 5 = high confidence vandalism\")\n    reasoning: str\n    suggested_action: str\n\n# src/agent.py\nfrom typing import AsyncIterator\nimport ollama\n\nfrom .schemas import FilterSignal, AgentVerdict\nfrom . import config\n\nSYSTEM_PROMPT = \"\"\"You are a Wikipedia edit-monitoring assistant. You will be \\\nshown metadata about an edit that tripped an automated filter for a large \\\ndeletion or unusually rapid editing. Decide whether this looks like likely \\\nvandalism or a legitimate edit (a rewrite, a cleanup, a merge). Respond with \\\na JSON object matching the required schema. Be specific in your reasoning, \\\nreference the actual numbers you were given.\"\"\"\n\ndef _build_user_prompt(signal: FilterSignal) -> str:\n    e = signal.event\n    return (\n        f\"Page: {e.title}\\n\"\n        f\"User: {e.user} ({'anonymous' if e.is_anonymous else 'registered'})\\n\"\n        f\"Bytes removed: {signal.bytes_removed}\\n\"\n        f\"Recent edit count by this user: {signal.recent_edit_count}\\n\"\n        f\"Edit summary left by user: \\\"{e.comment or '(none)'}\\\"\\n\"\n        f\"Trigger reason: {signal.reason}\\n\"\n    )\n\nasync def evaluate_signal(signal: FilterSignal) -> AsyncIterator[str | AgentVerdict]:\n    \"\"\"Streams the model's raw output as it's generated (str chunks), then\n    yields a final validated AgentVerdict once the stream completes. The\n    caller tells the two apart with isinstance().\"\"\"\n    client = ollama.AsyncClient(host=config.OLLAMA_HOST)\n\n    stream = await client.chat(\n        model=config.OLLAMA_MODEL,\n        messages=[\n            {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n            {\"role\": \"user\", \"content\": _build_user_prompt(signal)},\n        ],\n        format=AgentVerdict.model_json_schema(),\n        stream=True,\n        options={\"temperature\": 0.1},\n    )\n\n    full_text = \"\"\n    async for chunk in stream:\n        piece = chunk[\"message\"][\"content\"]\n        full_text += piece\n        if piece:\n            yield piece  # live token, for the broadcaster to forward immediately\n\n    verdict = AgentVerdict.model_validate_json(full_text)\n    yield verdict\n```\n\n**What this does:** `format=AgentVerdict.model_json_schema()`\n\nis the detail that makes this a senior-grade agent rather than a chatbot with extra steps. Ollama enforces that schema directly on generation, so the completed response is guaranteed valid JSON matching AgentVerdict, not \"**usually valid JSON I then have to defensively parse.**\" `evaluate_signal`\n\nstill streams every raw chunk out as it arrives, yielding plain strings for live display, and only yields the final, validated `AgentVerdict`\n\nobject once the full stream completes, which is what lets a connected client watch the reasoning appear in real time while the calling code downstream still gets a fully type-checked object to act on.\n\n## # Build Section 4: Broadcasting Live Reasoning to Clients\n\n``` python\n# src/broadcaster.py\nimport asyncio\nimport json\nfrom typing import AsyncIterator\n\nclass Broadcaster:\n    def __init__(self, max_queue_size: int = 100):\n        self._subscribers: set[asyncio.Queue] = set()\n        self.max_queue_size = max_queue_size\n\n    def subscribe(self) -> asyncio.Queue:\n        queue: asyncio.Queue = asyncio.Queue(maxsize=self.max_queue_size)\n        self._subscribers.add(queue)\n        return queue\n\n    def unsubscribe(self, queue: asyncio.Queue) -> None:\n        self._subscribers.discard(queue)\n\n    async def publish(self, payload: dict) -> None:\n        \"\"\"Fans a payload out to every subscriber. A subscriber whose\n        queue is full gets the message dropped rather than blocking the\n        whole pipeline, a slow client should never be able to slow down\n        the agent's actual processing loop.\"\"\"\n        message = json.dumps(payload)\n        for queue in list(self._subscribers):\n            try:\n                queue.put_nowait(message)\n            except asyncio.QueueFull:\n                continue\n\n    async def stream(self) -> AsyncIterator[str]:\n        \"\"\"An async generator a caller can loop over to receive messages,\n        used directly by the SSE endpoint in main.py.\"\"\"\n        queue = self.subscribe()\n        try:\n            while True:\n                message = await queue.get()\n                yield message\n        finally:\n            self.unsubscribe(queue)\n```\n\n**What this does:** each connected client gets its own `asyncio.Queue`\n\n, and `publish`\n\nfans a message out to every queue independently using `put_nowait`\n\nwrapped in a `try/except`\n\n, so one slow or stalled subscriber degrades gracefully by silently dropping a message for that client instead of ever blocking the loop that's actually processing live Wikipedia edits. That separation matters more than it looks like it should: without it, a single slow browser tab could quietly stall the entire agent. One genuinely useful thing testing this surfaced: `stream()`\n\nis an async generator, and async generators are lazy; the `subscribe()`\n\ncall inside it doesn't actually run until something first calls `__anext__()`\n\non it. In the real FastAPI endpoint, this is a non-issue since iteration starts immediately, but it's exactly the kind of subtlety that catches people writing their own tests for this pattern, and it caught mine on the first attempt before I fixed the test itself.\n\n## # Wiring It Together\n\n``` python\n# src/main.py\nimport asyncio\nimport logging\nfrom contextlib import asynccontextmanager\n\nfrom fastapi import FastAPI, Request\nfrom sse_starlette.sse import EventSourceResponse\n\nfrom .broadcaster import Broadcaster\nfrom .filters import Stage1Filter\nfrom .stream_source import wikipedia_event_stream\nfrom .agent import evaluate_signal\nfrom .schemas import AgentVerdict\n\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(\"streaming-local-agent\")\n\nbroadcaster = Broadcaster()\nstage1 = Stage1Filter()\n\nasync def run_pipeline() -> None:\n    \"\"\"Consumes the live stream forever, runs stage 1 on every event,\n    and only calls the LLM stage on events that survive it.\"\"\"\n    async for event in wikipedia_event_stream():\n        signal = stage1.evaluate(event)\n        if signal is None:\n            continue\n\n        logger.info(\"Stage 1 flagged: %s by %s (%s)\", signal.event.title, signal.event.user, signal.reason)\n        await broadcaster.publish({\"type\": \"flagged\", \"title\": signal.event.title, \"reason\": signal.reason})\n\n        try:\n            async for item in evaluate_signal(signal):\n                if isinstance(item, str):\n                    await broadcaster.publish({\"type\": \"token\", \"title\": signal.event.title, \"text\": item})\n                elif isinstance(item, AgentVerdict):\n                    await broadcaster.publish({\n                        \"type\": \"verdict\", \"title\": signal.event.title, \"user\": signal.event.user,\n                        **item.model_dump(),\n                    })\n        except Exception:\n            logger.exception(\"Stage 2 failed for %s, skipping this signal\", signal.event.title)\n\n@asynccontextmanager\nasync def lifespan(app: FastAPI):\n    task = asyncio.create_task(run_pipeline())\n    logger.info(\"Streaming local agent started, watching for edits...\")\n    yield\n    task.cancel()\n    logger.info(\"Streaming local agent shutting down\")\n\napp = FastAPI(title=\"Streaming Local Agent\", lifespan=lifespan)\n\n@app.get(\"/events\")\nasync def events(request: Request):\n    async def event_generator():\n        async for message in broadcaster.stream():\n            if await request.is_disconnected():\n                break\n            yield message\n    return EventSourceResponse(event_generator())\n\n@app.get(\"/health\")\ndef health():\n    return {\"status\": \"ok\"}\n```\n\n**What this does:** `run_pipeline`\n\nis the actual spine of the whole service; everything above is a supporting cast. It's wrapped in a `try/except`\n\naround the Stage 2 call specifically, so one malformed model response or one Ollama hiccup logs an error and moves on to the next event instead of silently killing the background task and leaving the agent running but permanently blind.\n\nThe `lifespan`\n\ncontext manager starts that pipeline as a background task the moment the app boots and cancels it cleanly on shutdown, the correct modern FastAPI pattern rather than the older `@app.on_event`\n\ndecorators. The `/events`\n\nroute is where everything converges: opening it streams every `flagged`\n\n, `token`\n\n, and `verdict`\n\nmessage live as newline-delimited SSE data, and checking `request.is_disconnected()`\n\non every loop means a closed browser tab gets cleaned up instead of leaking a queue forever.\n\n### // How to Run It\n\nWith Ollama installed and a model pulled:\n\n```\nollama pull llama3.1:8b\nollama serve   # if it isn't already running as a background service\n```\n\nThen, from the project root:\n\n```\npython -m venv venv\nsource venv/bin/activate\npip install -r requirements.txt\nuvicorn src.main:app --reload\n```\n\nWith that running, open a second terminal and watch the live feed:\n\n```\ncurl -N http://localhost:8000/events\n```\n\nOr point a browser tab at `http://localhost:8000/events`\n\ndirectly; most browsers render an SSE stream as plain text arriving incrementally. Within a few minutes on an active wiki, you should see `flagged`\n\nmessages arrive as Stage 1 catches large deletions or edit bursts, followed by a stream of `token`\n\nmessages as the local model reasons about it live, ending in a `verdict`\n\nmessage with a structured severity score. Boring edits, the vast majority of the traffic, never appear at all, which is exactly the point.\n\n## # A Note on Scaling This Up\n\nThe in-process `asyncio.Queue`\n\nbroadcaster and the single background task in this build are the right amount of infrastructure for one machine watching one stream. At real production scale, watching multiple sources, running multiple consumer processes, surviving a service restart without losing in-flight events, the natural upgrade is swapping the direct stream connection and in-memory broadcaster for a real message bus like Kafka sitting between the producer and the reasoning stage.\n\n## # Wrapping Up\n\nThe actual lesson underneath all of this code isn't about Wikipedia, or Ollama, or FastAPI specifically, it's that efficiency stops being an optimization you bolt on later, the moment an agent goes from \"**answers when asked**\" to \"** always on**.\" A chat agent that sits idle costs nothing. A streaming agent is, by definition, always consuming something, and every design choice in this build, the two-stage funnel, the bounded-memory eviction, the graceful degradation on a slow subscriber, the automatic reconnect on a dropped connection, exists because an always-on system that can't sustain itself indefinitely isn't actually done, no matter how well it worked in the first five minutes you watched it run.\n\nis a software engineer and technical writer passionate about leveraging cutting-edge technologies to craft compelling narratives, with a keen eye for detail and a knack for simplifying complex concepts. You can also find Shittu on\n\n[Shittu Olumide](https://www.linkedin.com/in/olumide-shittu/)", "url": "https://wpnews.pro/news/building-a-streaming-local-ai-agent", "canonical_source": "https://www.kdnuggets.com/building-a-streaming-local-ai-agent", "published_at": "2026-08-13 14:00:15+00:00", "updated_at": "2026-08-13 14:35:19.130962+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-infrastructure", "developer-tools"], "entities": ["LangChain", "Google", "Ollama", "Wikipedia", "FastAPI", "Python", "llama3.1:8b"], "alternates": {"html": "https://wpnews.pro/news/building-a-streaming-local-ai-agent", "markdown": "https://wpnews.pro/news/building-a-streaming-local-ai-agent.md", "text": "https://wpnews.pro/news/building-a-streaming-local-ai-agent.txt", "jsonld": "https://wpnews.pro/news/building-a-streaming-local-ai-agent.jsonld"}}