NEON-CITY / CosySim

A developer built CosySim, a local-first, open AI simulation framework featuring a living cyberpunk city where every NPC is a governed LLM agent with persistent memory, a real economy, and day/night cycles. The system includes 35 launch targets, 1,040 skills, a 38-stage interceptor pipeline, and a training flywheel, all created through agentic coding and published for others to learn from.

OK, this is an old one. This started out as me trying to get realtime voice working 8 months ago. I ended up having to learn spec decoding, chunking, finetuned router models etc. I built up a 1100 test and telemety suite around it, Custom LMStudio clients and server etc. It eventually evolved into what you see today. There is ALOT more here than meets the eye. This is a complete self managed, self improving, self finetuning, self coding local management system with a vector DB with 6 cascades that leverages notebooklm in ways it isn’t exactly meant to be used, With an entire Scene Creation system/Simulator with drag and drop or even single prompt scene generation and asset studio. Every scene uses a local AI for something. All scenes are independant - yet act inside a living, breathing city with a real economy, day/night cycles, alliances, news cycles etc. Each agent has their own phone and can call, text, hack any other agent. You have The same phone. and much much more. It really is just a showcase of what is possible. All old credentials are expired/changed etc, so don’t bother. lol. Anyway, as usual If you any part useful or interesting, feel free to take and use or improve on yourself This is presented here for one purpose only - Providing as much information and idea’s as possible to everyone so people can take inspiration or find something useful and advance the field in some way A local-first, open AI simulation framework where every NPC is a real, governed LLM agent — and the world remembers. 35 launch targets · ~1,040 skills · 38-stage interceptor pipeline · 6-tier knowledge router · a training flywheel — built almost entirely through agentic coding , and published so humans and AI agents can learn from it. Why this repo exists.CosySim is meant to beread. It is a working, end-to-end example of what local agents + agentic coding can build: a living cyberpunk city whose residents reason on a local model, recall the past from a persistent knowledge base, react to a live economy and faction war, and quietly turn every interaction into training data that improves the next one. Take any piece you like — the interceptor pipeline, the LMStudio steering, the NLM↔Nexus flywheel, the ARGUS toolkit — and use it in your own project. Pick the door that matches why you came: You want to… | Go to | Deep-dive doc | |---|---|---| Run it in 5 minutes | | docs/OPERATIONS.md docs/ARCHITECTURE.md content/scenes/ docs/MCP FRAMEWORK.md docs/NEXUS.md engine/training/ , training/ docs/ API REFERENCE.md docs/ARGUS METHODOLOGY.md docs/DESIGN SYSTEM V2.md docs/INDEX.md Prerequisites:Python 3.13, LMStudio running on :1234 with a chat model loaded. Optional: ComfyUI :8188 for image/video, a TTS server :8600 for voice. 1. Install pip install -r requirements.txt && npm install 2. Configure secrets nothing real is committed — see "Security & configuration" cp .env.example .env then fill in any keys you have; LMStudio works with no auth 3. Launch python tui.py interactive Terminal UI recommended — ←/→/↑/↓ to navigate, Enter to launch python launcher.py --core or: auto-start core services + main scenes python launcher.py neoncity or: a single scene → http://localhost:5563 python launcher.py --list see all 35 targets + live port status Then open the hub at http://localhost:8500 — the NEON CITY landing page — and jack in. Handy python cli.py ask "prompt" query the local model stack 38 models python scripts/oracle.py full system diagnostic health · errors · perf python scripts/smart test.py --smoke fast test sweep ~15 files This repo is safe to fork : no live credentials are committed. Real secrets live only in gitignored local files. .env .env.example engine/config.py . LMStudio needs no auth by default, so an empty .env is enough to run the world. config/default.yaml ${ENV VAR} at read time via a SecretManager hook. config/nlm rpcids.yaml config/nlm rpcids.example.yaml The NEONCITY “Dark Renaissance” landing — the front door to a city of local agents. CosySim is a local-first, open AI simulation framework where every NPC is a real, governed LLM agent — and the world they live in actually remembers. It runs 35 launch targets 18 game scenes + 11 services + 6 creation tools as Flask/Socket.IO servers on your own machine, powered entirely by local inference LMStudio , a persistent knowledge layer Nexus KMS , and NotebookLM distillation. No cloud API is required for core gameplay. It is built to be read. This is a flagship example of what agentic coding + local agents can produce: ~1,040 skills across 99 packs, a 36-stage interceptor pipeline, a 6-tier knowledge router, and a training flywheel — all wired together and documented so that humans and AI agents alike can learn from it and borrow from it . The elevator pitch: Spin up a neon cyberpunk city on your laptop. Talk to its residents. They reason with a local model, recall what you did last week from a persistent knowledge base, react to a live economy and faction war, and quietly turn every conversation into training data that makes the next conversation better. Claim | How it’s actually true in the code | |---|---| Every NPC is a real local agent | Each reply runs through AgentGovernor.reply in engine/mcp/comms framework.py → VirtualAgent → LMSClient.infer stream against LMStudio on localhost:1234 . No scripted dialog trees. | The city remembers | State is a live tree MCPFramework singleton: scenes → characters → world → factions . Persistent memory, rules, and 3.7K+ Q&A pairs live in Nexus KMS SQLite + FTS5, :8700 . | Frontier-level results from local models | The 6-tier NexusQueryRouter answers from cache/vector/FTS first and only falls back to local inference last — then auto-stores the answer. NotebookLM free Gemini, grounded sits between as a distillation tier. Every interaction makes the next one cheaper and sharper. | It governs itself | 36 interceptors engine/agents/interceptors/ inject mood, knowledge, scene rules, faction standing, and heat awareness — then shape, log, and sync the response. Skills enforce cooldowns, costs, and prerequisites. | Open and inspectable | Vanilla-JS frontend no build step , Google-style docstrings, version-stamped change logs, and a deep docs/ tree with INDEX.md as the door. | CosySim is a reusable engine + swappable content + tunable config . Scenes subclass BaseScene ; the engine provides agents, governance, knowledge, world simulation, and inference; YAML config tunes behavior without touching code. ┌──────────────────────────────────────────────────────────────────────────┐ │ BROWSER — Neon HUD v2 vanilla JS · Jinja2 · Socket.IO client │ │ cosysim-telemetry.js · cosysim-particles.js · design tokens.css v2 │ └─────────────────────────────────┬────────────────────────────────────────┘ │ Socket.IO / REST ┌─────────────────────────────────▼────────────────────────────────────────┐ │ 35 LAUNCH TARGETS ports 5555–8800 │ │ ┌── GAME 18 ──────┐ ┌── SERVICE 11 ─────┐ ┌── CREATION 6 ──────┐ │ │ │ penthouse, neoncity│ │ nexus kms, hub, │ │ canvas, asset studio,│ │ │ │ oracle, casino, │ │ tts, command center,│ │ creation kit, │ │ │ │ heist, tavern, … │ │ intel hub, … │ │ creator, … │ │ │ └───────────────────┘ └─────────────────────┘ └──────────────────────┘ │ └─────────────────────────────────┬────────────────────────────────────────┘ │ ┌─────────────────────────────────▼────────────────────────────────────────┐ │ SKILLS + MCP PIPELINE │ │ engine/skills/builtin/ ~1,040 @skill across 99 packs │ │ engine/mcp/ MCPFramework state tree · 43 tool modules · DialogSystem │ │ engine/agents/interceptors/ 36 pre/post hooks AgentGovernor │ └─────────────────────────────────┬────────────────────────────────────────┘ │ ┌─────────────────────────────────▼────────────────────────────────────────┐ │ ENGINE LAYER engine/ │ │ lmstudio/ ServerController · LMLink federation · StreamProcessor │ │ nexus/ NexusClient · 6-tier QueryRouter · KnowledgePipeline · NLM │ │ world/ WorldSim economy ticks · PlayerState · Missions · Crew │ │ agents/ VirtualAgent · AgentGovernor · AgentRouter · OutputEvaluator │ │ training/ DataCollector · BenchmarkRunner · FinetuneOrchestrator │ └─────────────────────────────────┬────────────────────────────────────────┘ │ external processes ┌─────────────────────────────────▼────────────────────────────────────────┐ │ LMStudio :1234 · Nexus KMS :8700 · ComfyUI :8188 · TTS :8600 │ │ local LLM knowledge image/video Qwen3 voice │ └──────────────────────────────────────────────────────────────────────────┘ The engine subsystems map cleanly onto folders, so the diagram doubles as a directory map: Layer | Where | Does what | |---|---|---| Browser HUD | content/shared/ , scene templates/ + static/ | Neon HUD v2, particles, telemetry — pure vanilla JS, no build step | Targets | content/scenes/{name}/ , engine/control plane registry.py | 35 Flask/FastAPI/Streamlit/Node servers, each a BaseScene subclass | Skills + MCP | engine/skills/ , engine/mcp/ | @skill capabilities, the MCPFramework state tree, 43 @mcp tool modules, governance | Engine | engine/lmstudio · nexus · world · agents · training | inference, knowledge, simulation, agent lifecycle, the data flywheel | External | LMStudio · Nexus KMS · ComfyUI · TTS | Local inference, knowledge backbone, media generation, voice | This is the heart of the system — the path a single message takes is the same for every NPC in every scene. It is implemented in AgentGovernor.reply engine/mcp/comms framework.py and the interceptor pipeline. Player message ──Socket.IO / REST──► Scene ──► DialogSystem.add turn │ ▼ ┌─ AgentGovernor.reply ────────────────────────────────────────────────┐ │ │ │ 1. Build ResponseContext scene, agent id, user message, manifest │ │ │ │ 2. AUTO skills run BEFORE the LLM; cooldown + prereq gated │ │ e.g. search memory → result injected into context │ │ │ │ 3. run pre ─ ~21 PRE interceptors, ascending priority ───────────► │ │ 5 MoodDrift · 6 NexusPrompt hydrate knowledge │ │ 8 CharRegistry · 10 Router pick model · 15 Scene context │ │ 40 FactionContext · 50 PersonalityGuard · 60 PolicyEnforcer │ │ any PRE interceptor may set ctx "abort" / ctx "skip llm" │ │ │ │ 4. LLM CALL VirtualAgent.reply governance context │ │ └► LMSClient.infer stream ──SSE──► LMStudio :1234 │ │ │ │ 5. Parse StreamProcessor / ContentRouter extract inline tags: │ │ MOOD:happy IMAGE:prompt ACTION:x STAT:health+10 VOICE │ │ │ │ 6. run post ─ ~7 POST interceptors ─────────────────────────────► │ │ 75 HeatAwareness · 80 ResponseShaper · 85 TTS · 90 Logger │ │ 92 MoodSync/SpectatorBroadcast · 93 Relationship │ └─────────────────────────────────────┬───────────────────────────────────┘ ▼ STATE WRITES + CASCADE • CharacterRegistry mood / relationship deltas • PlayerState credits / rep / heat STAT:… tags • ComfyUI image queue IMAGE:… tags • OutputEvaluator.score → DataCollector training flywheel • WorldSim / EventCascade → broadcast to other subscribed scenes ▼ Scene emits via Socket.IO: chat message · HUD update · portrait mood · TTS audio A few details that make this more than a wrapper: ResponseContext is a mutable bag system prompt , messages , reply , auto results , plus post-call metadata like mood tags , response id , and is stateful . Interceptors read and write it in priority order; any PRE interceptor can abort the LLM call entirely. engine/mcp/comms framework.py : auto run before the call, result injected , optional offered to the model as a tool , and required the model must call it . Auto-skill invocation is throttled by a real COOLDOWN TRACKER and prerequisite chain v1.59.0 . STAT:health+10 , MOOD:x , IMAGE:prompt inline, parsed by StreamProcessor RE STAT = \ STAT: \w+ +- \d+ \ and routed to game state, mood, and the ComfyUI queue. WorldSim tick ~60s emits SimEvent s economy / faction shift / NPC action / weather ; EventCascade broadcasts to every subscribed scene, so an NPC’s action in one room ripples through the HUD and faction standings everywhere. The city breathes. Six factions fight for control. The night never ends.— content/scenes/neoncity/neoncity scene.py NEON CITY is CosySim’s flagship: not a single screen but a persistent, autonomous world that the GAME-pillar scenes all plug into. The economy keeps moving, factions keep fighting over turf, NPCs keep walking their daily routines, and the weather keeps rolling through neon-soaked rain — whether or not a player is logged in. When you do walk in, the world has a memory: your last heist raised the heat, your faction standing changes the prices you’re quoted, and a botched job last night is still rippling through three other scenes. The whole thing runs on local inference LMStudio . No cloud, no API keys for the simulation itself — a city full of agents reasoning on your own GPU. The GAME pillar is a set of independent Flask/Socket.IO scenes each on its own port, inheriting BaseScene , but they are windows into the same world state , not separate games. NEON CITY :5563 is the hub; the rest are districts and venues you move through. Scene | Display name | What it is | |---|---|---| neoncity | NEON CITY | Living-world hub: economy, factions, missions, crew, cyberspace, board mode | phone | SIGNAL | Cyberdeck — messaging, 0xGH0ST contacts, mini-games, the always-on HUD | penthouse | THE PENTHOUSE | 3D character room three.js r184 , curtain-wall skyline, autonomous NPCs | heist | THE HEIST | Crew-driven heist runs that feed heat and faction consequences back into the city | arena | THE COLOSSEUM | Combat matches; bookmaking and fighter queues spawned by the world sim | casino | CLUB NOIR | Gambling, VIP access gated by faction standing | lounge | THE VELVET PIT | Speakeasy social scene, ambient events, brokered deals | tavern | THE RUSTY ANCHOR | Fantasy RPG tavern, barter economy, dice | realm | LitRPG world | Dual-agent Director + companion , d20 combat, inventory | grid | THE GRID | District board / strategy layer over territory control | games | THE ARCADE | Trivia, chess puzzles, leaderboards, AI opponents | | … | plus gallery , cyberspace , and the broader GAME catalogue | The authoritative catalogue lives in engine/control plane registry.py resolved to ports by engine/port registry.py ; the launcher, TUI, and the in-browser THE TERMINAL catalogue all derive their lists from it. See docs/SCENES.md https://github.com/nihilistau/CosySim/blob/master/docs/SCENES.md for the full pillar tables. NEON CITY’s “aliveness” comes from background daemon threads in engine/world/ that tick on a game clock 1 real second ≈ 1 game minute; 60 real seconds = 1 game hour . WorldSim engine/world/world sim.py is the event engine. It fires scripted-but-stochastic events on per-task intervals — NPC actions every ~60s, ambient mood shifts, faction shifts, 0xGH0ST hacker messages, arena match queues, major world events, and passive economy ticks. Every SimEvent is a appended to a 200-entry ring buffer, b persisted to world sim history, and c broadcast on the EventBus . A get digest scene call returns “what you missed” the moment you enter a scene. LivingWorld engine/world/living world.py is the orchestrator. Each tick it coordinates every subsystem in order: 1. Game clock → read time-of-day from WorldState 2. NPC routines → RoutineManager moves NPCs to scheduled locations 3. Faction AI → each faction makes one strategic decision every 5th tick 4. Market tick → supply/demand drift, territory-weighted prices 5. Weather cycle → Markov transition CLEAR→NEON RAIN→STORM→BLACKOUT… 6. Stochastic events → fire + propagate consequences to market & NPCs The subsystems, by fileEarlier versions simulated a world but the simulation was largely cosmetic — agents could say STAT:arousal+10 , a faction could “expand territory”, and nothing actually changed . The v1.59 “consequential world” and v1.60 “Living Systems” passes closed those loops. This is the part worth studying: it’s where a pile of independent systems became a world with cause and effect. Loop | Before | After the closed loop | Where | |---|---|---|---| Stat tags applied | STAT:trust-5 parsed then discarded | StatSyncInterceptor priority 91 writes tags to real character state before mood rules evaluate them | engine/agents/interceptors/stat sync.py | Economy settlement | Buying/selling was flavour text | Market. settle buy/ settle sell debit the wallet, move inventory, and raise heat for illegal goods | engine/world/market.py | World→market shocks | World events never reached the market | Market.subscribe to world events maps gang war→weapons up , festival→luxury up , shortage→category surge | living world. init subsystems → market.py | Faction gating | Standing only gated casino VIP | faction gates gives every scene shop access, ally discounts −10% , rival surcharges +15% , and standing-locked missions | engine/world/faction gates.py | Player-aware factions | Factions ignored the player | Faction AI weights decisions by your standing — allies expand near you, rivals raid your turf; shifts broadcast on NEONCITY FACTION SHIFT | engine/world/faction ai.py | Mission consequences | Missions resolved in isolation | MissionManager. apply consequences applies rep/heat/faction-control deltas on success and failure; chains branch on outcome & standing | mission.py + mission chains.py | Equipment matters | Equipped gear gave +0 | get equipment bonuses aggregates equipped items into skill/stat deltas that skill-checks consume | engine/world/equipment effects.py | Cross-scene ripples | Events stayed local | EventCascade fans WorldSim events to subscribed scenes e.g. a CRIME event reaches phone , tavern , casino , heist | engine/world/event cascade.py | The net effect: a heist that goes wrong raises city-wide heat, which HeatAwarenessInterceptor pipeline priority 75 makes NPCs aware of; the resulting faction shift reprices the black market; and the next mission in the chain unlocks a different branch because your standing changed. Every magic number here is config-driven mission.consequences. , economy.event shocks. , territory.faction ai. so the whole consequence economy is tunable without touching code.