NEON-CITY / CosySim

OK, this is an old one. This started out as me trying to get realtime voice working 8 months ago. I ended up having to learn spec decoding, chunking, finetuned router models etc. I built up a 1100 test and telemety suite around it, Custom LMStudio clients and server etc. It eventually evolved into what you see today. There is ALOT more here than meets the eye. This is a complete self managed, self improving, self finetuning, self coding local management system with a vector DB with 6 cascades that leverages notebooklm in ways it isn’t exactly meant to be used, With an entire Scene Creation system/Simulator with drag and drop or even single prompt scene generation and asset studio. Every scene uses a local AI for something. All scenes are independant - yet act inside a living, breathing city with a real economy, day/night cycles, alliances, news cycles etc.

Each agent has their own phone and can call, text, hack any other agent. You have The same phone. and much much more. It really is just a showcase of what is possible. All old credentials are expired/changed etc, so don’t bother. lol.

Anyway, as usual If you any part useful or interesting, feel free to take and use or improve on yourself! This is presented here for one purpose only - Providing as much information and idea’s as possible to everyone so people can take inspiration or find something useful and advance the field in some way!

A local-first, open AI simulation framework where every NPC is a real, governed LLM agent — and the world remembers.

35 launch targets · ~1,040 skills · 38-stage interceptor pipeline · 6-tier knowledge router · a training flywheel —

built almost entirely through agentic coding, and published so humans and AI agents can learn from it.

Why this repo exists.CosySim is meant to beread. It is a working, end-to-end example of what local agents + agentic coding can build: a living cyberpunk city whose residents reason on a local model, recall the past from a persistent knowledge base, react to a live economy and faction war, and quietly turn every interaction into training data that improves the next one. Take any piece you like — the interceptor pipeline, the LMStudio steering, the NLM↔Nexus flywheel, the ARGUS toolkit — and use it in your own project.

Pick the door that matches why you came:

You want to… | Go to | Deep-dive doc | |---|---|---| Run it in 5 minutes | |

docs/OPERATIONS.md

docs/ARCHITECTURE.md

content/scenes/

docs/MCP_FRAMEWORK.md

docs/NEXUS.md

engine/training/

, training/

docs/*_API_REFERENCE.md

docs/ARGUS_METHODOLOGY.md

docs/DESIGN_SYSTEM_V2.md

docs/INDEX.md

Prerequisites:Python 3.13,[LMStudio]running on:1234

with a chat model loaded. Optional:[ComfyUI](:8188

) for image/video, a TTS server (:8600

) for voice.

pip install -r requirements.txt && npm install

cp .env.example .env          # then fill in any keys you have; LMStudio works with no auth

python tui.py                 # interactive Terminal UI (recommended) — ←/→/↑/↓ to navigate, Enter to launch
python launcher.py --core     # or: auto-start core services + main scenes
python launcher.py neoncity   # or: a single scene → http://localhost:5563
python launcher.py --list     # see all 35 targets + live port status

Then open the hub at ** http://localhost:8500** — the NEON CITY landing page — and jack in.

python cli.py ask "prompt"           # query the local model stack (38 models)
python scripts/oracle.py             # full system diagnostic (health · errors · perf)
python scripts/smart_test.py --smoke # fast test sweep (~15 files)

This repo is safe to fork: no live credentials are committed. Real secrets live only in gitignored local files.

.env

.env.example

engine/config.py

. LMStudio needs no auth by default, so an empty .env

is enough to run the world.config/default.yaml

${ENV_VAR}

at read time via a SecretManager

hook.config/nlm_rpcids.yaml

config/nlm_rpcids.example.yaml

The NEONCITY “Dark Renaissance” landing — the front door to a city of local agents.

CosySim is a local-first, open AI simulation framework where every NPC is a real, governed LLM agent — and the world they live in actually remembers. It runs 35 launch targets (18 game scenes + 11 services + 6 creation tools) as Flask/Socket.IO servers on your own machine, powered entirely by local inference (LMStudio), a persistent knowledge layer (** Nexus KMS**), and NotebookLM distillation. No cloud API is required for core gameplay.

It is built to be read. This is a flagship example of what agentic coding + local agents can produce: ~1,040 skills across 99 packs, a 36-stage interceptor pipeline, a 6-tier knowledge router, and a training flywheel — all wired together and documented so that humans and AI agents alike can learn from it and borrow from it.

The elevator pitch:

Spin up a neon cyberpunk city on your laptop. Talk to its residents. They reason with a local model, recall what you did last week from a persistent knowledge base, react to a live economy and faction war, and quietly turn every conversation into training data that makes the next conversation better.

Claim | How it’s actually true (in the code) | |---|---| Every NPC is a real local agent | Each reply runs through AgentGovernor.reply() in engine/mcp/comms_framework.py → VirtualAgent → LMSClient.infer_stream() against LMStudio on localhost:1234 . No scripted dialog trees. | The city remembers | State is a live tree (MCPFramework singleton: scenes → characters → world → factions). Persistent memory, rules, and 3.7K+ Q&A pairs live in Nexus KMS (SQLite + FTS5, :8700 ). | Frontier-level results from local models | The 6-tier NexusQueryRouter answers from cache/vector/FTS first and only falls back to local inference last — then auto-stores the answer. NotebookLM (free Gemini, grounded) sits between as a distillation tier. Every interaction makes the next one cheaper and sharper. | It governs itself | 36 interceptors (engine/agents/interceptors/ ) inject mood, knowledge, scene rules, faction standing, and heat awareness — then shape, log, and sync the response. Skills enforce cooldowns, costs, and prerequisites. | Open and inspectable | Vanilla-JS frontend (no build step), Google-style docstrings, version-stamped change logs, and a deep docs/ tree with INDEX.md as the door. |

CosySim is a reusable engine + swappable content + tunable config. Scenes subclass BaseScene

; the engine provides agents, governance, knowledge, world simulation, and inference; YAML config tunes behavior without touching code.

┌──────────────────────────────────────────────────────────────────────────┐
│  BROWSER — Neon HUD v2  (vanilla JS · Jinja2 · Socket.IO client)          │
│  cosysim-telemetry.js · cosysim-particles.js · design_tokens.css (v2)     │
└─────────────────────────────────┬────────────────────────────────────────┘
                                  │  Socket.IO / REST
┌─────────────────────────────────▼────────────────────────────────────────┐
│  35 LAUNCH TARGETS  (ports 5555–8800)                                     │
│  ┌── GAME (18) ──────┐ ┌── SERVICE (11) ─────┐ ┌── CREATION (6) ──────┐   │
│  │ penthouse, neoncity│ │ nexus_kms, hub,     │ │ canvas, asset_studio,│   │
│  │ oracle, casino,    │ │ tts, command_center,│ │ creation_kit,        │   │
│  │ heist, tavern, …   │ │ intel_hub, …        │ │ creator, …           │   │
│  └───────────────────┘ └─────────────────────┘ └──────────────────────┘   │
└─────────────────────────────────┬────────────────────────────────────────┘
                                  │
┌─────────────────────────────────▼────────────────────────────────────────┐
│  SKILLS  +  MCP PIPELINE                                                   │
│  engine/skills/builtin/  ~1,040 @skill across 99 packs                     │
│  engine/mcp/  MCPFramework state tree · 43 tool modules · DialogSystem     │
│  engine/agents/interceptors/  36 pre/post hooks (AgentGovernor)           │
└─────────────────────────────────┬────────────────────────────────────────┘
                                  │
┌─────────────────────────────────▼────────────────────────────────────────┐
│  ENGINE LAYER  (engine/)                                                   │
│   lmstudio/  ServerController · LMLink federation · StreamProcessor        │
│   nexus/     NexusClient · 6-tier QueryRouter · KnowledgePipeline · NLM    │
│   world/     WorldSim (economy ticks) · PlayerState · Missions · Crew      │
│   agents/    VirtualAgent · AgentGovernor · AgentRouter · OutputEvaluator  │
│   training/  DataCollector · BenchmarkRunner · FinetuneOrchestrator        │
└─────────────────────────────────┬────────────────────────────────────────┘
                                  │  external processes
┌─────────────────────────────────▼────────────────────────────────────────┐
│  LMStudio :1234  ·  Nexus KMS :8700  ·  ComfyUI :8188  ·  TTS :8600       │
│  (local LLM)        (knowledge)         (image/video)     (Qwen3 voice)    │
└──────────────────────────────────────────────────────────────────────────┘

The engine subsystems map cleanly onto folders, so the diagram doubles as a directory map:

Layer | Where | Does what | |---|---|---| Browser HUD | content/shared/ , scene templates/ + static/ | Neon HUD v2, particles, telemetry — pure vanilla JS, no build step | Targets | content/scenes/{name}/ , engine/control_plane_registry.py | 35 Flask/FastAPI/Streamlit/Node servers, each a BaseScene subclass | Skills + MCP | engine/skills/ , engine/mcp/ | @skill capabilities, the MCPFramework state tree, 43 @mcp_tool modules, governance | Engine | engine/lmstudio · nexus · world · agents · training | inference, knowledge, simulation, agent lifecycle, the data flywheel | External | LMStudio · Nexus KMS · ComfyUI · TTS | Local inference, knowledge backbone, media generation, voice |

This is the heart of the system — the path a single message takes is the same for every NPC in every scene. It is implemented in AgentGovernor.reply()

(engine/mcp/comms_framework.py

) and the interceptor pipeline.

Player message  ──Socket.IO / REST──►  Scene  ──►  DialogSystem.add_turn()
        │
        ▼
 ┌─ AgentGovernor.reply() ────────────────────────────────────────────────┐
 │                                                                         │
 │  1. Build ResponseContext   (scene, agent_id, user_message, manifest)   │
 │                                                                         │
 │  2. AUTO skills             run BEFORE the LLM; cooldown + prereq gated  │
 │       e.g. search_memory → result injected into context                 │
 │                                                                         │
 │  3. run_pre()  ─ ~21 PRE interceptors, ascending priority ───────────►  │
 │       5  MoodDrift  ·  6 NexusPrompt (hydrate knowledge)                 │
 │       8  CharRegistry  ·  10 Router (pick model)  ·  15 Scene context    │
 │       40 FactionContext  ·  50 PersonalityGuard  ·  60 PolicyEnforcer    │
 │       (any PRE interceptor may set ctx["abort"] / ctx["skip_llm"])       │
 │                                                                         │
 │  4. LLM CALL   VirtualAgent.reply(governance_context)                    │
 │                  └► LMSClient.infer_stream()  ──SSE──►  LMStudio :1234    │
 │                                                                         │
 │  5. Parse   StreamProcessor / ContentRouter extract inline tags:        │
 │       [MOOD:happy]  [IMAGE:prompt]  [ACTION:x]  [STAT:health+10] [VOICE] │
 │                                                                         │
 │  6. run_post()  ─ ~7 POST interceptors ─────────────────────────────►   │
 │       75 HeatAwareness · 80 ResponseShaper · 85 TTS · 90 Logger          │
 │       92 MoodSync/SpectatorBroadcast · 93 Relationship                   │
 └─────────────────────────────────────┬───────────────────────────────────┘
                                       ▼
   STATE WRITES + CASCADE
     • CharacterRegistry mood / relationship deltas
     • PlayerState credits / rep / heat   ([STAT:…] tags)
     • ComfyUI image queue                ([IMAGE:…] tags)
     • OutputEvaluator.score → DataCollector  (training flywheel)
     • WorldSim / EventCascade → broadcast to other subscribed scenes
                                       ▼
   Scene emits via Socket.IO:  chat message · HUD update · portrait mood · TTS audio

A few details that make this more than a wrapper:

ResponseContext

is a mutable bagsystem_prompt

, messages

, reply

, auto_results

, plus post-call metadata like mood_tags

, response_id

, and is_stateful

. Interceptors read and write it in priority order; any PRE interceptor can abort the LLM call entirely.engine/mcp/comms_framework.py

): auto

(run before the call, result injected), optional

(offered to the model as a tool), and required

(the model must call it). Auto-skill invocation is throttled by a real COOLDOWN_TRACKER

and prerequisite chain (v1.59.0).[STAT:health+10]

, [MOOD:x]

, [IMAGE:prompt]

inline, parsed by StreamProcessor

(_RE_STAT = \[STAT:(\w+)([+-]\d+)\]

) and routed to game state, mood, and the ComfyUI queue.WorldSim

tick (~60s) emits SimEvent

s (economy / faction shift / NPC action / weather); EventCascade

broadcasts to every subscribed scene, so an NPC’s action in one room ripples through the HUD and faction standings everywhere.

The city breathes. Six factions fight for control. The night never ends.—content/scenes/neoncity/neoncity_scene.py

NEON CITY is CosySim’s flagship: not a single screen but a persistent, autonomous world that the GAME-pillar scenes all plug into. The economy keeps moving, factions keep fighting over turf, NPCs keep walking their daily routines, and the weather keeps rolling through neon-soaked rain — whether or not a player is logged in. When you do walk in, the world has a memory: your last heist raised the heat, your faction standing changes the prices you’re quoted, and a botched job last night is still rippling through three other scenes.

The whole thing runs on local inference (LMStudio). No cloud, no API keys for the simulation itself — a city full of agents reasoning on your own GPU.

The GAME pillar is a set of independent Flask/Socket.IO scenes (each on its own port, inheriting BaseScene

), but they are windows into the same world state, not separate games. NEON CITY (:5563

) is the hub; the rest are districts and venues you move through.

Scene | Display name | What it is | |---|---|---| neoncity | NEON CITY | Living-world hub: economy, factions, missions, crew, cyberspace, board mode | phone | SIGNAL | Cyberdeck — messaging, 0xGH0ST contacts, mini-games, the always-on HUD | penthouse | THE PENTHOUSE | 3D character room (three.js r184), curtain-wall skyline, autonomous NPCs | heist | THE HEIST | Crew-driven heist runs that feed heat and faction consequences back into the city | arena | THE COLOSSEUM | Combat matches; bookmaking and fighter queues spawned by the world sim | casino | CLUB NOIR | Gambling, VIP access gated by faction standing | lounge | THE VELVET PIT | Speakeasy social scene, ambient events, brokered deals | tavern | THE RUSTY ANCHOR | Fantasy RPG tavern, barter economy, dice | realm | LitRPG world | Dual-agent (Director + companion), d20 combat, inventory | grid | THE GRID | District board / strategy layer over territory control | games | THE ARCADE | Trivia, chess puzzles, leaderboards, AI opponents | | … | plus gallery , cyberspace , and the broader GAME catalogue |

The authoritative catalogue lives in engine/control_plane_registry.py

(resolved to ports by engine/port_registry.py

); the launcher, TUI, and the in-browser THE TERMINAL catalogue all derive their lists from it. See docs/SCENES.md for the full pillar tables.

NEON CITY’s “aliveness” comes from background daemon threads in engine/world/

that tick on a game clock (1 real second ≈ 1 game minute; 60 real seconds = 1 game hour).

** WorldSim** (

engine/world/world_sim.py

) is the event engine. It fires scripted-but-stochastic events on per-task intervals — NPC actions every ~60s, ambient mood shifts, faction shifts, 0xGH0ST hacker messages, arena match queues, major world events, and passive economy ticks. Every SimEvent

is (a) appended to a 200-entry ring buffer, (b) persisted to world_sim

history, and (c) broadcast on the EventBus

. A get_digest(scene)

call returns “what you missed” the moment you enter a scene.** LivingWorld** (

engine/world/living_world.py

) is the orchestrator. Each tick it coordinates every subsystem in order:

1. Game clock        → read time-of-day from WorldState
2. NPC routines      → RoutineManager moves NPCs to scheduled locations
3. Faction AI        → each faction makes one strategic decision (every 5th tick)
4. Market tick       → supply/demand drift, territory-weighted prices
5. Weather cycle     → Markov transition (CLEAR→NEON_RAIN→STORM→BLACKOUT…)
6. Stochastic events → fire + propagate consequences to market & NPCs

The subsystems, by fileEarlier versions simulated a world but the simulation was largely cosmetic — agents could say [STAT:arousal+10]

, a faction could “expand territory”, and nothing actually changed. The v1.59 “consequential world” and v1.60 “Living Systems” passes closed those loops. This is the part worth studying: it’s where a pile of independent systems became a world with cause and effect.

Loop | Before | After (the closed loop) | Where | |---|---|---|---| Stat tags applied | [STAT:trust-5] parsed then discarded | StatSyncInterceptor (priority 91) writes tags to real character state before mood rules evaluate them | engine/agents/interceptors/stat_sync.py | Economy settlement | Buying/selling was flavour text | Market._settle_buy/_settle_sell debit the wallet, move inventory, and raise heat for illegal goods | engine/world/market.py | World→market shocks | World events never reached the market | Market.subscribe_to_world_events() maps gang_war→weapons up , festival→luxury up , shortage→category surge | living_world._init_subsystems → market.py | Faction gating | Standing only gated casino VIP | faction_gates gives every scene shop access, ally discounts (−10%), rival surcharges (+15%), and standing-locked missions | engine/world/faction_gates.py | Player-aware factions | Factions ignored the player | Faction AI weights decisions by your standing — allies expand near you, rivals raid your turf; shifts broadcast on NEONCITY_FACTION_SHIFT | engine/world/faction_ai.py | Mission consequences | Missions resolved in isolation | MissionManager._apply_consequences applies rep/heat/faction-control deltas on success and failure; chains branch on outcome & standing | mission.py + mission_chains.py | Equipment matters | Equipped gear gave +0 | get_equipment_bonuses aggregates equipped items into skill/stat deltas that skill-checks consume | engine/world/equipment_effects.py | Cross-scene ripples | Events stayed local | EventCascade fans WorldSim events to subscribed scenes (e.g. a CRIME event reaches phone , tavern , casino , heist ) | engine/world/event_cascade.py |

The net effect: a heist that goes wrong raises city-wide heat, which HeatAwarenessInterceptor

(pipeline priority 75) makes NPCs aware of; the resulting faction shift reprices the black market; and the next mission in the chain unlocks a different branch because your standing changed. Every magic number here is config-driven (mission.consequences.*

, economy.event_shocks.*

, territory.faction_ai.*

) so the whole consequence economy is tunable without touching code.

source & further reading

discuss.huggingface.co — original article Rakarrack-0.6.1 port making progress! ( AI assisted ) Cloud Storage Poll Welcome to Haiku basic(Haiku Docs, Haiku slide and Haiku sheets)