{"slug": "how-llms-remember-schedule-and-stream-tokens", "title": "how LLMs remember, schedule, and stream tokens", "summary": "A developer built a small CPU-based inference runtime called kutty-vllm in Python and NumPy to understand how large language models remember, schedule, and stream tokens, implementing a KV cache with block-based allocation to reduce memory waste. The project, detailed in a blog post, demonstrates prefill and decode phases and uses a block table to map logical token positions to physical memory blocks, similar to virtual memory.", "body_md": "when i started this, i thought inference was basically this:\n\n```\nwhile True:\n    logits = model(tokens)\n    token = logits[-1].argmax()\n    tokens.append(token)\n```\n\nwhich is correct. technically.\n\nbut, this tells us nothing about what happens when two people use the model at the same time. or where the model keeps all the previous tokens. or how ChatGPT starts printing before the complete response is ready.\n\nthe transformer only predicts the next token. there is a whole system around it that remembers the past, decides whose token to generate next, and sends that token back to the right person.\n\ni wanted to understand that part. so over the last four months, i built a small version of it on the CPU using Python and NumPy.\n\ni called it [kutty-vllm](https://github.com/ssenthilnathan3/kutty-vllm). “kutty” means small in Tamil :)\n\n## starting with a transformer\n\nthe first version was not an inference runtime. it was just a decoder-only transformer.\n\ntokens went through an embedding table, attention, and a feed-forward layer. i used RMSNorm before attention, and RoPE to give each token a position. NumPy handled all the matrix multiplications.\n\nthe weights were random because i only wanted to check if the implementation was correct. natural language could come later.\n\nand… it worked.\n\n```\nlogits = model.forward([257, 72, 105])\nprint(logits.shape)\n\n# (3, 259)\n```\n\nthen i wrote the obvious generation loop. every time the model generated a token, i passed the entire sequence through the model again.\n\n```\nOnce upon a time\nOnce upon a time + token 1\nOnce upon a time + token 1 + token 2\nOnce upon a time + token 1 + token 2 + token 3\n```\n\nthe problem is that the old tokens do not change. their keys and values were already computed in the previous pass. i was throwing them away and calculating the same thing again.\n\nthis is what the KV cache fixes.\n\n## remembering old tokens\n\nevery attention layer creates a query, key, and value for the current token. the query is only needed right now. but the key and value will be needed by every token that comes after it.\n\nso instead of throwing them away, i stored them.\n\n```\ncache.write(layer, position, key, value)\nkeys, values = cache.read(layer, position + 1)\n```\n\nnow the model has two different jobs.\n\n**prefill** processes the prompt and fills the cache.\n\n**decode** processes one new token using everything already in the cache.\n\ni kept the original slow implementation around for testing. if cached decoding was correct, it had to produce the same logits as recomputing the whole sequence.\n\n```\nexpected = model.forward(tokens)\nactual = np.stack([model.decode(token, cache) for token in tokens])\n\nnp.testing.assert_allclose(actual, expected, rtol=3e-5, atol=3e-5)\n```\n\nthis test was useful because attention bugs don’t always crash. usually, every shape is valid and the model happily gives you the wrong numbers.\n\n## the cache became an allocator\n\nmy first KV cache was one large NumPy array per request.\n\n```\nshape = (num_layers, max_seq_len, num_heads, head_dim)\nself.keys = np.empty(shape, dtype=np.float32)\nself.values = np.empty(shape, dtype=np.float32)\n```\n\neasy enough. but it allocates `max_seq_len`\n\neven when the request uses only a few tokens.\n\nsuppose the model supports 2,048 tokens. one user asks a short question and generates 20 tokens. that request still gets space for all 2,048.\n\nwith enough users, most of the KV memory would just be empty.\n\nthe fix looked a lot like virtual memory. i divided the cache into equal-sized blocks. a request sees its tokens as one continuous sequence, but those tokens can live in unrelated physical blocks.\n\n```\nrequest A:\n\nlogical block     0     1     2\n                  |     |     |\nphysical block    7     2     9\n```\n\neach request only needs a block table.\n\n```\nblock = block_table[position // block_size]\noffset = position % block_size\n\npool.keys[block, layer, offset] = key\n```\n\nwhen the request finishes, blocks 7, 2, and 9 go back into the free list. they don’t have to stay together. the next request can reuse any of them.\n\nthis is the idea behind paged KV caches. production runtimes have kernels that read the blocks directly during attention. mine gathers them into a NumPy array first, which is slower, but made the memory management visible.\n\nthis was also the point where model inference quietly turned into allocator debugging.\n\none version could leak blocks if a reservation failed halfway through. say a request needed three blocks and only two were free. it allocated those two, failed on the third, and never returned the first two.\n\ni fixed it by checking the complete reservation before allocating anything.\n\n```\nblocks_needed = logical_block + 1 - len(block_table)\n\nif blocks_needed > pool.free_blocks:\n    raise CacheExhausted(...)\n```\n\nthe operation had to either allocate everything or allocate nothing.\n\n## okay, now two users\n\nonce requests had their own caches, i could run more than one.\n\nthe naive version picked a few requests and made a batch. this works when they all start together and generate the same number of tokens. real requests obviously don’t do that.\n\nimagine these two:\n\n```\nA: 10 prompt tokens, wants 100 output tokens\nB: 400 prompt tokens, wants 5 output tokens\n```\n\nB finishes almost immediately. in a static batch, its row still occupies a slot until A finishes.\n\nthen request C arrives while A is still running. should C wait for the whole batch to end? there is an empty row sitting right there.\n\ncontinuous batching means there isn’t really a fixed batch. before every decode step, the scheduler looks at which requests can run **right now**.\n\n```\nstep 1: [A, B]\nstep 2: [A, B]\n           B finishes\nstep 3: [A, C]\nstep 4: [A, C]\n```\n\nmy scheduler ended up with a waiting queue and a running list.\n\n```\nwhile waiting and len(running) < max_batch_size:\n    request = waiting[0]\n\n    if blocks_required(request) > cache.free_blocks:\n        break\n\n    waiting.popleft()\n    running.append(request)\n```\n\nafter every step, completed requests leave the running list and release their cache blocks. on the next step, another request can use both the slot and the memory.\n\nthere was a deadlock in this too :)\n\nif a request needed more blocks than the entire cache contained, it sat at the front of the queue forever. it could never run. and because the scheduler was FIFO, nobody behind it could run either.\n\nthe engine looked busy because `waiting`\n\nwas not empty. but every call to `step()`\n\ndid nothing.\n\nnow `add_request()`\n\nrejects a request if its prompt and output budget cannot possibly fit.\n\n## my “batch” was a for loop\n\nafter continuous batching worked, i looked at the model call and noticed this:\n\n```\nfor request in running:\n    request.next_logits = model.decode(token, request.cache)\n```\n\ni had built a batched scheduler that executed every request one by one.\n\nnot exactly batching.\n\nthe large operations in a transformer are matrix multiplications. instead of multiplying one hidden vector at a time, i could stack all active requests and multiply them together.\n\nattention was the annoying part. every request had a different context length. each one also had a different block table.\n\nso the dense projections run as a batch, while attention reads each request’s cache separately.\n\n```\nqkv = normalized_batch @ layer.qkv\n\nfor row, cache in enumerate(caches):\n    keys, values = cache.read(layer, positions[row] + 1)\n    attention_rows.append(attend(q[row], keys, values))\n\nx = residual + np.stack(attention_rows) @ layer.out\n```\n\nthere is still a Python loop in attention. it is not going to compete with a fused C++ or CUDA kernel. but QKV, output projection, MLP, and vocabulary projection now operate on the actual batch.\n\nmore importantly, the scheduler calls the model once per step instead of once per request.\n\n## streaming fell out of the scheduler\n\ni expected token streaming to need a separate design. it didn’t.\n\none engine step already generates one token for every running request. i just had to return those tokens instead of hiding them inside the engine.\n\n```\n@dataclass\nclass TokenOutput:\n    request_id: str\n    token_id: int\n    text: str\n    finished: bool\n```\n\n`stream()`\n\nrepeatedly advances the shared engine and yields events belonging to one request.\n\n``` python\ndef stream(self, request_id):\n    while not request.finished:\n        for output in self.step():\n            if output.request_id == request_id:\n                yield output\n```\n\nthe important bit is that `self.step()`\n\nadvances everyone. streaming request A does not create a private generation loop for A. requests B and C continue moving through the same batches.\n\ncancellation fits into the same setup. if the request is waiting, remove it from the queue. if it is running, remove it from the batch and release its blocks.\n\n## everything worked, and the output was nonsense\n\nat this point the cache worked. batching worked. streaming worked. the demo printed this:\n\n```\n\\x14\\xf8\\x1c-A7\\xaf\"\\xe8\\xd0\\x0dm\\x16>R^f\n```\n\nwell… the model still had random weights.\n\ni was also using a byte tokenizer, so random token IDs became random bytes. some were control characters. one was a carriage return, which moved the cursor to the beginning of the terminal and overwrote the prompt. for a while i thought streaming was broken.\n\nescaping the bytes made the bug understandable, but it didn’t make the model useful.\n\nto generate language, i needed pretrained weights and the exact architecture those weights expected.\n\ni used TinyStories-1M. it is a small GPT-Neo model trained on short stories. it has eight layers and a hidden size of 64, which is small enough to run through this NumPy runtime without waiting forever.\n\nGPT-Neo is different from the model i started with. it uses learned position embeddings instead of RoPE. it uses LayerNorm instead of RMSNorm. the MLP uses GELU. its attention alternates between global and local windows.\n\nso i added another model implementation and mapped the checkpoint weights into NumPy arrays. PyTorch is only used to read the checkpoint file. it does not run inference.\n\n```\nmodel = GPTNeoModel.from_pretrained()\ntokenizer = GPT2Tokenizer.from_pretrained()\n\nengine = Engine(model, tokenizer=tokenizer)\n```\n\nand finally:\n\n```\nThe rabbit was a little girl named Timmy and a little girl named Emma.\n\nOne day, Lily and Lily was feeling very happy\n```\n\nnot a great story. but definitely better than `\\xf8`\n\n.\n\n## the whole thing\n\nthe runtime now looks like this:\n\nthere are many things it does badly.\n\nprefill still processes the prompt one token at a time. the paged cache gathers blocks instead of reading them directly inside attention. cache admission reserves the request’s maximum output length, which is safe but wastes capacity. there is no prefix caching. and NumPy has the final say on CPU performance.\n\nbut those are now problems i can point to in code instead of words i have read in an inference blog.\n\nthe main thing i took away is that generating a token and **serving** a token are different problems.\n\nthe transformer generates it.\n\nthe KV cache remembers everything that came before it. the scheduler decides when it gets generated. the streaming layer makes sure it reaches the right user.\n\n`model.forward()`\n\nwas the easy part.", "url": "https://wpnews.pro/news/how-llms-remember-schedule-and-stream-tokens", "canonical_source": "https://ssenthilnathan3.github.io/blog/kutty-vllm/", "published_at": "2026-08-24 00:00:00+00:00", "updated_at": "2026-08-24 07:45:31.536818+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research"], "entities": ["kutty-vllm", "NumPy", "Python"], "alternates": {"html": "https://wpnews.pro/news/how-llms-remember-schedule-and-stream-tokens", "markdown": "https://wpnews.pro/news/how-llms-remember-schedule-and-stream-tokens.md", "text": "https://wpnews.pro/news/how-llms-remember-schedule-and-stream-tokens.txt", "jsonld": "https://wpnews.pro/news/how-llms-remember-schedule-and-stream-tokens.jsonld"}}