The skillcheck static analyzer for SKILL.md files received a hardening update, fixing the description scorer, improving error handling for corrupt files, and correcting token estimate documentation. M…
Liquid AI, an AI company, released an open-source byte-pair encoding (BPE) tokenizer trainer called toktoktok on GitHub, built autonomously by two coding agents using Claude Opus 4.5 and Codex with GP…
A developer built a standing measurement tool called loadline to track how many context-window tokens MCP servers consume, running 72 trials across 14 servers. The measurements show that Claude's toke…
A developer built mcptoon, a CLI tool that reduces MCP server token usage by 91% by replacing verbose JSON schemas with a compact pipe-delimited format. The tool, which sits between AI coding agents a…
AI engineer Taha Azizi released Contextpress, a Python library (Apache 2.0, Python 3.10+) that refactors LLM context by running a deterministic NLP pipeline over message history to strip filler, dedup…
A developer measured the token cost of pasting files into an AI agent's context window, finding that a single 2,348-line source file consumes 40,324 tokens—one fifth of a 200,000-token window—while th…
A developer building LLM applications found that the model itself is only a third of the work, with the surrounding infrastructure—timeouts, retries, and token management—proving critical. They implem…
A Thoughtworks technologist built a 150,000-line application entirely with AI agents (Claude Code and Cursor) and then ran an experiment showing that refactoring a 17,155-line Rust data-access file re…
Gigatoken, a Rust-based tokenizer by Marcel Rød, achieves 30x to 40x faster CPU tokenization than traditional implementations, with 37.6x speedup on a million-token request, by optimizing BPE encoding…
Baseten model performance engineer Michael Feil shipped Baseten Tokenizer, a Rust-backed package that cuts CPU time for preparing long prompts for Moonshot AI's Kimi K3 model, achieving up to 18x fast…
Developer DeKlain4ik released Tokenazire, an MIT-licensed Python CLI tool that uses tiktoken to count tokens in code projects and check if they fit an LLM context window, defaulting to 200k tokens. Th…
Stanford PhD student Marcel Rød released Gigatoken, a Rust BPE tokenizer that encodes text at 24.53 GB/s on a 144-core AMD EPYC 9565 dual-socket setup, outperforming HuggingFace tokenizers by 989x and…
GigaToken, a new open-source tokenizer, claims to be up to ~1000x faster than HuggingFace's tokenizers and tiktoken for language model tokenization, achieving speeds of over 24 GB/s on an AMD EPYC 956…
Gigatoken, a new tokenizer implementation, claims to be the world's fastest, achieving speeds ~500-1000x faster than HuggingFace and ~100x faster than OpenAI's tiktoken on most machines, even though t…
A guide explains how to build a tiny large language model from scratch, covering tokenization, pretraining, supervised fine-tuning, and alignment. It aims to demystify the LLM pipeline for developers …
Tokens are the fundamental unit of text processing in large language models (LLMs), determining API costs, response speed, and context window limits. Different providers like OpenAI, Anthropic, and Go…
LectuLibre built an AI-powered platform that translates entire books using large language models, overcoming token limits by implementing a sliding window chunking algorithm based on paragraphs with o…
SoloEngine v0.3.0 introduces a checkpoint mechanism with three streaming checkpoints and a MessageQueue class for async message handling. The release also upgrades ModelManager to AutoComplete, refact…
A developer discovered that using OpenAI's tiktoken tokenizer to count tokens for Claude models leads to a 15-20% undercount, causing inaccurate cost estimates and context budgets. The correct approac…
A developer explains that RAG pipeline chunking strategies determine retrieval quality more than the embedding model or vector store, with most recall failures tracing back to how documents were split…