{"slug": "creating-a-niche-ai-benchmark-with-token-anxiety", "title": "Creating a niche AI Benchmark with token anxiety", "summary": "The AI community is creating niche benchmarks like skatebench and vendingbench, and a new one focuses on token efficiency by having two models play a secret word guessing game where an explainer streams a description token by token and a guesser tries to identify the word, with the goal of minimizing tokens used. The benchmark's creator notes that many models initially leaked the secret word, and the system prompts were adjusted to prevent that.", "body_md": "The AI benchmarks that we look at most focus on how well models can code, how good they are at math/science etc.\n\nBut niche benchmarks are fun. Take [skatebench](https://skatebench.t3.gg/) (guessing the skateboard trick by textual explanation of how the trick works) and [vendingbench](https://andonlabs.com/evals/vending-bench-2) (AI running a vending machine company simulation).\n\nWhile the main benchmarks have the goal of measuring how AI is improving. More niche, and at first glance somewhat ridiculous, benchmarks can be a creative exploration of what models can and cannot do. I think we need more of them. We might stumble into the more unexpected or just have some fun creating.\n\nLet's build one.\n\nI have one real-life constraint you probably also feel right now, especially if you are the kind of person reading about some random AI benchmark.\n\n*I spend too much on AI stuff.*\n\nSubscription max at one place, some smaller ones I keep forgetting to cancel, still have some unused credits on that platform etc.\n\nThis AI stuff is so exciting, tiring, and expensive all at the same time. But I can't stop playing around with it.\n\nWhat if I try to make a benchmark that is cheap to run. Or even better, has incentive to not output a lot of tokens?\n\nSimplest form are the exam like benchmarks: 1 prompt problem and evaluate the answer to true or false. Great simple setup to build a niche bench on. You can pour all your creativity in the questions itself but I wanted something that messed with this structure a little bit.\n\nTokens come in 1 by 1, with the output speed of LLMs you would almost forget that. It is only visible on the API level if you turn on streaming. So what if we focus on a per token granularity of evaluation. I haven't seen many benchmarks have been done with an individual token by token dynamic. Let's do that! Let's anti-tokenmax, tokenminimize, tokenmin... you know what I mean.\n\nThe idea: *A secret word guessing game.*\n\nWe have 2 models working together as a pair, an explainer, and a guesser.\n\nThe explainer model gets a secret word to explain, streams its explanation of the word token by token, and the guesser model can, after every added token, guess the word it's trying to describe. The guesser model can either guess a word, or stay silent and wait for more tokens.\n\nThe explainer model is obviously not allowed to output the secret word. However, as we will see that is not always obvious to the models.\n\nI tried to keep the system prompts as minimal as possible of course, token stingy as we try to be.\n\nSystem prompt for the explainer model:\n\n```\nYou are playing 30 seconds the word game. \nYou are explaining the word to the user. \nStart your explanation right away. \nDo not mention the word, part of the word or a translation of the word in your explanation. If you do you lose. \nYour goal is to help them guess the word in as little time as possible.\n```\n\nThe \"30 seconds\" in the system prompt is a reference to a Dutch game that is a similar secret word guessing game which helps me not having to explain the game as much to the models: We are tokenminning right?\n\nAlso at first I had an even more minimal system prompt by leaving out the emphasis on not mentioning the secret word. However, first version there were a lot of models leaking the word itself by prefixing their explanation start with `Bombastic: {continued actual explaination of the secret word...}`\n\n.\n\nSystem prompt for the guesser model:\n\n```\nYou are playing 30 seconds. Guess the word from the explanation so far.\nYou must call the react tool every turn: action=pass if you need more explanation, \nor action=guess with one word if you have a guess.\nBe fast, but every guess distracts your teammate, so pass when unsure.\nDon't guess the same word multiple times; your teammate only confirms a correct guess. \nSo if you already guessed a word, that is not the correct word.\n```\n\nIn the guesser prompt it's visible that I used a tool-call for the guess, I tried without it before, just giving it instructions to output 1 word or `...`\n\nfor passing. But this had its downside as the models often responded with things like:\n\n`My guess is: {word}`\n\nor tried to pass with something more than just `...`\n\nfor a pass which meant I would have to hack around and use string matching to see if it had the intention to actually guess or tried to pass.\n\nBut tool-calling was also not perfect as Mistral tried to guess the tool name: `submit`\n\nitself as a guess... And overall it increased the token use of the benchmark.\n\nNow I realize 1 token of 1 model is not equal to another model token. And we will partly rank the influence of tokenizers. But for this simple bench I think this is fine to do.\n\nI took the top open source models from [Artificial Analysis](https://artificialanalysis.ai/?model-filters=open-source&models=mimo-v2-5-pro%2Cgemini-3-5-flash-lite%2Cclaude-sonnet-5%2Cminimax-m3%2Cgpt-5-6-luna%2Cqwen3-7-max%2Cgemini-3-6-flash%2Cgrok-4-5%2Cclaude-opus-4-8%2Cclaude-4-5-haiku-reasoning%2Cclaude-opus-5%2Cgpt-5-6-terra%2Cgemma-4-31b%2Cclaude-fable-5%2Cmuse-spark-1-1%2Cgpt-5-6-sol%2Cmistral-medium-3-5%2Cgpt-5-5-pro%2Cgpt-oss-120b%2Cglm-5-2%2Ckimi-k3) that have the option to turn thinking off and that were available on the platform I was using*\n\nThe benchmarked models:\n\nThe words in the benchmark are picked by downloading this [Kaggle dataset](https://www.kaggle.com/datasets/rtatman/english-word-frequency) and scrolling down to about rank 80,000 and picking some that looked fun. Figured about 10 would do the trick just fine.\n\nMy first thought was that the rank would have some meaning in how hard it was to explain as the words would possibly occur less in their training data. On the other hand from a more anthropomorphic perspective common words could be hard to explain well. Example: some of the most used words in English are `a`\n\n, `the`\n\n, `in`\n\nand possibly these are harder to explain than `firetruck`\n\nwhich is ranked way lower.\n\nHowever models are not humans, where I would find more difficulty in explaining `in`\n\nvs `firetruck`\n\n. I tried running the word `in`\n\nand GLM-5.2 guessed it after this:\n\nIt's a tiny word, just two letters! It's a preposition that shows location or position…\n\nNow I don't think a human under time pressure would come up quickly with such a good explanation on the spot, at least not me. To feel a bit better about ourselves humans: There was also a model which locked into \"at\" after this and would just keep repeating without switching to anything else.\n\nThen there was again GLM-5.2 that also guessed \"worm\" after:\n\nIt's a tiny wo...\n\n(If the explainer was going to say worm, worm would not be a logical guess to make because it is not allowed to mention the secret word.)\n\nMaybe the correct way of thinking was to let the difficulty emerge from how the models perform on the task empirically.\n\nI decided to first prompt the explainer for a full explanation of the secret word and save that in its original token-by-token streamed output.\n\nNo limit on how many output tokens, I let it naturally stop which didn't take longer than 120 tokens usually. Then for the guesser I called the model inference API with its system prompt and token by token the explainer's output.\n\nExample:\n\n```\nexplaining 'flatscreen' with openai/gpt-oss-120b\nIt’s a modern type of visual display, usually thin and lightweight, that shows moving pictures and images. It replaces the bulky cathode‑ray tubes of older televisions and monitors, often uses LED or LCD technology, and can be mounted on a wall or placed on a tabletop. Because the surface is essentially two‑dimensional, it gives a clear, bright picture without any curvature.\nguessing with openai/gpt-oss-120b\n[1/80]\n  in:  I\n  out: ... (pass)\n[2/80]\n  in:  It\n  out: ... (pass)\n[3/80]\n  in:  It’s\n  out: ... (pass)\n\n... (continued steps)\n\n[13/80]\n  in:  It’s a modern type of visual display, usually thin and\n  out: monitor\n[14/80]\n  in:  It’s a modern type of visual display, usually thin and lightweight\n  out: screen\n[15/80]\n  in:  It’s a modern type of visual display, usually thin and lightweight,\n  out: panel\n[16/80]\n  in:  It’s a modern type of visual display, usually thin and lightweight, that\n  out: ... (pass)\n[17/80]\n  in:  It’s a modern type of visual display, usually thin and lightweight, that shows\n  out: ... (pass)\n[18/80]\n  in:  It’s a modern type of visual display, usually thin and lightweight, that shows moving\n  out: television\n[19/80]\n  in:  It’s a modern type of visual display, usually thin and lightweight, that shows moving pictures\n  out: ... (pass)\n[20/80]\n  in:  It’s a modern type of visual display, usually thin and lightweight, that shows moving pictures and\n  out: flat-screen\nguessed at step 20\nsolved tokens=19 guesses=8 silences=12 steps=20\n```\n\nSame model was used for the guesser <> explainer pairs.\n\nBelow shows how well the models were able to get the secret word. MiniMax failed the toolcalling for 2 words resulting in missing these. The 2 most intelligent models Kimi, GLM end up on solving the most. But that is not the case for all the metrics I measure.\n\nFor all the rounds I tracked:\n\nInteresting to see that for example one of the smaller models, Gemma, when solving, it was quite minimal on tokens spent to do so.\n\nThe words that ate tokens were *almost right forever*.\n\nSecret: `accelerometers`\n\n.\nGuess: `accelerometer`\n\n.\nAgain. Again. Again.\n\nSame happened to `underestimate`\n\nvs `underestimating`\n\n.\n\nThey kept guessing the near morphology until the explainer runs out despite the system prompt explicitly telling the model not to repeat. Not trying any variations of the word.\n\nSome insight in models choosing to pass, and stay silent. Gemma shouts away while Mistral takes not distracting their teammates very seriously.\n\nGemma guessing bombastic in only 3 tokens somehow.\n\n`High-sounding`\n\nwhich is 3 tokens `High|-|sounding`\n\nand this specific combination might point the model to a particular direction. This doesn't seem like a human probable guess I wonder how often and close bombastic and these tokens are in the training data of base models.\n\nAlso note that Kimi leaked the answer here and was able to just guess it very quickly because of that.\n\nI wanted to tokenmin, but in the end my drive for a better benchmark did steer me off course a little.\n\nAppending guesses history, adding toolcalling to improve the output quality all added a lot to the tokens used.\n\nNot fully what we set out for but necessary changes in the end to make the benchmark more interesting.\n\nHere is the full (estimated) breakdown of the tokens used in different places.\n\nAs for the cost of running the benchmark:\n€0 marginal, fixed subscription on [featherless.ai](https://featherless.ai/) did the trick (not sponsored).\n\nSome ideas that would still be fun to explore:\n\nIn this setup one model has to convey information to another one, and the other has to prove understanding. I wonder what this could mean for multi agent task handoff. Where we could let models decide how much information they take in before getting to work, or how we can reduce tokens on the model that has to handover an assignment to another.\n\n* [featherless.ai](https://featherless.ai/) offers unlimited tokens for a fixed price, so perfect for not spending extra on tokens (not sponsored)", "url": "https://wpnews.pro/news/creating-a-niche-ai-benchmark-with-token-anxiety", "canonical_source": "https://thijsbrits.nl/blog/20260804/", "published_at": "2026-08-12 08:27:11+00:00", "updated_at": "2026-08-12 08:41:19.094439+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research"], "entities": ["skatebench", "vendingbench", "Andon Labs"], "alternates": {"html": "https://wpnews.pro/news/creating-a-niche-ai-benchmark-with-token-anxiety", "markdown": "https://wpnews.pro/news/creating-a-niche-ai-benchmark-with-token-anxiety.md", "text": "https://wpnews.pro/news/creating-a-niche-ai-benchmark-with-token-anxiety.txt", "jsonld": "https://wpnews.pro/news/creating-a-niche-ai-benchmark-with-token-anxiety.jsonld"}}