{"slug": "sampling-inkling-and-the-alias-pattern", "title": "Sampling Inkling and the Alias Pattern", "summary": "Thinking Machines' Inkling, a 975B total / 41B active MoE model with Apache 2.0 license and 1M context, is being trialed via a bash wrapper alias 'claude-inkling' that bridges Anthropic's CLI to the OpenAI messages format. The wrapper supports both Baseten direct endpoint and Vercel AI Gateway, with the direct route initially hitting rate limits (100k tokens/min) that interrupted work. The trial uses pass/fail sampling on real tasks, and Inkling's behavior matches its MoE spec: fast prefill, slower decode, and requiring a large output budget for reasoning.", "body_md": "In July I wrote [The Model Sheet](/blog/choosing-an-llm) — how a filename like `Qwen2.5-Coder-32B-Instruct-Q5_K_M.gguf`\n\nbreaks into family, domain, params, training stage, quantization, and format. Those fields predict behavior before you download. What comes after the spec sheet is the trial: load the weights, run a real prompt, see if the model behaves as advertised.\n\nLately that trial has been Thinking Machines' Inkling: 975B total / 41B active MoE, Apache 2.0, 1M context. Most modern open-weight releases now ship with the OpenAI messages format (`system`\n\n/ `user`\n\n/ `assistant`\n\nroles with `content`\n\narrays). That is the only reason the `claude-inkling`\n\nwrapper works. No translation layer needed — just point `ANTHROPIC_BASE_URL`\n\nat the endpoint.\n\nThe Baseten route (and the rate limit that killed it)\n\nBaseten's direct endpoint (`https://inference.baseten.co`\n\n) has a native `/v1/messages`\n\nroute. For a verified workspace, it skips the Basic-tier rate cap (100k tokens/min) — a single Claude Code turn exhausts that instantly. The key lives at `~/.config/baseten/key`\n\n. The wrapper (`~/.local/bin/claude-inkling`\n\n) reads it, exports the endpoint and model identifier (`thinkingmachines/inkling`\n\nor `inkling-small`\n\n), then executes `claude`\n\n.\n\nIn early trials we hit the cap immediately. Reading files with Inkling Large consumed tokens fast enough that the rate limit interrupted work before anything finished. The fix was temporary: switch back to the [Vercel AI Gateway](/blog/we-dont-want-ai-pets-fix-your-connectors) (`claude-gateway`\n\n) to complete the task. The Gateway pools through a single bill with provider failover, so the interruption was minimal. Once the workspace is verified or a dedicated deployment is provisioned, Baseten direct is the stable path. For now the wrapper supports both: `INKLING_VIA=baseten`\n\nfor direct, default for Gateway.\n\nThe alias: `claude-inkling`\n\nThe wrapper is a thin bash script. It exports Anthropic endpoint variables, disables telemetry and non-essential model calls (so local decode isn't wasted on background tasks), sets a generous output budget (32k by default), and executes `claude`\n\n. Same pattern as `claude-laguna`\n\n(local `llama-server`\n\n) and `claude-deepseek`\n\n(local DeepSeek V4-Flash via `ds4`\n\n). One alias per model, one endpoint per alias, zero proxy overhead.\n\nBecause Inkling — like most new releases — speaks the OpenAI messages format, the client sends `{\"model\":\"thinkingmachines/inkling\",\"messages\":[...]}`\n\nand gets back `{\"type\":\"message\",\"content\":\"...\"}`\n\n. The alias just bridges Anthropic's CLI naming convention and the provider's model registry.\n\nWhat the trial looks like\n\nSampling is not a benchmark run. It is a short, representative task from real work — a coding snippet, a reasoning chain, a format conversion — scored pass or fail against a known answer. The model either produces working output or it doesn't. Binary. I keep the prompt set in a plain file under version control. The next release has to earn the swap.\n\nInkling's behavior so far aligns with its MoE spec: 41B active out of 975B total. Fast prefill, slower decode (bandwidth-bound, like any large model), and a generous output budget needed for reasoning spans before visible answers appear. That matches other reasoning-model releases — if `max_tokens`\n\nis too small, the budget goes to hidden thinking and the content field returns empty. Fix: large token budget, or a non-reasoning variant for short replies.\n\nThis connects back to [The Benchmark](/blog/the-benchmark): a leaderboard rank collapses too many dimensions into one number. The fields from [The Model Sheet](/blog/choosing-an-llm) — dense vs MoE, active params, quantization level, training stage — are the actual variables. The alias wrapper just makes them accessible to the CLI I already use.\n\nWhere this fits: the local lab keeps Llama-3.3-70B (storyteller) and Qwen2.5-Coder-32B (architect) hot on Metal via `llama-server`\n\n(`llama70`\n\n:8080, `coder32`\n\n:8081). The `claude-inkling`\n\nalias is the hosted complement: frontier reasoning or very long context goes through the Gateway; once verified, Baseten direct takes over.\n\nNext: finish the Baseten workspace verification so direct routing replaces the Gateway fallback, and update the prompt set file with the Inkling trial results. See the running notes in [ ~/llm-lab/session-log.md](/llm-lab/session-log.md) for the live measurements.", "url": "https://wpnews.pro/news/sampling-inkling-and-the-alias-pattern", "canonical_source": "https://zackproser.com/blog/sampling-inkling-baseten-alias", "published_at": "2026-08-02 00:00:00+00:00", "updated_at": "2026-08-02 19:28:13.101200+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["Thinking Machines", "Inkling", "Baseten", "Vercel AI Gateway", "Anthropic", "Claude Code", "Llama-3.3-70B", "Qwen2.5-Coder-32B"], "alternates": {"html": "https://wpnews.pro/news/sampling-inkling-and-the-alias-pattern", "markdown": "https://wpnews.pro/news/sampling-inkling-and-the-alias-pattern.md", "text": "https://wpnews.pro/news/sampling-inkling-and-the-alias-pattern.txt", "jsonld": "https://wpnews.pro/news/sampling-inkling-and-the-alias-pattern.jsonld"}}