Sampling Inkling and the Alias Pattern

Thinking Machines' Inkling, a 975B total / 41B active MoE model with Apache 2.0 license and 1M context, is being trialed via a bash wrapper alias 'claude-inkling' that bridges Anthropic's CLI to the OpenAI messages format. The wrapper supports both Baseten direct endpoint and Vercel AI Gateway, with the direct route initially hitting rate limits (100k tokens/min) that interrupted work. The trial uses pass/fail sampling on real tasks, and Inkling's behavior matches its MoE spec: fast prefill, slower decode, and requiring a large output budget for reasoning.

In July I wrote The Model Sheet /blog/choosing-an-llm — how a filename like Qwen2.5-Coder-32B-Instruct-Q5 K M.gguf breaks into family, domain, params, training stage, quantization, and format. Those fields predict behavior before you download. What comes after the spec sheet is the trial: load the weights, run a real prompt, see if the model behaves as advertised. Lately that trial has been Thinking Machines' Inkling: 975B total / 41B active MoE, Apache 2.0, 1M context. Most modern open-weight releases now ship with the OpenAI messages format system / user / assistant roles with content arrays . That is the only reason the claude-inkling wrapper works. No translation layer needed — just point ANTHROPIC BASE URL at the endpoint. The Baseten route and the rate limit that killed it Baseten's direct endpoint https://inference.baseten.co has a native /v1/messages route. For a verified workspace, it skips the Basic-tier rate cap 100k tokens/min — a single Claude Code turn exhausts that instantly. The key lives at ~/.config/baseten/key . The wrapper ~/.local/bin/claude-inkling reads it, exports the endpoint and model identifier thinkingmachines/inkling or inkling-small , then executes claude . In early trials we hit the cap immediately. Reading files with Inkling Large consumed tokens fast enough that the rate limit interrupted work before anything finished. The fix was temporary: switch back to the Vercel AI Gateway /blog/we-dont-want-ai-pets-fix-your-connectors claude-gateway to complete the task. The Gateway pools through a single bill with provider failover, so the interruption was minimal. Once the workspace is verified or a dedicated deployment is provisioned, Baseten direct is the stable path. For now the wrapper supports both: INKLING VIA=baseten for direct, default for Gateway. The alias: claude-inkling The wrapper is a thin bash script. It exports Anthropic endpoint variables, disables telemetry and non-essential model calls so local decode isn't wasted on background tasks , sets a generous output budget 32k by default , and executes claude . Same pattern as claude-laguna local llama-server and claude-deepseek local DeepSeek V4-Flash via ds4 . One alias per model, one endpoint per alias, zero proxy overhead. Because Inkling — like most new releases — speaks the OpenAI messages format, the client sends {"model":"thinkingmachines/inkling","messages": ... } and gets back {"type":"message","content":"..."} . The alias just bridges Anthropic's CLI naming convention and the provider's model registry. What the trial looks like Sampling is not a benchmark run. It is a short, representative task from real work — a coding snippet, a reasoning chain, a format conversion — scored pass or fail against a known answer. The model either produces working output or it doesn't. Binary. I keep the prompt set in a plain file under version control. The next release has to earn the swap. Inkling's behavior so far aligns with its MoE spec: 41B active out of 975B total. Fast prefill, slower decode bandwidth-bound, like any large model , and a generous output budget needed for reasoning spans before visible answers appear. That matches other reasoning-model releases — if max tokens is too small, the budget goes to hidden thinking and the content field returns empty. Fix: large token budget, or a non-reasoning variant for short replies. This connects back to The Benchmark /blog/the-benchmark : a leaderboard rank collapses too many dimensions into one number. The fields from The Model Sheet /blog/choosing-an-llm — dense vs MoE, active params, quantization level, training stage — are the actual variables. The alias wrapper just makes them accessible to the CLI I already use. Where this fits: the local lab keeps Llama-3.3-70B storyteller and Qwen2.5-Coder-32B architect hot on Metal via llama-server llama70 :8080, coder32 :8081 . The claude-inkling alias is the hosted complement: frontier reasoning or very long context goes through the Gateway; once verified, Baseten direct takes over. Next: finish the Baseten workspace verification so direct routing replaces the Gateway fallback, and update the prompt set file with the Inkling trial results. See the running notes in ~/llm-lab/session-log.md /llm-lab/session-log.md for the live measurements.