In July I wrote The Model Sheet — how a filename like Qwen2.5-Coder-32B-Instruct-Q5_K_M.gguf
breaks into family, domain, params, training stage, quantization, and format. Those fields predict behavior before you download. What comes after the spec sheet is the trial: load the weights, run a real prompt, see if the model behaves as advertised.
Lately that trial has been Thinking Machines' Inkling: 975B total / 41B active MoE, Apache 2.0, 1M context. Most modern open-weight releases now ship with the OpenAI messages format (system
/ user
/ assistant
roles with content
arrays). That is the only reason the claude-inkling
wrapper works. No translation layer needed — just point ANTHROPIC_BASE_URL
at the endpoint.
The Baseten route (and the rate limit that killed it)
Baseten's direct endpoint (https://inference.baseten.co
) has a native /v1/messages
route. For a verified workspace, it skips the Basic-tier rate cap (100k tokens/min) — a single Claude Code turn exhausts that instantly. The key lives at ~/.config/baseten/key
. The wrapper (~/.local/bin/claude-inkling
) reads it, exports the endpoint and model identifier (thinkingmachines/inkling
or inkling-small
), then executes claude
.
In early trials we hit the cap immediately. Reading files with Inkling Large consumed tokens fast enough that the rate limit interrupted work before anything finished. The fix was temporary: switch back to the Vercel AI Gateway (claude-gateway
) to complete the task. The Gateway pools through a single bill with provider failover, so the interruption was minimal. Once the workspace is verified or a dedicated deployment is provisioned, Baseten direct is the stable path. For now the wrapper supports both: INKLING_VIA=baseten
for direct, default for Gateway.
The alias: `claude-inkling`
The wrapper is a thin bash script. It exports Anthropic endpoint variables, disables telemetry and non-essential model calls (so local decode isn't wasted on background tasks), sets a generous output budget (32k by default), and executes claude
. Same pattern as claude-laguna
(local `llama-server`
) and `claude-deepseek`
(local DeepSeek V4-Flash via `ds4`
). One alias per model, one endpoint per alias, zero proxy overhead.
Because Inkling — like most new releases — speaks the OpenAI messages format, the client sends {"model":"thinkingmachines/inkling","messages":[...]}
and gets back {"type":"message","content":"..."}
. The alias just bridges Anthropic's CLI naming convention and the provider's model registry.
What the trial looks like
Sampling is not a benchmark run. It is a short, representative task from real work — a coding snippet, a reasoning chain, a format conversion — scored pass or fail against a known answer. The model either produces working output or it doesn't. Binary. I keep the prompt set in a plain file under version control. The next release has to earn the swap.
Inkling's behavior so far aligns with its MoE spec: 41B active out of 975B total. Fast prefill, slower decode (bandwidth-bound, like any large model), and a generous output budget needed for reasoning spans before visible answers appear. That matches other reasoning-model releases — if max_tokens
is too small, the budget goes to hidden thinking and the content field returns empty. Fix: large token budget, or a non-reasoning variant for short replies.
This connects back to The Benchmark: a leaderboard rank collapses too many dimensions into one number. The fields from The Model Sheet — dense vs MoE, active params, quantization level, training stage — are the actual variables. The alias wrapper just makes them accessible to the CLI I already use.
Where this fits: the local lab keeps Llama-3.3-70B (storyteller) and Qwen2.5-Coder-32B (architect) hot on Metal via llama-server
(llama70
:8080, coder32
:8081). The claude-inkling
alias is the hosted complement: frontier reasoning or very long context goes through the Gateway; once verified, Baseten direct takes over.
Next: finish the Baseten workspace verification so direct routing replaces the Gateway fallback, and update the prompt set file with the Inkling trial results. See the running notes in ~/llm-lab/session-log.md for the live measurements.