{"slug": "welcome-inkling-by-thinking-machines", "title": "Welcome Inkling by Thinking Machines", "summary": "Thinking Machines released Inkling, a ~1 trillion parameter multimodal LLM with a 1M context window, on Hugging Face. The open model natively processes text, image, and audio inputs, features a Mixture-of-Experts architecture with 41B active parameters, and includes speculative MTP layers for faster inference. Day-0 support is available in transformers, SGLang, and llama.cpp.", "body_md": "Viewer • Updated • 1k • 991 • 36\n\n# Welcome Inkling by Thinking Machines\n\n[Update on GitHub](https://github.com/huggingface/blog/blob/main/thinkingmachines-inkling.md)\n\nTLDR; Inkling by Thinking Machines is out on Hugging Face. Inkling is a huge multimodal LLM that understands all modalities (image, audio, text), has agentic capabilities, and supports 1M context. It comes in full BF16 and a well-calibrated NVFP4 variant, and includes speculative MTP layers for faster inference. There’s day-0 support in transformers, SGLang, and llama.cpp.\n\n## What makes Inkling special?\n\nInkling is the first large open model with **~1T parameters** and **1M context window** to natively receive **image, text, and audio inputs**, trained on **45 trillion tokens of text, images, audio and video.** It’s focused on reasoning across modalities such as audio, images, and text; and is intended for domain adaptation via fine-tuning. We’ve tinkered with this model to build some demos and explore the architecture, and we think it’s great for building a new wave of multimodal reasoning apps.\n\n## Overall Capabilities and Architecture\n\nInkling is a decoder-only multimodal Mixture-of-Experts model with 975B total and 41B active parameters. There are a lot of things going on, so let’s break each part down:\n\n- Decoder-only: This means that the architecture supports causal autoregressive generation, like in most state-of-the-art LLMs.\n- Multimodal: The model can ingest text, audio, and images.\n- Mixture of Experts (MoE): The feed forward networks inside each layer are sparse, achieving faster inference because only 41B parameters are active at any given time. The model has 256 experts, as we’ll see later.\n\nHere’s a quick glance of the architecture.\n\n**Relative attention:** Instead of RoPE, which is the usual method to inject positional information in transformers models, Inkling uses relative attention to encode position information. Each attention layer learns position directly in the attention logits. Aside from key-query-values, there's a fourth projection producing a per-token, per-head relative feature R. This projection tensor is then tweaked with distance information (distance between the key and the query vector) and propagated into the attention module.\n\n**Hybrid attention:** The decoder layers alternate between global attention (attending to the full context length at once) and sliding window attention (attending to a fixed context window in a sliding fashion). The architecture has a pattern of 5:1 sliding window to global attention layers. This hybrid attention scheme provides efficiency in computation. The final layer uses global attention to help build feature-rich representations.\n\n**Short convolution:** The model uses a distinctive short 1D convolution, or `SConv`\n\nover the hidden states. SConv reads the current token and the previous `W-1`\n\nhidden states, with `W`\n\nbeing the sliding window size. The intuition here is that SConv helps with local attention while freeing the attention and MoE modules from local representations.\n\n**MoE with shared experts sink:** In Inkling, the router scores both routed experts and shared experts. Top-k selection is performed over 6 experts, plus 2 shared experts always active.\n\n**Vision understanding:** The model includes a simple hierarchical MLP patchifier consisting of several linear layers. Each layer merges pixels progressively, until the final layer produces one embedding per patch.\n\n**Audio understanding:** The architecture employs a discretized mel spectrogram, where each of the audio chunks (of 100 ms) are converted to the mel scale and then classified into the exact mel spectrogram bin.\n\nThe multimodal towers are relatively simple modules, unlike other models that employ separate encoders for each modality. Each image patch passes through the image embedding tower and the audio chunk is passed through the audio embedding tower to get both media embeddings. Image inputs also include an additional temporal dimension for video processing. We expect this capability to be useful for downstream fine-tuning, but we haven’t evaluated out-of-the-box video performance. The tower folds the patch grid, a small local block of neighboring tokens is stacked into the channel dimension and goes through hMLP. The audio waveform is converted to mel scale, which is then classified into a discrete mel bin. These mel bin values are embedded in the audio embedding tower and the embeddings are then summed to construct the final audio input.\n\n## Inference Support\n\nInkling comes with day-0 transformers support and is supported in major inference engines like SGLang and vLLM.\n\nThis model is huge. The bf16 checkpoint requires 2 TB of VRAM, while the nvfp4 version requires 600 GB of VRAM. You can try the model through serverless inference routers like Inference Providers, or use ggml quants for local deployment with llama.cpp.\n\n### Transformers\n\nThe easiest way to infer with `transformers`\n\ndirectly is to use the `any-to-any`\n\npipeline. You can use either the 16 bit `\"thinkingmachines/Inkling\"`\n\non Hopper or later GPUs, or the quantized NVFP4 checkpoint `\"thinkingmachines/Inkling-NVFP4\"`\n\non Blackwell Nvidia GPUs. Make sure to have the latest version of transformers (5.14.0 was released today) (`pip install -U transformers`\n\n).\n\n``` python\nfrom transformers import pipeline\n\nmodel_id = \"thinkingmachines/Inkling\"\n# model_id = \"thinkingmachines/Inkling-NVFP4\"\n\npipe = pipeline(\"any-to-any\", model=model_id)\n```\n\nAfter initializing the pipeline, you can pass in the prompt as follows.\n\n```\nimage_url = (\n    \"https://huggingface.co/datasets/merve/vl-test-suite/\"\n    \"resolve/main/pills.jpg\"\n)\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\n                \"type\": \"image\",\n                \"image\": image_url,\n            },\n            {\n                \"type\": \"text\",\n                \"text\": \"Do components in this supplement interact with each other?\",\n            },\n        ],\n    },\n]\noutput = pipe(\n    messages,\n    max_new_tokens=2000,\n    return_full_text=False,\n    reasoning_effort=\"medium\",\n)\noutput[0][\"generated_text\"]\n```\n\nGoing one level lower, you can use Auto classes. For inference, you can use the `AutoModelForMultimodalLM`\n\nclass for models and `AutoProcessor`\n\nclass for processors. For different reasoning tasks, the tokenizer takes in a `reasoning_effort`\n\nargument. Existing options for reasoning effort are `\"none\"`\n\n, `\"minimal\"`\n\n, `\"low\"`\n\n, `\"medium\"`\n\n, `\"high\"`\n\n, `\"xhigh\"`\n\n, and `\"max\"`\n\n.\n\n``` python\nfrom transformers import AutoModelForMultimodalLM, AutoProcessor\n\nmodel_id = \"thinkingmachines/Inkling\"\nprocessor = AutoProcessor.from_pretrained(model_id)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n    model_id,\n    dtype=\"auto\",\n    device_map=\"auto\",\n)\n\nmessages = [\n    {\"role\": \"system\", \"content\": \"You should only answer with a number.\"},\n    {\"role\": \"user\", \"content\": \"What is 17 * 23?\"},\n]\n\ninputs = processor.apply_chat_template(\n    messages,\n    add_generation_prompt=True,\n    tokenize=True,\n    return_dict=True,\n    return_tensors=\"pt\",\n    reasoning_effort=\"high\",\n).to(model.device)\n\noutput = model.generate(**inputs, max_new_tokens=2000)\ngenerated_tokens = output[0][inputs[\"input_ids\"].shape[1] :]\nprint(processor.decode(generated_tokens, skip_special_tokens=False))\n```\n\nFor multimodal inference, you can use the same classes. We provide example snippets for each different modality in the model card.\n\n## Text with image inference\n\n``` python\nfrom transformers import AutoModelForMultimodalLM, AutoProcessor\n\nmodel_id = \"thinkingmachines/Inkling\"\nprocessor = AutoProcessor.from_pretrained(model_id)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n    model_id,\n    dtype=\"auto\",\n    device_map=\"auto\",\n)\n\nimage_url = (\n    \"https://huggingface.co/datasets/merve/vl-test-suite/\"\n    \"resolve/main/pills.jpg\"\n)\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\n                \"type\": \"image\",\n                \"image\": image_url,\n            },\n            {\n                \"type\": \"text\",\n                \"text\": \"Do any of the components in this supplement interact?\",\n            },\n        ],\n    },\n]\n\ninputs = processor.apply_chat_template(\n    messages,\n    tokenize=True,\n    add_generation_prompt=True,\n    reasoning_effort=\"medium\",\n    return_dict=True,\n    return_tensors=\"pt\",\n).to(model.device)\ninput_len = inputs[\"input_ids\"].shape[-1]\n\noutputs = model.generate(**inputs, max_new_tokens=2000)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\n\nprocessor.parse_response(response)\n```\n\nInkling also takes in audio input. Below is an example inference snippet, which still uses the same `AutoModelForMultimodalLM`\n\nclass.\n\n## Text with audio inference\n\n``` python\nfrom transformers import AutoModelForMultimodalLM, AutoProcessor\n\nmodel_id = \"thinkingmachines/Inkling\"\n\nprocessor = AutoProcessor.from_pretrained(model_id)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n    model_id,\n    dtype=\"auto\",\n    device_map=\"auto\",\n)\n\naudio_url = (\n    \"https://huggingface.co/datasets/merve/vl-test-suite/\"\n    \"resolve/main/example_audio.mp3\"\n)\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"text\", \"text\": \"Transcribe the following speech to text.\"},\n            {\n                \"type\": \"audio\",\n                \"audio\": audio_url,\n            },\n        ],\n    },\n]\n\ninputs = processor.apply_chat_template(\n    messages,\n    tokenize=True,\n    return_dict=True,\n    return_tensors=\"pt\",\n    add_generation_prompt=True,\n).to(model.device)\ninput_len = inputs[\"input_ids\"].shape[-1]\n\noutputs = model.generate(**inputs, max_new_tokens=512)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\n\nprocessor.parse_response(response)\n```\n\nFor more realistic parallel deployment in a cluster of several nodes, please refer to the [Slurm](#slurm-scripts) section below.\n\n### SGLang\n\nSGLang is one of the fastest deployment frameworks for Inkling at the time of release, as it includes a custom model implementation. The launch command below shards the model across 8 GPUs and serves an OpenAI-compatible API on port 30000.\n\n```\npip install sglang\n\npython3 -m sglang.launch_server \\\n --model-path thinkingmachine/Inkling \\\n --tp-size 8 \\\n --served-model-name inkling \\\n --host 0.0.0.0 \\\n --port 30000\n```\n\nMatch `--tp-size`\n\nto your GPU count. Add `--mem-fraction-static`\n\n(e.g. `0.85`\n\n) if you need to leave more headroom for the KV cache.\n\n### vLLM\n\nvLLM is strong for production serving. A single `vllm serve`\n\ncommand downloads the weights from the Hub, shards the model across your GPUs with tensor parallelism, and starts an OpenAI-compatible server on port 8000.\n\n```\npip install vllm\n\nvllm serve thinkingmachine/Inkling \\\n  --tensor-parallel-size 8 \\\n  --served-model-name inkling\n```\n\nIn practice, you will need multiple nodes and a distribution tool like SLURM (see below). Key parameters are `--tensor-parallel-size`\n\nto the number of GPUs on your node, and use `--max-model-len`\n\nto cap the context window if you hit KV-cache memory limits.\n\n```\ncurl http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"inkling\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]\n  }'\n```\n\n### Remote Inference with Hugging Face Inference Providers\n\nYou can infer with this model using several inference providers through Hugging Face. You can see all the code snippets to consume [here](https://huggingface.co/thinkingmachines/inkling?inference_provider=fastest&language=python&client=openai&inference_api=true). Below you can see how to use with the OpenAI client.\n\n``` python\nimport os\n\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"https://router.huggingface.co/v1\",\n    api_key=os.environ[\"HF_TOKEN\"],\n)\n\ncompletion = client.chat.completions.create(\n    model=\"thinkingmachines/Inkling:auto\",\n    messages=[\n        {\n            \"role\": \"user\",\n            \"content\": \"What is the capital of France?\",\n        },\n    ],\n)\n\nprint(completion.choices[0].message)\n```\n\nUsing the `“:auto”`\n\nsuffix routes to your preferred provider in your settings; you can also use `“cheapest”`\n\nor `“:fastest”`\n\nas well. For this release, we cover the inference costs for 2 hours within the release for everyone.\n\nNote: audio support in Inference Providers is work in progress and will be added shortly.\n\n### Local Inference with llama.cpp and Unsloth\n\nYou can use `llama.cpp`\n\nto run quantized versions of the model on limited hardware. Unsloth have quantized the model down to 1-bit precision, reducing VRAM consumption by 95% over the original model.\n\n```\nllama serve -hf unsloth/inkling-GGUF:UD-IQ1_S\n```\n\nThis starts an OpenAI-compatible server running at `http://localhost:8000`\n\n`/v1`\n\nthat you connect to in your preferred tool or clients. Heading there, you can start chatting with the model, and set it up with your favorite MCPs, pass in images or files conveniently and more!\n\nLlama cpp also ships with a built-in UI that supports tools, mcp, and agentic workloads. Checkout Inkling running at 1-bit precision in the llama app:\n\nInkling GGUFs are also runnable in Unsloth Studio with dynamic 1-bit GGUFs which retain ~74.2% of top-1% accuracy whilst being 86% smaller.\n\n## Use Cases\n\n### Agentic coding with Pi\n\nPi is a minimal coding agent harness you can use with different language models. You can use Pi with either an inference engine server endpoint, such as llama.cpp, or with Inference Providers on Hugging Face by adding this to your `~/.pi/agent/models.json`\n\nafter installation.\n\n```\n{\n  \"providers\": {\n    \"inference-providers\": {\n      \"baseUrl\": \"https://router.huggingface.co/v1\",\n      \"api\": \"openai-completions\",\n      \"apiKey\": \"hf_...\",\n      \"models\": [\n        {\n          \"id\": \"thinkingmachines/Inkling\"\n        }\n      ]\n    }\n  }\n}\n```\n\nThen you can start Pi in your project directory by calling `pi`\n\nand you’re good to go! In this demo, we give the model a hard math reasoning problem and it uses tools in pi to solve it.\n\nInkling is focused on broad multimodality reasoning and low token consumption, so try it out with document processing or audio tasks.\n\n### Multi Token Prediction Drafters\n\nMTP adds extra layers to the model that predict several tokens at once, not just the next one. During inference, the extra layers act as “drafters” for speculative decoding, speeding up generation without compromising performance. With MTP, you get the exact same generated outputs, multipliers in generation speed-up at small memory cost in VRAM (due to serving the drafter). Thinking Machines also provides an MTP drafter with this release.\n\n``` python\nimport torch\nfrom transformers import AutoModelForMultimodalLM, AutoProcessor\n\nprocessor = AutoProcessor.from_pretrained(\"thinkingmachines/Inkling\")\nmodel = AutoModelForMultimodalLM.from_pretrained(\n    \"thinkingmachines/Inkling\",\n    dtype=torch.bfloat16,\n    device_map=\"auto\",\n)\n\n# Preprocess the inputs.\n...\ngenerated = model.generate(\n    **inputs,\n    max_new_tokens=1000,\n    do_sample=False,\n    use_mtp=True,\n)\nprint(processor.decode(generated[0], skip_special_tokens=True))\n```\n\n### Multimodal Vision\n\nWe have prepared a small suite of reasoning questions from expert-level sources and university entrance exams. We have taken photos of the screen with watermarks in the screenshot to challenge the model. The model has solved all of them on high one, failed one in highest and medium reasoning efforts, so we provide a link to the model answers for you to check out how the model sounds and provide the number of tokens the model has taken to solve each of them. Note that we provide no system prompts in these vibe evals, and these reasoning questions should often be run with a good system prompt. The vibe eval images and results live [here](https://huggingface.co/buckets/merve/inkling).\n\n| Category | Question | Number of Tokens (Reasoning Effort Medium) | Number of Tokens (Reasoning Effort High) | Number of Tokens (Reasoning Effort Max) |\n|---|---|---|---|---|\n| Open-ended Drug Interactions | Which components interact here? | 1,893 ✅ | 2,367 ✅ | 3,688 ✅ |\n| Physics Question (MMMU-Pro) | Answer the question in the image. | 1,357 ✅ | 3,323 ✅ | 3,314 ✅ |\n| Multilingual Physics Question | Answer the Turkish question given in the image. | 1,435 ✅ | 2,129 ✅ | 3,162 ✅ |\n| Bar Exam | Answer the question in the image. | 1,117 ✅ | 2,137 ✅ | 1,676 ✅ |\n| Infographics Question Answering (Open-ended) | Based on the information presented, approximately how many times larger is the projected summer warming period in the Arctic than the time over which substantial Arctic warming has already been observed? | 1,378 ❌ | 3,859 ✅ | 6000 (exceeded token budget) |\n\n**Few notes on vibes:**\n\n- Instead of directly answering the question on infographic, the model first turns text on image to text to ground itself.\n- Prompting matters a lot to save tokens in reasoning, for instance, asking vague questions like “which components interact here?” with an image of the back of a pill, the model first needs to see what we mean by interactions here.\n- Multi-choice question answers helped the model a lot in structuring its own reasoning, for open-ended questions the model struggled compared to MCQA, however, this is a common issue for many models. The usual chain of thought was OCR → characterize → evaluate each option → answer.\n- 0.7 reasoning effort (medium) seems to provide a good trade-off.\n\n### Multimodal Audio\n\nWe have vibe-evaluated the model on some audio reasoning examples from BigBenchAudio and a few multilingual audio examples of [GlobeAudio](https://huggingface.co/datasets/iNLP-Lab/GlobeAudio) (Russian and Chinese multi-choice questions asking the last word in transcription). The [BigBenchAudio](https://huggingface.co/datasets/ArtificialAnalysis/big_bench_audio) examples we tested consist of logical statements and questions that either ask for formal fallacies (whether an argument can be logically deduced from the context given in audio) or object counting (stating multiple distinctive objects in the audio, asking for the total count of a certain one). Although this benchmark is initially made for speech-to-speech reasoning, we just want to see audio reasoning capabilities of this model. For GlobeAudio, the questions are relatively straightforward, so we ran with reasoning efforts of 0.1. We ran the first example of each language within GlobeAudio. All tests pass on all questions and efforts, except for second formal fallacy example on lowest effort, so we only provide the number of tokens spent in each question against reasoning effort. Vibe eval results and audio files live [here](https://huggingface.co/buckets/merve/inkling).\n\n| GlobeAudio | Question | Number of completion tokens (Reasoning effort lowest) | Number of completion tokens (Reasoning effort medium) |\n|---|---|---|---|\n| Russian (asks for last word) | Какое последнее слово в аудиозаписи? 1. Россия 2. Свидетелем 3. Москва 4. Событий Choose the single correct option and answer with its exact text. | 130 | 179 |\n| Russian (asks for profession of the speaker) | Кем, скорее всего, работает говорящая? 1. Репортершей 2. Блоггершей 3. Учительницей истории 4. Ведущей развлекательного шоу Choose the single correct option and answer with its exact text. | 105 | 136 |\n| Chinese (asks for speaking rate) | 播报员的语速有何变化？ 1. 突然变快 2. 突然变慢 3. 保持不变 4. 时快时慢 Choose the single correct option and answer with its exact text. | 111 | 289 |\n\n| Big Bench Audio | Completion Tokens (lowest) | Completion Tokens (medium) | Number of completion tokens ( highest) |\n|---|---|---|---|\n| Formal Fallacy (10) | 285 | 335 | 444 |\n| Formal Fallacy (39) | 275 (fails) | 555 | 778 |\n| Object Counting (680) | 150 | 233 | 161 |\n\n**Some notes on the vibes:**\n\n- Similar to vision, the model first transcribes the speech before answering the question.\n- It resists decoys: in Russian test, the model picked the right answer despite other answers appearing in the audio.\n- Similar to vision, usual chain of thought is transcribe → characterize → evaluate each option → answer.\n- The effort helps reasoning and not hearing. Audio question answering was much cheaper than images.\n\n### Post-training\n\nIf you would like to use Inkling for post-training, Thinking Machines have built `tinker`\n\n, a managed tool for post-training open weight models. Their cookbook includes examples for fine-tuning, distillation, and reinforcement learning.\n\nWe post trained Inkling with tinker and OpenEnv, an agentic RL environment tool. We used the ECHO algorithm that trains a model to predict the environment without a verifier, applying next-token cross-entropy loss to tokens produced by the environment, alongside the usual policy learning on agent actions. This teaches the policy an implicit world model without requiring a separate model, teacher, or additional rollouts. Check out the [example](https://github.com/huggingface/OpenEnv/blob/main/examples/echo_world_model/backends/tinker_echo_demo.py).\n\n## RL Example with Tinker and OpenEnv\n\n```\ngit clone https://github.com/huggingface/OpenEnv.git\ncd OpenEnv\n\n# Add TINKER_API_KEY=... to .env, then run:\nuv run --env-file .env \\\n  examples/echo_world_model/backends/tinker_echo_demo.py\n```\n\nIf you’re working with Transformers Reinforcement Learning we suggest using Inkling as a teacher model in a knowledge distillation setup. For example, take advantage of Inkling’s document understanding abilities to improve the performance of a smaller (on-device) model. In [this example](https://github.com/huggingface/trl/blob/main/examples/scripts/gold.py), we use the transformer reinforcement learning library and the GOLD algorithm to distill knowledge. GOLD is handy here because it matches token logits between different tokenizers, so you can distill to any model on the hub.\n\n## SLURM Scripts\n\nTo deploy Inkling on a cluster, we provide SLURM scripts serving with transformers API, as well as how to query the endpoint with different modalities. You can adapt these scripts to vLLM or SGlang by updating the commands. These scripts live [here](https://huggingface.co/buckets/merve/inkling).\n\n## SLURM `sbatch` Script\n\n``` bash\n#!/bin/bash\n#SBATCH --job-name=inkling-generate\n#SBATCH --partition=hopper-prod\n#SBATCH --qos=normal\n#SBATCH --nodes=4\n#SBATCH --gpus-per-node=8\n#SBATCH --ntasks-per-node=1\n#SBATCH --cpus-per-task=88\n#SBATCH --mem=0\n#SBATCH --exclusive\n#SBATCH --time=01:00:00\n#SBATCH --output=/fsx/merve/logs/inkling-generate-%j.out\n\n# Usage:\n#   sbatch submit_inkling_generate.sbatch\n#   PROMPT=\"What is in this image?\" IMAGE=/path/cat.png sbatch submit_inkling_generate.sbatch\n#   PROMPT=\"Transcribe this.\" AUDIO=/path/clip_16k.wav sbatch submit_inkling_generate.sbatch\n\nset -euo pipefail\n\nMODEL_PATH=${MODEL_PATH:-/fsx/pedro/tm/models/tml-model-share}\nPROCESSOR_PATH=${PROCESSOR_PATH:-${MODEL_PATH}}\nPROMPT=${PROMPT:-\"Explain tensor parallelism in simple terms.\"}\nIMAGE=${IMAGE:-}\nAUDIO=${AUDIO:-}\nMAX_NEW_TOKENS=${MAX_NEW_TOKENS:-512}\nTHINKING_EFFORT=${THINKING_EFFORT:-}\nTRANSFORMERS_SRC=${TRANSFORMERS_SRC:-/fsx/merve/transformers-tm/new-model-addition-tm}\nSCRIPT=${SCRIPT:-${TRANSFORMERS_SRC}/scripts/generate_inkling.py}\n\nNNODES=${SLURM_NNODES:-4}\nGPUS_PER_NODE=${GPUS_PER_NODE:-8}\nMASTER_PORT=${MASTER_PORT:-29500}\n\nmodule load cuda/12.8\nsource /fsx/merve/cluster/bin/activate\n\nexport FI_PROVIDER=efa\nexport HF_HOME=${HF_HOME:-/fsx/merve/hf_cache}\nexport PYTHONPATH=\"${TRANSFORMERS_SRC}/src:${PYTHONPATH:-}\"\nexport PYTHONUNBUFFERED=1\n\nHEAD_NODE=$(scontrol show hostnames \"$SLURM_JOB_NODELIST\" | head -n1)\nHEAD_IP=$(srun --nodes=1 --ntasks=1 -w \"$HEAD_NODE\" hostname -I | awk '{print $1}')\n\necho \"Nodes      = $SLURM_JOB_NODELIST\"\necho \"Head node  = $HEAD_NODE ($HEAD_IP)\"\necho \"Model      = $MODEL_PATH\"\necho \"Processor  = $PROCESSOR_PATH\"\necho \"Prompt     = $PROMPT\"\necho \"Image(s)   = ${IMAGE:-<none>}\"\necho \"Audio(s)   = ${AUDIO:-<none>}\"\n\nSCRIPT_ARGS=(--model-path \"$MODEL_PATH\" --processor-path \"$PROCESSOR_PATH\"\n             --prompt \"$PROMPT\" --max-new-tokens \"$MAX_NEW_TOKENS\")\nfor img in $IMAGE; do SCRIPT_ARGS+=(--image \"$img\"); done\nfor aud in $AUDIO; do SCRIPT_ARGS+=(--audio \"$aud\"); done\n[[ -n \"$THINKING_EFFORT\" ]] && SCRIPT_ARGS+=(--thinking-effort \"$THINKING_EFFORT\")\n\nsrun --ntasks-per-node=1 torchrun \\\n  --nnodes=\"$NNODES\" --nproc-per-node=\"$GPUS_PER_NODE\" \\\n  --rdzv-backend=c10d --rdzv-endpoint=\"${HEAD_IP}:${MASTER_PORT}\" --rdzv-id=\"$SLURM_JOB_ID\" \\\n  \"$SCRIPT\" \"${SCRIPT_ARGS[@]}\"\n```\n\nAlso available [here](https://huggingface.co/buckets/huggingface/inkling-blog-assets/resolve/submit_inkling_generate.sbatch?download=true).\n\n## Python generation script\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Multimodal generation with the inkling (tml) model in plain Transformers.\n\nRuns one prompt, optionally with attached image(s) and/or audio clip(s), and\nprints the model's answer. The model is sharded across all visible GPUs with\n`tp_plan=\"auto\"`, so launch it under `torchrun` (see\n`submit_inkling_generate.sbatch`) for a multi-GPU or multi-node checkpoint.\n\n    torchrun --nnodes=1 --nproc-per-node=8 generate_inkling.py \\\n        --model-path /path/to/tml-model-share \\\n        --image cat.png --prompt \"What is in this image?\"\n\n    torchrun ... generate_inkling.py --audio clip.wav --prompt \"Transcribe this.\"\n    torchrun ... generate_inkling.py --prompt \"Explain tensor parallelism.\"\n\"\"\"\n\nfrom __future__ import annotations\n\nimport argparse\nimport io\nimport os\nimport urllib.request\nimport wave\n\nimport numpy as np\n\n# Set the allocator configuration before importing torch.\nos.environ.setdefault(\"PYTORCH_CUDA_ALLOC_CONF\", \"expandable_segments:True\")\nimport torch  # noqa: E402\n\nfrom transformers import (  # noqa: E402\n    AutoProcessor,\n    InklingConfig,\n    InklingForConditionalGeneration,\n)\n\ndef load_image(source: str):\n    from PIL import Image\n\n    if source.startswith((\"http://\", \"https://\")):\n        with urllib.request.urlopen(source, timeout=60) as response:\n            source = io.BytesIO(response.read())\n    return Image.open(source).convert(\"RGB\")\n\ndef load_wav(path: str, expected_rate: int) -> np.ndarray:\n    \"\"\"Decode a mono float32 waveform from PCM WAV using the standard library.\"\"\"\n    with wave.open(path, \"rb\") as wav_file:\n        rate = wav_file.getframerate()\n        n_channels = wav_file.getnchannels()\n        sample_width = wav_file.getsampwidth()\n        frames = wav_file.readframes(wav_file.getnframes())\n\n    if rate != expected_rate:\n        raise SystemExit(\n            f\"{path}: sample rate {rate} != {expected_rate}; \"\n            \"resample to 16 kHz first\"\n        )\n\n    if sample_width == 2:\n        samples = np.frombuffer(frames, dtype=np.int16).astype(np.float32)\n        samples /= 32768.0\n    elif sample_width == 4:\n        samples = np.frombuffer(frames, dtype=np.int32).astype(np.float32)\n        samples /= 2147483648.0\n    else:\n        raise SystemExit(\n            f\"{path}: unsupported sample width {sample_width} bytes \"\n            \"(use 16/32-bit PCM)\"\n        )\n\n    if n_channels > 1:\n        samples = samples.reshape(-1, n_channels).mean(axis=1)\n\n    return samples\n\ndef build_messages(\n    prompt: str,\n    images: list[str],\n    audios: list[str],\n    sampling_rate: int,\n    effort: float | None,\n):\n    \"\"\"Build a user message with attached media followed by the text question.\n\n    An optional reasoning effort is rendered as a leading system message.\n    \"\"\"\n    content: list[dict[str, object]] = []\n    for image in images:\n        content.append({\"type\": \"image\", \"image\": load_image(image)})\n    for audio in audios:\n        content.append({\"type\": \"audio\", \"audio\": load_wav(audio, sampling_rate)})\n    content.append({\"type\": \"text\", \"text\": prompt})\n\n    messages = []\n    if effort is not None:\n        messages.append(\n            {\n                \"role\": \"system\",\n                \"content\": [\n                    {\n                        \"type\": \"text\",\n                        \"text\": f\"Thinking effort level: {effort}\",\n                    },\n                ],\n            },\n        )\n    messages.append({\"role\": \"user\", \"content\": content})\n    return messages\n\ndef main() -> None:\n    parser = argparse.ArgumentParser(\n        description=\"Multimodal generation with the inkling (tml) model.\"\n    )\n    parser.add_argument(\n        \"--model-path\",\n        default=os.environ.get(\n            \"MODEL_PATH\",\n            \"/fsx/pedro/tm/models/tml-model-share\",\n        ),\n    )\n    parser.add_argument(\n        \"--processor-path\",\n        default=os.environ.get(\"PROCESSOR_PATH\"),\n        help=\"Defaults to --model-path.\",\n    )\n    parser.add_argument(\n        \"--prompt\",\n        default=\"Explain tensor parallelism in simple terms.\",\n    )\n    parser.add_argument(\n        \"--image\",\n        action=\"append\",\n        default=[],\n        help=\"Image file path or HTTP(S) URL (repeatable).\",\n    )\n    parser.add_argument(\n        \"--audio\",\n        action=\"append\",\n        default=[],\n        help=\"16 kHz mono PCM WAV path (repeatable).\",\n    )\n    parser.add_argument(\n        \"--max-new-tokens\",\n        type=int,\n        default=int(os.environ.get(\"MAX_NEW_TOKENS\", \"512\")),\n    )\n    parser.add_argument(\n        \"--thinking-effort\",\n        type=float,\n        default=None,\n        help=\"Reasoning effort from 0.0 to 1.0.\",\n    )\n    parser.add_argument(\"--do-sample\", action=\"store_true\")\n    parser.add_argument(\"--temperature\", type=float, default=1.0)\n    parser.add_argument(\"--top-p\", type=float, default=1.0)\n    # torchrun may append launcher arguments after the script path.\n    args, _ = parser.parse_known_args()\n\n    rank = int(os.environ.get(\"RANK\", \"0\"))\n    local_rank = int(os.environ.get(\"LOCAL_RANK\", \"0\"))\n    torch.cuda.set_device(local_rank)\n    device = torch.device(\"cuda\", local_rank)\n\n    processor = AutoProcessor.from_pretrained(\n        args.processor_path or args.model_path\n    )\n    config = InklingConfig.from_pretrained(args.model_path)\n    model = InklingForConditionalGeneration.from_pretrained(\n        args.model_path,\n        config=config,\n        dtype=torch.bfloat16,\n        tp_plan=\"auto\",\n    )\n    model.eval()\n\n    messages = build_messages(\n        args.prompt,\n        args.image,\n        args.audio,\n        processor.feature_extractor.sampling_rate,\n        args.thinking_effort,\n    )\n    inputs = processor.apply_chat_template(\n        messages,\n        tokenize=True,\n        return_dict=True,\n        return_tensors=\"pt\",\n        add_generation_prompt=True,\n    ).to(device)\n    if \"pixel_values\" in inputs:\n        # The vision tower runs in the model dtype.\n        inputs[\"pixel_values\"] = inputs[\"pixel_values\"].to(\n            dtype=torch.bfloat16\n        )\n\n    generate_kwargs = {\n        \"max_new_tokens\": args.max_new_tokens,\n        \"do_sample\": args.do_sample,\n        \"pad_token_id\": config.eos_token_id,\n    }\n    if args.do_sample:\n        generate_kwargs[\"temperature\"] = args.temperature\n        generate_kwargs[\"top_p\"] = args.top_p\n\n    with torch.no_grad():\n        generated = model.generate(**inputs, **generate_kwargs)\n\n    if rank == 0:\n        new_tokens = generated[0, inputs[\"input_ids\"].shape[1] :]\n        answer = processor.tokenizer.decode(\n            new_tokens,\n            skip_special_tokens=False,\n        )\n        print(\n            f\"[prompt {inputs['input_ids'].shape[1]} tok \"\n            f\"| images {len(args.image)} | audios {len(args.audio)} \"\n            f\"| generated {new_tokens.numel()} tok]\",\n            flush=True,\n        )\n        print(answer, flush=True)\n\nif __name__ == \"__main__\":\n    main()\n```\n\nAlso available [here](https://huggingface.co/buckets/huggingface/inkling-blog-assets/resolve/generate_inkling.py?download=true).\n\n## Benchmark Results\n\n| Benchmark | GLM-5.2 | GLM-5.1 | Qwen3.7-Max | MiniMax M3 | DeepSeek-V4-Pro | Claude Opus 4.8 | GPT-5.5 | Gemini 3.1 Pro |\n|---|---|---|---|---|---|---|---|---|\n| Reasoning | ||||||||\n| HLE | 40.5 | 31 | 41.4 | 37 | 37.7 | 49.8* | 41.4* | 45 |\n| HLE (w/ Tools) | 54.7 | 52.3 | 53.5 | - | 48.2 | 57.9* | 52.2* | 51.4* |\n| CritPt | 20.9 | 4.6 | 13.4 | 3.7 | 12.9 | 20.9 | 27.1 | 17.7 |\n| AIME 2026 | 99.2 | 95.3 | 97 | - | 94.6 | 95.7 | 98.3 | 98.2 |\n| HMMT Nov. 2025 | 94.4 | 94 | 95 | 84.4 | 94.4 | 96.5 | 96.5 | 94.8 |\n| HMMT Feb. 2026 | 92.5 | 82.6 | 97.1 | 84.4 | 95.2 | 96.7 | 96.7 | 87.3 |\n| IMOAnswerBench | 91.0 | 83.8 | 90 | - | 89.8 | 83.5 | - | 81 |\n| GPQA-Diamond | 91.2 | 86.2 | 90 | 93 | 90.1 | 93.6 | 93.6 | 94.3 |\n| Coding | ||||||||\n| SWE-bench Pro | 62.1 | 58.4 | 60.6 | 59 | 55.4 | 69.2 | 58.6 | 54.2 |\n| NL2Repo | 48.9 | 42.7 | 47.2 | 42.1 | 35.5 | 69.7 | 50.7 | 33.4 |\n| DeepSWE | 46.2 | 18 | 18 | 20 | 8 | 58 | 70 | 10 |\n| ProgramBench | 63.7 | 50.9 | - | - | 47.8 | 71.9 | 70.8 | 39.5 |\n| Terminal Bench 2.1 (Terminus-2) | 81.0 | 63.5 | 75 | 65 | 64 | 85 | 84 | 74 |\n| Terminal Bench 2.1 (Best Reported Harness) | 82.7 | 69 | - | - | - | 78.9 | 83.4 | 70.7 |\n| FrontierSWE (Dominance) | 74.4 | 30.5 | - | - | 29.0 | 75.1 | 72.6 | 39.6 |\n| PostTrainBench | 34.3 | 20.1 | - | - | - | 37.2 | 28.4 | 21.6 |\n| SWE-Marathon | 13.0 | 1.0 | - | - | - | 26.0 | 12.0 | 4.0 |\n| Agentic | ||||||||\n| MCP-Atlas (Public Set) | 76.8 | 71.8 | 76.4 | 74.2 | 73.6 | 77.8 | 75.3 | 69.2 |\n| Tool-Decathlon | 48.2 | 40.7 | - | - | 52.8 | 59.9 | 55.6 | 48.8 |", "url": "https://wpnews.pro/news/welcome-inkling-by-thinking-machines", "canonical_source": "https://huggingface.co/blog/thinkingmachines-inkling", "published_at": "2026-07-15 00:00:00+00:00", "updated_at": "2026-07-15 18:43:50.402535+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure"], "entities": ["Thinking Machines", "Hugging Face", "Inkling", "transformers", "SGLang", "llama.cpp"], "alternates": {"html": "https://wpnews.pro/news/welcome-inkling-by-thinking-machines", "markdown": "https://wpnews.pro/news/welcome-inkling-by-thinking-machines.md", "text": "https://wpnews.pro/news/welcome-inkling-by-thinking-machines.txt", "jsonld": "https://wpnews.pro/news/welcome-inkling-by-thinking-machines.jsonld"}}