I tried running it on Colab:
Thanks for sharing Llamatron, and especially for making the checks part of the release. The interesting question is not merely whether a roughly 59 GB Nemotron checkpoint can be published as an 18,725,978,368-byte GGUF. It is whether deciding where to spend the remaining precision lets a compact model retain useful behavior better than a simpler compression at a comparable size.
I tried the published GGUF on a Google Colab NVIDIA L4, using JamePeng’s CUDA-enabled llama_cpp_python. The exact file passed a full SHA-256 comparison with the Hub’s expected digest, the native runtime reported 53/53 layers offloaded to the GPU, and short inference worked. On five deliberately simple questions, the identifiable final answers were semantically correct in 5/5 cases.
There was also a practical wrinkle: in this configuration, the raw chat API’s message.content included thinking-like English text and a literal </think> marker before the final answer. For the three prompts that explicitly requested a strict final-only format, the final answer was useful, but the whole raw string did not meet the request: JSON-only, one-short-sentence-only, and translation-only were 0/3 compliant when tested against the entire returned content. That is a model/template/runtime/response-parsing observation; I would not attribute it to the importance-guided quantization.
| Check | Observed | Scope |
|---|---|---|
| Published artifact | 18,725,978,368 bytes; full-file SHA-256 matched |
Exact GGUF identity, not quality |
| Colab execution | L4; native CUDA loaded; 53/53 GPU layers |
Short-run compatibility in this environment |
| Five small tasks | 5/5 final semantic answers correct | Tiny hand-written sanity check |
| Strict raw-output tasks | 0/3 satisfied the whole-string format | Extra thinking-like material in content |
| Repeat run | Same five content strings, finish reasons, and usage fields |
Fixed model and deterministic settings |
| Guided versus plain | Not independently measured | No exact same-size plain control in my run |
This is a basic outside reproduction, not a benchmark score. “The final answer was correct” and “the API’s raw string satisfies a strict parser” are different acceptance criteria.
First, the smaller artifact is demonstrably runnable on at least one current Colab GPU configuration. Its file identity, backend, placement, and actual text generation were all checked. That is useful independently of the method comparison.
Second, a small output-token cap can obscure an otherwise reachable final answer. In an earlier smoke check with the same GGUF and prompt, a 128-token cap ended with finish_reason="length" during thinking-like content; with 256 tokens, the model finished its answer at 186 generated tokens. The first 128 generated tokens were exactly the prefix of the longer run. This does not establish the cause of the author’s own guided/plain examples, but it shows why template, thinking mode, stop reason, and generated-token budget should be recorded in short-output comparisons.
Third, the author-reported held-out English perplexities that I found in the model-card material are very close: guided 4.4819 ± 0.11663 and plain 4.4778 ± 0.11706 over 16 chunks of 1,024 tokens. The displayed guided mean is 0.0041 higher; that does not show a PPL improvement. Nor does a tiny difference between marginal means prove statistically meaningful inferiority, statistical equivalence, or the absence of task-specific benefits. I did not rerun this PPL comparison, and the matched per-chunk results were not part of my Colab probe.
So I would keep three questions separate:
finish_reason and the whole returned message.content versus reasoning_content, and parser behavior before deciding whether the weights failed to follow an instruction.
I’m not suggesting a large new evaluation as a prerequisite for discussing the release. These are options for different reader goals. The specific observations and their limitations are below for anyone interested in reproduction.
- Exact artifact, Colab environment, and what was verified The small outside result is positive and concrete: the exact Llamatron GGUF loaded with native CUDA offload on Colab L4 and returned correct final answers to five simple, varied prompts. I also found a useful output-routing caveat for strict consumers of the raw chat API. These observations add some evidence about running and integrating the published artifact, not a verdict on the author’s choice of quantization.
The same-size plain comparison remains the right anchor for deciding what importance guidance preserves, while a proper BF16 comparison asks another question again. If paired raw outputs or the precise plain control from the existing build checks are already available, even a few such records could make the discussion more informative without requiring another large-model run. Otherwise, I think the release, its documented checks, and this limited reproduction are already worthwhile information for future readers.
Thanks again for sharing both the compact model and the method behind it.