cd /news/large-language-models/my-first-1b-transformer-model-traine… · home › topics › large-language-models › article
[ARTICLE · art-160708] src=discuss.huggingface.co ↗ pub= topic=large-language-models verified=true sentiment=· neutral

My First 1B transformer model trained from scratch has published

An independent test of the published Freedream-1B-Instruct-V1 checkpoint on an NVIDIA L4 in BF16 confirmed the roughly 977M-parameter model loads and generates output, but found the released tokenizer_v2.json cannot round-trip the opening brace "{" through the direct tokenizers loader used by chat.py, encoding it as [178, 0] with ID 0 as <|unk|> under tokenizers 0.22.1 and 0.23.3. In the tester's smoke tests, two structured-output attempts generated <|unk|> IDs despite no UNK in the input prompts, and a sampled JSON attempt hit a raised 1,024-new-token ceiling without producing the requested object. The tester characterized the results as selected examples rather than a model benchmark or architecture assessment.

read2 min views1 publishedOct 30, 2026

For now, I tried running it myself: I ran the published Freedream-1B-Instruct-V1 on an NVIDIA L4 in BF16. The approximately 977M-parameter checkpoint loaded successfully, a forward pass returned finite logits, and an instrumented version of the repository’s inference code generated responses. This was an independent run of the released model, not a recreation of the training setup.

For example, to “What is the capital of Japan? Answer in one sentence.”, the sampled response began: The capital of Japan is Tokyo, …

Tokyo was correct; the rest of the generated sentence added details that should not be treated as verified facts. In a separate greedy run, the model also introduced itself as Freedream. These are individual smoke tests, not an accuracy or identity-stability benchmark.

One technical finding was particularly reproducible: the published tokenizer_v2.json cannot round-trip the opening brace { through the direct tokenizers used by chat.py. On the pinned artifact, encode("{", add_special_tokens=False).ids produced [178, 0], with ID 0 representing <|unk|>. The same result appeared under tokenizers 0.22.1 and 0.23.3. The inserted leading space is a separate ByteLevel behavior; losing { matters for JSON objects and Python dictionaries.

In the L4 traces, two structured-output attempts also generated <|unk|> IDs, although none of the input prompts contained UNK. Since Tokenizer.decode() normally skips special tokens, the displayed text can conceal those IDs. Other JSON attempts failed without generating UNK, however, so tokenizer coverage is not a complete explanation for the observed instruction-following behavior.

I also increased the initial 48-new-token smoke-test ceiling to 1,024 for a small follow-up panel. A sampled Python function eventually stopped but was still incorrect; a sampled JSON attempt reached the larger cap without producing the requested object. Several greedy controls stopped early yet also missed the requested format. The results fit some of the limitations already noted in the model card, but they are selected examples, not a model benchmark or architecture assessment.

I have included the reproducible details below so other readers can distinguish successful execution, tokenizer behavior, stopping, and answer correctness. The experiment does not evaluate the reported training optimizations.

── more in #large-language-models 4 stories · sorted by recency
── more on @freedream-1b-instruct-v1 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/my-first-1b-transfor…] indexed:0 read:2min 2026-10-30 · —