{"slug": "my-first-1b-transformer-model-trained-from-scratch-has-published", "title": "My First 1B transformer model trained from scratch has published", "summary": "An independent test of the published Freedream-1B-Instruct-V1 checkpoint on an NVIDIA L4 in BF16 confirmed the roughly 977M-parameter model loads and generates output, but found the released tokenizer_v2.json cannot round-trip the opening brace \"{\" through the direct tokenizers loader used by chat.py, encoding it as [178, 0] with ID 0 as <|unk|> under tokenizers 0.22.1 and 0.23.3. In the tester's smoke tests, two structured-output attempts generated <|unk|> IDs despite no UNK in the input prompts, and a sampled JSON attempt hit a raised 1,024-new-token ceiling without producing the requested object. The tester characterized the results as selected examples rather than a model benchmark or architecture assessment.", "body_md": "For now, I tried running it myself:\n\nI ran the published [Freedream-1B-Instruct-V1](https://huggingface.co/Digilidaahz/Freedream-1B-Instruct-V1/tree/f0abc549c5584749b3e59193694728f1a7416983) on an NVIDIA L4 in BF16. The approximately **977M-parameter** checkpoint loaded successfully, a forward pass returned finite logits, and an instrumented version of the repository’s inference code generated responses. This was an independent run of the released model, not a recreation of the training setup.\n\nFor example, to **“What is the capital of Japan? Answer in one sentence.”**, the sampled response began:\n\nThe capital of Japan is Tokyo, …\n\n*Tokyo* was correct; the rest of the generated sentence added details that should not be treated as verified facts. In a separate greedy run, the model also introduced itself as Freedream. These are individual smoke tests, not an accuracy or identity-stability benchmark.\n\nOne technical finding was particularly reproducible: the published [`tokenizer_v2.json`](https://huggingface.co/Digilidaahz/Freedream-1B-Instruct-V1/blob/f0abc549c5584749b3e59193694728f1a7416983/tokenizer_v2.json) cannot round-trip the opening brace **`{`** through the direct `tokenizers` loader used by `chat.py`. On the pinned artifact, `encode(\"{\", add_special_tokens=False).ids` produced **`[178, 0]`**, with ID `0` representing `<|unk|>`. The same result appeared under `tokenizers` **0.22.1** and **0.23.3**. The inserted leading space is a separate ByteLevel behavior; losing `{` matters for JSON objects and Python dictionaries.\n\nIn the L4 traces, two structured-output attempts also generated `<|unk|>` IDs, **although none of the input prompts contained UNK**. Since `Tokenizer.decode()` normally skips special tokens, the displayed text can conceal those IDs. Other JSON attempts failed **without** generating UNK, however, so tokenizer coverage is not a complete explanation for the observed instruction-following behavior.\n\nI also increased the initial **48-new-token** smoke-test ceiling to **1,024** for a small follow-up panel. A sampled Python function eventually stopped but was still incorrect; a sampled JSON attempt reached the larger cap without producing the requested object. Several greedy controls stopped early yet also missed the requested format. The results fit some of the limitations already noted in the model card, but they are **selected examples, not a model benchmark or architecture assessment**.\n\nI have included the reproducible details below so other readers can distinguish **successful execution, tokenizer behavior, stopping, and answer correctness**. The experiment does not evaluate the reported training optimizations.", "url": "https://wpnews.pro/news/my-first-1b-transformer-model-trained-from-scratch-has-published", "canonical_source": "https://discuss.huggingface.co/t/my-first-1b-transformer-model-trained-from-scratch-has-published/191209#post_2", "published_at": "2026-10-30 05:04:47+00:00", "updated_at": "2026-10-30 05:17:26.484401+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-tools"], "entities": ["Freedream-1B-Instruct-V1", "Digilidaahz", "NVIDIA L4", "Hugging Face", "tokenizers"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/my-first-1b-transformer-model-trained-from-scratch-has-published", "markdown": "https://wpnews.pro/news/my-first-1b-transformer-model-trained-from-scratch-has-published.md", "text": "https://wpnews.pro/news/my-first-1b-transformer-model-trained-from-scratch-has-published.txt", "jsonld": "https://wpnews.pro/news/my-first-1b-transformer-model-trained-from-scratch-has-published.jsonld"}}