{"slug": "sentence-piece-model-from-gguf", "title": "sentence piece model from gguf", "summary": "A developer shared a Python script that extracts a SentencePiece tokenizer model from GGUF files, enabling the use of the tokenizer with the sentencepiece library. The approach reads token, score, and token type fields from the GGUF metadata and reconstructs a ModelProto, allowing direct encoding and decoding of text without loading the full model.", "body_md": "\n\n``` python\nfrom gguf.gguf_reader import GGUFReader\npython\ndef get_string_array_field(gguf_reader: GGUFReader, key: str):\n    f = gguf_reader.get_field(key)\n    return [bytes(f.parts[d]).decode(\"utf-8\") for d in f.data]\n\ndef get_field(gguf_reader: GGUFReader, key: str):\n    f = gguf_reader.get_field(key)\n    if f is None:\n        return 0\n    if len(f.data) != 1:\n        raise NotImplementedError(f\"multiple data is not supported\")\n    part = f.parts[f.data[0]]\n    if len(part) != 1:\n        raise NotImplementedError(f\"multiple parts are not supported\")\n    value = part[0]\n    if isinstance(value, np.float32):\n        return float(value)\n    elif isinstance(value, np.uint32):\n        return int(value)\n    return value\ngguf_path = \"/path/to/mistral-7b-v0.1.Q6_K.gguf\"\ngguf_reader = GGUFReader(str(gguf_path))\ntokens = get_string_array_field(gguf_reader, \"tokenizer.ggml.tokens\")\n# merges = get_string_array_field(gguf_reader, \"tokenizer.ggml.merges\")\n# merges = [tuple(m.split(\" \")) for m in merges]\nscores = [p.item() for p in gguf_reader.get_field(\"tokenizer.ggml.scores\").parts[5:]]  #  why 5!?\ntoken_types = [p.item() for p in gguf_reader.get_field(\"tokenizer.ggml.token_type\").parts[5:]]\npython\nimport sentencepiece as spm\npython\nimport sentencepiece.sentencepiece_model_pb2 as model\nnormalizer_spec = model.NormalizerSpec(\n    name=\"identity\",\n    precompiled_charsmap=b\"\",\n    add_dummy_prefix=True,\n    remove_extra_whitespaces=False,\n    normalization_rule_tsv=b\"\",\n)\ntrainer_spec = model.TrainerSpec(\n  model_type=\"BPE\",\n  vocab_size=32000,\n  input_format=\"text\",\n  split_by_unicode_script=True,\n  split_by_whitespace=True,\n  split_by_number=True,\n  treat_whitespace_as_suffix=False,\n  split_digits=True,\n  allow_whitespace_only_pieces=True,\n  vocabulary_output_piece_score=True,\n  byte_fallback=True,\n  unk_id=0,\n  bos_id=1,\n  eos_id=2,\n  pad_id=-1,\n  unk_piece=\"<unk>\",\n  bos_piece=\"<s>\",\n  eos_piece=\"</s>\",\n  pad_piece=\"<pad>\",\n  pretokenization_delimiter=\"\",\n)\nm = model.ModelProto(trainer_spec=trainer_spec, normalizer_spec=normalizer_spec)\nfor token, score, token_type in zip(tokens, scores, token_types):\n    m.pieces.append(model.ModelProto.SentencePiece(piece=token, score=score, type=token_type))\ntokenizer = spm.SentencePieceProcessor(model_proto=m.SerializeToString())\ntokenizer.encode(\"how tall is obama\")\n[910, 9369, 349, 818, 2786]\ntokenizer.decode([910, 9369, 349, 818, 2786])\n'how tall is obama'\ntokenizer.id_to_piece(910)\n'▁how'\n```\n\n", "url": "https://wpnews.pro/news/sentence-piece-model-from-gguf", "canonical_source": "https://gist.github.com/veryfadly588-afk/1d11d3e77ddef6bd905f55d2679882e6", "published_at": "2026-09-07 15:11:18+00:00", "updated_at": "2026-09-07 15:28:25.932607+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models"], "entities": ["GGUF", "SentencePiece", "Mistral 7B"], "alternates": {"html": "https://wpnews.pro/news/sentence-piece-model-from-gguf", "markdown": "https://wpnews.pro/news/sentence-piece-model-from-gguf.md", "text": "https://wpnews.pro/news/sentence-piece-model-from-gguf.txt", "jsonld": "https://wpnews.pro/news/sentence-piece-model-from-gguf.jsonld"}}