AMD EPYC 9565

Stanford PhD student Marcel Rød released Gigatoken, a Rust BPE tokenizer that encodes text at 24.53 GB/s on a 144-core AMD EPYC 9565 dual-socket setup, outperforming HuggingFace tokenizers by 989x and…

GigaToken, a new open-source tokenizer, claims to be up to ~1000x faster than HuggingFace's tokenizers and tiktoken for language model tokenization, achieving speeds of over 24 GB/s on an AMD EPYC 956…