03:32
2026-07-24
github.com
developer-tools
Gigatoken-rb – A Ruby port of gigatoken that's 1.6x faster
Gigatoken-rb, a Ruby port of the gigatoken BPE tokenizer, achieves 12 GB/s (2.78 billion tokens per second) on a Mac Studio M4 Max, running 1.6x faster than the upstream Python package on the same Rus…