New Rust tokenizer processes text nearly 1,000x faster

Gigatoken is an MIT-licensed Rust implementation of byte-pair encoding tokenization that runs at 24.53 GB/s on a 144-core AMD EPYC chip. That's roughly 989x faster than Hugging Face's tokenizers library and 681x faster than OpenAI's tiktoken, achieved with a custom pretokenizer rather than a faster merge algorithm.

Why it matters: Tokenization is a fixed cost paid on every LLM training run and every inference request, so an order-of-magnitude speedup there compounds across pipelines that currently burn meaningful CPU time on it. Being MIT-licensed and open source, it's the kind of infrastructure improvement that tends to get absorbed into existing tooling quickly rather than requiring teams to adopt a new framework.