Less Is More: Reducing Token Counts Without Compromising Performance

Tokenization directly affects the inference efficiency of large language models, since fragmented tokenization increases sequence length and generation cost.

Although longer, multi-word tokens can reduce fertility, naively adding them often degrades language model performance.

We propose Thunder-Tok, a subword tokenizer that reduces fertility while preserving downstream performance.

Thunder-Tok first constructs a large seed vocabulary from corpus substrings and filters structurally incomplete candidates, including invalid Unicode byte fragments and word-boundary violations.

It then prunes the seed vocabulary using a likelihood-based token score derived from a uniform Jensen lower bound of the training-data probability.

Experiments show that Thunder-Tok reduces fertility by approximately 25% in English and 9% in Korean compared with the standard BPE tokenizer while maintaining competitive performance.