Tokenization

August 2026 – Vladislav Kruglikov

Tokenization converts raw text into a sequence of discrete token IDs that a language model can process. The tokenizer determines the model's vocabulary and strongly affects sequence length, training efficiency, and how well the model represents different languages, code, and rare words.

Character level tokenization

Character level tokenization represents text as one token per character. It requires no learned segmentation and can encode any text composed of characters in its vocabulary without splitting rare words into special cases.

This approach gives a very small vocabulary at the cost of much longer sequences. The cost is especially painful for standard self-attention during prefill because its computation and attention matrix grow quadratically with sequence length, $O(n^2)$ . A text that needs several times more tokens can therefore require far more than several times the attention work.

Word level tokenization

Word level tokenization assigns one token to each complete word. It can encode long text with fewer token IDs than character level tokenization, but it requires a very large vocabulary and still needs a way to handle unseen words. Even for English, which has relatively limited inflectional morphology, the vocabulary can be enormous. There is no exact word count because the result depends on whether inflected forms, compounds, technical terms, and other categories are counted separately. A frequently cited estimate is roughly one million words.

A large word level vocabulary also creates data sparsity. Forms such as cat, cats, and cat's have separate token embeddings and output parameters, so an occurrence of one form does not directly update the parameters of the others. The shared Transformer layers can still learn relationships between them from context, but each rare form receives fewer direct training examples. Learning good representations for all of them therefore generally requires more data. Subword tokenization reduces this problem by allowing related and previously unseen words to share smaller units, increasing parameter sharing across word forms, an effect that is particularly useful in low-data settings (Zhu et al., 2019).

Character level BPE

Byte Pair Encoding (BPE) provides a compromise between character level and word level tokenization. The character based version begins with individual characters and repeatedly finds the most frequent adjacent pair in the training corpus, merges that pair into a new token, and adds it to the vocabulary. Training stops after a chosen number of merges, which controls the vocabulary size. When the learned tokenizer is applied, frequent words or character sequences can become single tokens, while rare words are split into smaller known pieces.

Byte level BPE

Byte level BPE first converts text to UTF-8 and uses the 256 possible byte values as its initial vocabulary. It then learns the same kind of frequent-pair merges as character level BPE. Since every valid UTF-8 string is composed of these bytes, the tokenizer can encode any Unicode text without an unknown token. Even an emoji introduced after the tokenizer was trained remains representable, although it may be split into several byte tokens until a future tokenizer learns a merge for that sequence.

WordPiece

WordPiece is a subword tokenization method. Like character based BPE, it begins with a small vocabulary and incrementally adds larger character sequences. The difference is how it selects them. BPE merges the most frequent adjacent pair, while WordPiece favors pairs that occur together more often than expected from the frequencies of their individual parts.

A common WordPiece construction assigns every adjacent pair $(a,b)$ the score

\operatorname{score}(a,b)=\frac{\operatorname{count}(a,b)}{\operatorname{count}(a)\operatorname{count}(b)}

The pair count cannot exceed either individual count, so $\operatorname{count}(a,b)\leq\operatorname{count}(a)$ and $\operatorname{count}(a,b)\leq\operatorname{count}(b)$ . If all three counts are equal, every occurrence of $a$ is followed by $b$ and every occurrence of $b$ is preceded by $a$ . This is strong evidence that the pair behaves as a reusable unit, so adding $ab$ to the vocabulary is useful. WordPiece adds the merged token without removing $a$ or $b$ , since the individual pieces may still be needed to construct other words.

The pair with the highest score is merged. Suppose $(a,b)$ occurs 20 times, while $a$ occurs 100 times and $b$ occurs 80 times. Its score is $20/(100\times80)=0.0025$ . Another pair $(c,d)$ may occur only 10 times, but if $c$ occurs 20 times and $d$ occurs 10 times, its score is $10/(20\times10)=0.05$ . WordPiece merges $(c,d)$ first because its two parts occur together much more consistently. BPE would merge $(a,b)$ first because it considers only the pair counts, 20 and 10. This scoring rule is described in the Hugging Face WordPiece guide.

Unigram

Unigram builds its vocabulary in the opposite direction from BPE and WordPiece. It starts with a large set of candidate pieces, including individual characters and frequent substrings, then assigns a probability to every piece. For a segmentation $s=(s_1,\ldots,s_n)$ , the model multiplies the probabilities of its pieces

P(s)=\prod_{i=1}^{n}P(s_i)

The model is called Unigram because it treats the probability of each piece as independent of its neighbors. A string can usually be segmented in several ways, so training considers all valid segmentations and estimates the piece probabilities that maximize the likelihood of the corpus.

After estimating the probabilities, the algorithm measures how much the corpus likelihood would decrease if each candidate were removed. It prunes the least useful pieces, estimates the probabilities again, and repeats until it reaches the desired vocabulary size. Necessary single-character pieces are retained so that ordinary input remains representable. During encoding, the tokenizer uses dynamic programming to find the segmentation with the highest probability. Because the model assigns probabilities to alternative segmentations, it can also sample different segmentations during training as a form of regularization (Kudo, 2018).

Tokenization density and inference speed

Tokenization density measures the average number of tokens produced per word in a corpus

F=\frac{N_{\text{tokens}}}{N_{\text{words}}}

Lower density means that the tokenizer represents the same text with fewer tokens.

Tokenization density affects inference because a language model operates on tokens rather than directly on words or characters. More input tokens increase prefill work, make the attention matrix larger, consume more KV-cache memory, and use more of the model's fixed context window. More output tokens are even more expensive for latency because autoregressive decoding normally generates one token per model step. If one tokenizer represents the same answer with twice as many tokens, producing it can require roughly twice as many sequential decoding steps.