Learn a from your own text. Write tokenizer.py, merge the most frequent character pairs and check that → decode returns the same text.
4
BPE tokenizer
A gives each text piece a number. Our character-based starts with individual characters, counts adjacent pairs and repeatedly merges the most frequent pair. The order of merges matters: encoding must replay that order.
This small teaching keeps separate and reserves 0 for unknown characters. It is not a full byte-level production tokenizer. It saves both the and to tokenizer.json. A trained with one vocabulary cannot safely use a different one.
Why tokenize — computers calculate only with numbers
A model only knows how to add and multiply. It cannot calculate with the letters "cat", so text must become a of integer IDs. The tokenizer does this.
: text → list of IDs. With the tokenizer trained on the default , "cat" → [9, 7, 23].
decode: list of IDs → text. [9, 7, 23] → "cat".
IDs are needed in three places.
: an ID picks one row of a table (step 5).
Output head: it gives a score for every ID in the vocabulary. The number of IDs is the number of next it can choose from.
: it looks at the of the correct ID (step 6).
The IDs themselves have no meaning. Nothing guarantees that IDs 9 and 10 are similar characters. Meaning appears through , in the embedding of each ID.
What unit to split by — characters, words, subwords
There are three ways to choose the token unit.
Characters: the vocabulary is small (a few dozen for English) and unknown characters are rare. But text gets long, so the model sees less content at once, and the model must do all the work of building word meanings from characters.
Words: text gets short, but the vocabulary grows to hundreds of thousands, and words not in the dictionary (new names, typos, ) cannot be handled.
Subwords (BPE and others): frequent pieces become one token, rare words are split into several pieces. It balances vocabulary size and text length, so most today use it.
This is also why you train the tokenizer on your own data. General-purpose tokenizers learn mostly from English text, so they chop text in other languages or special fields into small pieces. For example, " 150k" is 3 tokens with a tokenizer trained on job postings, but 6 tokens with a general-purpose tokenizer. Fewer tokens means more content fits in the same length, and training is faster too.
The BPE algorithm — merge the pair that appears together most often
BPE (Byte Pair Encoding) starts from characters and repeatedly merges the two pieces that most often appear side by side into one new token.
The starting vocabulary is each character that appears in the data.
Count every pair of adjacent tokens across the whole data.
Add the most frequent pair to the vocabulary as a new token, and replace every occurrence of that pair in the data with the new token.
Repeat the set number of times (merges in settings.json), or until no pair appears 2 or more times.
The experiment below merges "low lower lowest" four times. The text starts at 16 tokens and gets shorter with each merge, ending at 6 tokens. Watch how a token merged earlier becomes material for the next merge, as in l+o → lo and lo+w → low, so longer and longer pieces appear. As in the third merge, " "+"low", spaces also join pieces. That is because does not split on spaces beforehand. Real usually split by spaces and first (pre-splitting, pre-tokenizer) and then merge only within those pieces.
Check it yourself lab/bpe_trace.py
from tokenizer import BPE, merge
text = 'low lower lowest'
tok = BPE.train(text, 4)
ids = [tok.chars[c] for c in text]
show = lambda ids: '|'.join(tok.vocab[i] for i in ids)
print('start:', show(ids), len(ids))
for (a, b), new_id in tok.merges:
ids = merge(ids, (a, b), new_id)
print(repr(tok.vocab[a]), '+', repr(tok.vocab[b]), '->', show(ids), len(ids))
Create a lab folder in ~/xslm, type the into a file with the same name and run it. -m lab.name puts the (~/xslm) on the import , so it loads the tokenizer.py, engine.py and model.py you wrote in earlier steps. If you run python3 lab/name.py instead, looks only inside the lab folder and raises .
cd ~/xslm
mkdir -p lab
micro lab/bpe_trace.py
python3 -m lab.bpe_trace
This output comes from the default data and settings. If you changed the data or settings, the numbers will differ.
vocab: the list of tokens in ID order. The position in the list is the ID.
merges: the list of merge rules. One rule is [[left ID, right ID], new ID].
IDs are assigned like this.
ID 0 is always �. It stands for characters not in the vocabulary.
From ID 1 come the characters in the , in sorted order. In the default data, newline, space, period, colon, capital A and Q and 21 lowercase letters — 27 characters in all — are IDs 1–27.
After that, IDs follow the merge order. In the default data, the first merge is ":"+" " → ": " (ID 28), the second is "Q"+": " → "Q: " (ID 29), and the third is "A"+": " → "A: " (ID 30).
So vocabulary size = 1 + kinds of characters + number of merges, which for the default data is 1 + 27 + 12 = 40. Because IDs are assigned in sorted order, training again on the same data always gives the same IDs.
Encoding — reapply the rules in the order they were learned
To turn new text into IDs, first find the ID of each character (0 for unknown ones), then apply the saved merge rules one by one in the order they were learned.
Order matters because later rules use tokens made by earlier rules as material. The rule that makes "Q: " (Q + ": ") only matches if ": " has already been made. If you change the order, you get different tokens than during training.
The merge scans the list from the left and merges without overlapping. If two items match the pair, it puts in the new ID and skips two items; otherwise it copies one item as is. So merging a+a in "aaa" gives [aa, a], never [a, aa].
decode just takes the vocab piece for each ID and joins them. Since every piece is made by joining original characters, decoding what you encoded returns exactly the original text (). Only unknown characters are turned into � and do not come back.
There is also a trap at boundaries. In the training data, "A:" is always followed by a space, so "A: " was merged into one token. If a ends with "A:" without the space, it becomes two tokens, "A" and ":", which never appeared separately in the training data. End the generation prompt with "A: ", exactly like the training format, space included.
Newline is never merged — where generation stops
When train in tokenizer.py counts pairs, it skips a pair if the joined text of the two pieces contains a newline. So newline always remains a token on its own (ID 1).
The reason is that generate.py stops generating when it samples the newline token. In the training data, answers end with a newline, so the model learns to output a newline when it finishes an answer. If a period and newline were merged into one token like ".\n", the newline would hide inside another token and generation could not find where to stop.
Real language models do this with a special end token (). XSLM uses newline like an EOS.
Unknown characters — ID 0 � and byte-level BPE
The XSLM tokenizer knows only the characters that appear in the training data. The default data has no capital C, no Korean and no emoji, so in the experiment below the C of Cat, 안녕 and 😀 all become ID 0. ID 0 only means "unknown character" and the original character is lost, so decoding gives �.
Many tokenizers since start from bytes instead of characters (byte-level BPE).
UTF-8 writes one character in 1–4 bytes. As the experiment shows, a is 1 byte, 안 is 3 bytes and 😀 is 4 bytes.
There are only 256 possible bytes, so a starting vocabulary of 256 can write any text in the world and no character is ever unknown.
In exchange, unseen characters get split into several byte tokens.
XSLM is character-based to keep it easy to read. The fix is to include the characters you will use in the training data.
Check it yourself lab/unknown.py
import json
from pathlib import Path
from tokenizer import BPE
tok = BPE(**json.loads(Path('data/tokenizer.json').read_text()))
for text in ['cat', 'Cat', '안녕', '😀']:
print(text, tok.encode(text), list(text.encode('utf-8')))
cd ~/xslm
mkdir -p lab
micro lab/unknown.py
python3 -m lab.unknown
Real tokenizers put agreed-upon tokens that never appear in text at the front of the vocabulary. These four commonly sit at IDs 0–3.
[PAD]: fills the end of shorter texts when texts of different lengths are computed as one group (batch).
[UNK]: a piece not in the vocabulary. The same as XSLM’s ID 0 �.
[BOS]: marks the beginning of a text.
[EOS]: marks the end of a text. Generation stops when this token appears.
Data format markers ([QUESTION], [ANSWER] and so on) can also be registered as special tokens so they are never split. Chat models use special tokens to separate the user’s and the assistant’s turns.
Of these, XSLM keeps only [UNK], as ID 0. It trains one window at a time so it needs no padding, newline serves as the end marker, and BPE merges Q: and A: into single tokens ("Q: ", "A: ") on its own.
Choosing the vocabulary size — what the number of merges changes
The vocabulary size (vocab_size) is the number of kinds of tokens, and IDs run from 0 to vocab_size − 1. When you add merges, two things move in opposite directions.
Text gets shorter. In the experiment below, the training data is 309 tokens with 0 merges, 218 tokens with 12, and 138 tokens with 48. More text fits in the 12 context slots.
The model gets bigger. The embedding table (token) and the output head (head) have one row per vocabulary entry, so when the vocabulary goes from 40 to 76, each of the two tables grows from 320 numbers to 608.
If it is too big, there are many rare tokens and too few examples to learn each one. Large data usually uses a vocabulary of 16,000–32,000, and very small data (10KB or less) 500–1,000. A rough estimate is "number of characters in the data ÷ 100–500".
Merging can stop earlier than the set count. If no pair appears 2 or more times, merging gains nothing. This is common with small data and is normal.
If you retrain the tokenizer, the meaning of the IDs changes, so the model must also be trained from scratch. If the tokenizer and the model have different vocabulary sizes, the row count of the embedding table does not match and the checkpoint cannot be loaded.
Check it yourself lab/token_count.py
from pathlib import Path
from tokenizer import BPE
text = Path('data/train.txt').read_text(encoding='utf-8')
for count in [0, 6, 12, 24, 48]:
tok = BPE.train(text, count)
print('merges', count, 'vocab', len(tok.vocab), 'tokens', len(tok.encode(text)))
cd ~/xslm
mkdir -p lab
micro lab/token_count.py
python3 -m lab.token_count
Type tokenizer.py, run it and inspect the . Follow merge with a pencil: when two IDs match, the new ID and advance two positions; otherwise advance one. Encode then decode a training sentence.
cd ~/xslm
micro tokenizer.py
python3 tokenizer.py
python3 -m json.tool data/tokenizer.json
merge combines one pair without overlapping. train learns the ordered rules. encode replays them; decode joins vocabulary pieces. The final assertion checks reversibility.
import json
from collections import Counter
from pathlib import Path
def merge(ids, pair, new_id):
result, i = [], 0
while i < len(ids):
if tuple(ids[i:i + 2]) == tuple(pair):
result.append(new_id)
i += 2
else:
result.append(ids[i])
i += 1
return result
class BPE:
def __init__(self, vocab, merges):
self.vocab, self.merges = vocab, merges
self.chars = {c: i for i, c in enumerate(vocab) if len(c) == 1}
@classmethod
def train(cls, text, count):
vocab = ['\ufffd'] + sorted(set(text) - {'\ufffd'})
ids = [vocab.index(c) for c in text]
merges = []
for _ in range(count):
pairs = Counter((a, b) for a, b in zip(ids, ids[1:])
if '\n' not in vocab[a] + vocab[b])
if not pairs:
break
pair, frequency = pairs.most_common(1)[0]
if frequency < 2:
break
new_id = len(vocab)
vocab.append(vocab[pair[0]] + vocab[pair[1]])
merges.append([list(pair), new_id])
ids = merge(ids, pair, new_id)
return cls(vocab, merges)
def encode(self, text):
ids = [self.chars.get(c, 0) for c in text]
for pair, new_id in self.merges:
ids = merge(ids, pair, new_id)
return ids
def decode(self, ids):
return ''.join(self.vocab[i] for i in ids)
def state(self):
return {'vocab': self.vocab, 'merges': self.merges}
if __name__ == '__main__':
settings = json.loads(Path('settings.json').read_text())
text = Path('data/train.txt').read_text(encoding='utf-8')
tokenizer = BPE.train(text, settings['merges'])
Path('data/tokenizer.json').write_text(
json.dumps(tokenizer.state(), ensure_ascii=False), encoding='utf-8')
assert tokenizer.decode(tokenizer.encode(text)) == text
print('vocab:', len(tokenizer.vocab), 'merges:', len(tokenizer.merges))
print('roundtrip: OK', tokenizer.encode('Q: hi\nA: '))
Line-by-line walkthrough
tokenizer.py merge — merging one pair without overlap
def merge(ids, pair, new_id): result, i = [], 0 while i < len(ids): if tuple(ids[i:i + 2]) == tuple(pair): result.append(new_id) i += 2 else: result.append(ids[i]) i += 1 return result
It scans the ids list from the left with i. ids[i:i + 2] is the current item and the next one (one item at the end). If it equals pair, it puts one new_id in the result and adds 2 to i to skip two items. Otherwise it puts the current item in as is and adds 1. It converts to tuple for the comparison because rules read from JSON are lists (a list and a tuple are not equal with ==). It passes through the list once, so the time is proportional to its length.
tokenizer.py Building a tokenizer from saved state
class BPE: def __init__(self, vocab, merges): self.vocab, self.merges = vocab, merges self.chars = {c: i for i, c in enumerate(vocab) if len(c) == 1}
A BPE object has only two things: vocab and merges. So it can be rebuilt by unpacking the read from tokenizer.json straight into BPE(…) as keyword arguments, which is what train.py and generate.py do. chars is a "character → ID" lookup table of only the one-character tokens. It is used in the first stage of encode to find the ID of each character quickly.
tokenizer.py train — learning the merge rules
@classmethod def train(cls, text, count): vocab = ['\ufffd'] + sorted(set(text) - {'\ufffd'}) ids = [vocab.index(c) for c in text] merges = [] for _ in range(count): pairs = Counter((a, b) for a, b in zip(ids, ids[1:]) if '\n' not in vocab[a] + vocab[b]) if not pairs: break pair, frequency = pairs.most_common(1)[0] if frequency < 2: break new_id = len(vocab) vocab.append(vocab[pair[0]] + vocab[pair[1]]) merges.append([list(pair), new_id]) ids = merge(ids, pair, new_id) return cls(vocab, merges)
@classmethod makes a function called as BPE.train(…) without an object, which builds and returns an object.
The first vocabulary puts \ufffd (�) at ID 0 and appends the characters in the text, sorted with sorted. Because they are sorted, the IDs are always the same.
ids is the list of each character’s ID (vocab.index).
Each iteration counts adjacent pairs (a, b) with Counter. zip(ids, ids[1:]) pairs up neighboring items. If the joined text of the two pieces contains a newline, it is not counted.
most_common(1) gives the most frequent pair and its count. On a tie, the pair counted first, that is, the pair that appeared first in the text, wins.
It stops if there are no pairs or the most frequent pair appears only once.
The new ID is (vocab), the next ID in the vocabulary. It adds the joined text of the two pieces to the vocabulary, records the rule [[a, b], new ID], and replaces ids with the merged list to prepare the next iteration.
tokenizer.py encode — text to IDs
def encode(self, text): ids = [self.chars.get(c, 0) for c in text] for pair, new_id in self.merges: ids = merge(ids, pair, new_id) return ids
chars.get(c, 0) finds a character’s ID, or gives 0 (�) if there is none. Then it merges with each rule in merges, one by one, in the order they were learned. It scans the list again for every rule, so it slows down with many rules and long text. Real libraries merge faster using a priority table of rules.
tokenizer.py decode and state — IDs to text, saving state
def decode(self, ids): return ''.join(self.vocab[i] for i in ids) def state(self): return {'vocab': self.vocab, 'merges': self.merges}
decode takes the vocab piece for each ID and joins them with ''.join. state returns a dictionary that can be saved with json. Rules are recorded as lists instead of tuples, so the shape stays the same after saving to JSON and reading it back.
It trains with merges from settings.json and data/train.txt, and saves to tokenizer.json with ensure_ascii=False. Without this option, Korean text would be saved as escapes like \uc548. The encodes → decodes all the training data and checks that it matches the original exactly. The last line shows what IDs the generation prompt "Q: hi" followed by "A: " on the next line becomes. With the default data it is [29, 13, 14, 1, 30], that is, "Q: ", h, i, newline, "A: ".
Check your understanding
roundtrip: OK means the entire training text survives encode → decode. Change the merge count in settings.json and compare token counts. Test an unseen character: it becomes �, not a newly learned token.
If you get stuck
No merges can be valid when no pair repeats. If you change text or merge rules, rebuild the tokenizer and start new model ; keep old checkpoints in an exported .
Frequently asked questions
Why build it yourself instead of using a tokenizer library?
In real development you use libraries such as tokenizers or tiktoken. They are written in and similar languages and process big data quickly. XSLM builds it in 60 lines so you can see for yourself how merge rules are made and in what order they are applied. The tokenizer.json a library saves contains the same vocab and merges.
roundtrip is OK, but my question turns into �.
roundtrip checks only the training data. If your question has characters not in the training data (capital letters, Korean, symbols), they become ID 0. Add lines with those characters to raw.txt and redo prepare → tokenizer → training with a new checkpoint.
Is one token one word?
No. A token is a piece BPE saw often, so it can be a word with a space attached like "is " or part of a word like " ca". In real models, one English word is on average a little more than one token, and other languages may be split even finer. That is why pricing and context length are counted in tokens, not words.
