Build your folder, and settings before . Create ~/xslm, write settings.json and prepare.py, and split your Q/A text into training and .
3
Project structure
Keep everything in ~/xslm. holds raw text, text, text and the . checkpoints holds learned . The files live at the . Run all from that folder.
Each raw line contains Q: question | A: answer. prepare.py removes duplicate lines and keeps every fifth distinct line for validation. It replaces the separator with a . Validation is an unseen quiz; do not train the tokenizer on it. Use text you are allowed to use and invent examples instead of copying personal data.
Folder structure — who makes each file and when
When finished, the ~/xslm folder looks like this.
settings.json — you write it. The settings every reads.
data/raw.txt — you write it. The raw questions and answers.
data/train.txt and data/val.txt — prepare.py makes them. and validation data.
data/tokenizer.json — tokenizer.py makes it. The and .
checkpoints/model.json — train.py makes it and updates it on every training run. The weights and training state.
runner.log and runner.pid — created when you start the in the background. The output and the .
The other .py files — you write them. The program sources.
lab/ — experiments for each step (optional).
Keeping sources, data and results apart makes it clear what you may delete. Generated files in data/ and checkpoints/ can always be rebuilt, but the sources you wrote and raw.txt cannot, so keep them with an export.
Run order and rebuilding — what to redo when something changes
Files flow from front to back, so if you change something near the front, rebuild everything after it.
If you changed raw.txt: prepare.py → tokenizer.py → move the checkpoint aside and run train.py.
If you changed settings.json even slightly (merges, width, , steps, — anything): if you changed merges, start from tokenizer.py; otherwise move the checkpoint aside and run train.py right away.
If you only want more training: leave settings.json as it is and set the count with an option, like train.py --steps 50. Training resumes from the checkpoint.
If you trained more: stop and restart the running Runner so it loads the new weights.
train.py checks these rules itself. If the full settings, tokenizer or data fingerprint saved in the checkpoint differ in any way from the current ones, it stops with . Continuing to train old weights on changed data or settings would misalign the meaning or shape of and break things silently. To start fresh, move the old file aside, like checkpoints/model.json checkpoints/old.json.
The data format creates the ability — Q: … | A: …
One line of raw.txt is "Q: question | A: answer". prepare.py breaks the line before " | A:", so in the training data it becomes two lines.
Q: hi
A: hi friend.
By seeing this shape again and again, the learns: "the line after a Q: line starts with A:, and after A: comes answer-like text, then the line ends". When generating, if you give it a Q: question and then "A: " on the next line, the model continues with an answer out of its learned habit.
You can choose any format you like. You can also add markers such as [QUESTION]…[/QUESTION] and [ANSWER]…[/ANSWER]. The only rule to keep is consistency.
Every line uses the same markers in the same order.
The same meaning uses the same word. For example, unify phone, Tel and ☎ all as TEL:.
If you change the format, change the that uses it too. In , the default in generate.py and the "Q: … A: " in xchat.py are tied to this format.
The model learns a task only if the data has many examples of that input-to-output shape. teaches continuing text, question→answer teaches answering, and document→summary teaches summarizing.
Good data and bad data — amount, quality and rights
A model only imitates what it has seen. It is like someone who grew up hearing only Japanese not being able to speak English. So the data is the model.
Encoding: always save as . Other encodings break characters.
Good data: questions and answers in the same format, natural sentences that carry information.
Bad data: text that is only code or logs, text with too many emoji or addresses, broken sentences, the same sentence repeated.
Cleanup: remove leading and trailing spaces and blank lines, and keep each identical line only once. prepare.py does this.
Variety: write the same answer with several different questions (hi, hello, hey). That gives it a chance to answer questions it has never seen.
Rights and personal data: use only text you have the right to use, and invented examples instead of real names or phone numbers.
You need more data than you might think. As a rough guide, a model starts to follow the format at about 3,000 question-answer pairs, and the format becomes stable at about 10,000 pairs. XSLM’s 15 lines are a tiny practice set to check that the whole process really runs. Start by running the whole process end to end with small fake data like this, then add more data.
Training data and validation data — keep an unseen test aside
If you score only on training data, you cannot tell memorizing from learning. So you set part of it aside as validation data that is never used for training. If training goes down while goes up, the model has started memorizing ().
prepare.py splits like this.
It numbers the deduplicated lines from 0.
Lines whose number leaves remainder 0 when divided by 5 (lines 0, 5 and 10) go to validation, the rest to training. The 15 default lines become 12 training lines and 3 validation lines.
It splits whole question-answer pairs before turning them into . Cutting by characters would put the front of a pair in training and the back in validation, leaking the test questions ().
The tokenizer also learns from training data only. Nobody sees the validation data until the final test.
With very large data, the last 1% of all tokens is sometimes cut off for validation. With small data it is safer to split by question-answer pairs, as XSLM does.
Turn big data into token IDs ahead of time
XSLM turns all of train.txt into token IDs every time you run train.py. The data is a few hundred characters, so it is instant. When data grows to tens of , converting every time is wasteful, so you convert it once and save it to a file.
It saves token IDs as uint16 (2-byte integers). If the vocabulary is smaller than 65,536, each fits in 2 .
It is a number array, not text, so it becomes files called train.bin and val.bin.
During training, numpy.memmap reads only the needed parts instead of loading the whole file into .
When data grows even larger, it is split into several files that are trained in turn. The flow is the same: text → token IDs → fixed-length windows → training.
Switch to your own data — try questions and answers in your own language
Once you have finished the course, try replacing raw.txt with your own data.
Write one "Q: question | A: answer" per line, with at least five different lines. The assert in prepare.py checks the format and the number of lines.
Keep answers short. is 12 tokens, so the model sees only the last 12 tokens. In character-based , every character of a language like Korean starts as one token, so if the question and answer are long, the model cannot see the beginning.
The tokenizer turns characters not in the training data into the unknown character (ID 0). Put the characters you will use in questions into the data too.
Languages with many kinds of characters, such as Korean, get a bigger vocabulary and fewer repeated pairs, so merging tends to stop early. In that case, add more lines and raise merges.
Type and try
Create the folders first. Open each source below, type it into a file with exactly that name, save and close. Use for .py files, and for settings.json and raw.txt. settings.json is , so copy its punctuation carefully. Run prepare.py only after saving both the data and the script.
These are the small course defaults. width is the size of a token ; context is the number of visible tokens; learning_rate is the update size. Keep the same settings when resuming.
Type one Q/A pair per line. These invented English examples keep the first experiment short. Replace them later with your own consistent text.
Q: hello | A: hello friend.
Q: hi | A: hi friend.
Q: name | A: i am xslm.
Q: sun | A: the sun is warm.
Q: cat | A: a cat can nap.
Q: dog | A: a dog can run.
Q: red | A: red is a color.
Q: blue | A: blue is a color.
Q: learn | A: i learn from text.
Q: code | A: i can learn code.
Q: moon | A: the moon is in the sky.
Q: rain | A: rain is water.
Q: book | A: i can read a book.
Q: tea | A: tea is warm.
Q: bye | A: bye friend.
prepare.pyComplete source to type
Read UTF-8 text → remove duplicates → split whole records → write train and validation files. Separating records before tokenization avoids sharing the exact same example between both sets.
from pathlib import Path
lines = [line.strip() for line in Path('data/raw.txt').read_text(encoding='utf-8').splitlines() if line.strip()]
lines = list(dict.fromkeys(lines))
assert len(lines) >= 5, 'write at least five different Q/A lines'
assert all(' | A:' in line and line.startswith('Q:') for line in lines)
train = [line for i, line in enumerate(lines) if i % 5 != 0]
val = [line for i, line in enumerate(lines) if i % 5 == 0]
for name, rows in [('train', train), ('val', val)]:
text = '\n'.join(row.replace(' | A:', '\nA:') for row in rows) + '\n'
Path('data/' + name + '.txt').write_text(text, encoding='utf-8')
print(name, len(rows))
Line-by-line walkthrough
Read the source you typed, piece by piece. The numbers on the left are the real line numbers in the file. Match them with the line number in as you read.
prepare.py Reading the raw file and removing duplicates
from pathlib import Pathlines = [line.strip() for line in Path('data/raw.txt').read_text(encoding='utf-8').splitlines() if line.strip()]lines = list(dict.fromkeys(lines))
(…).read_text(encoding='utf-8') reads the whole file as text, and splitlines() splits it into lines. The comprehension [line.strip() for line in … if line.strip()] removes leading and trailing spaces and drops blank lines. dict.fromkeys(lines) removes duplicates by putting the lines in as , and since Python dictionaries keep insertion order, the order of first appearance stays the same. With set, the order could differ on each run and change the training/validation split.
prepare.py Format check
assert len(lines) >= 5, 'write at least five different Q/A lines'assert all(' | A:' in line and line.startswith('Q:') for line in lines)
Two asserts check the data. There must be at least five different lines so at least one can be set aside for validation. all(…) is true only when every line starts with Q: and contains " | A:". If even one line is wrong, training formats get mixed, so it is better to stop here. The second assert has no message, so you only see AssertionError. In that case, look over the format of raw.txt.
prepare.py Splitting into training and validation
train = [line for i, line in enumerate(lines) if i % 5 != 0]val = [line for i, line in enumerate(lines) if i % 5 == 0]
enumerate(lines) gives (number, line) in turn. i % 5 is the remainder of the number divided by 5, so lines 0, 5, 10… go to val and the rest to train. 15 lines become 12 and 3. It splits by a rule rather than at random, so every run gives the same split.
prepare.py Saving in two-line format
for name, rows in [('train', train), ('val', val)]: text = '\n'.join(row.replace(' | A:', '\nA:') for row in rows) + '\n' Path('data/' + name + '.txt').write_text(text, encoding='utf-8') print(name, len(rows))
For train and val each, it replaces " | A:" in every line with a newline + "A:" to put the question and answer on two lines, joins the pairs with newlines, adds one newline at the end and saves. The last answer must also end with a newline so the model learns "a newline after the answer" from every answer. print shows how many pairs were written.
Check your understanding
Run and . You should see settings.json and prepare.py. The sample produces 12 training records and 3 validation records. Add your own Q/A line and explain which file must be rebuilt.
If you get stuck
usually means the wrong folder or filename. Run ~/xslm and ls data. means Python differs: use four spaces. Finished reference files are in ~/xslm-reference for comparison; they do not fill your project automatically.
Frequently asked questions
Can 15 lines really train anything?
It is enough to check that the process runs, and far too little for a useful conversation. With only 12 training pairs, the model quickly tries to memorize them as they are, and the validation loss actually goes up (step 6, "overfitting"). The more lines you add, the more room there is to learn rules rather than memorize.
Why leave validation data out of tokenizer training too?
The tokenizer is also trained. If character combinations that appear only in validation sentences got into the merge rules, validation sentences would turn into shorter, cleaner tokens than training sentences, making the test easier. To imitate a real service receiving text it has never seen, validation data must be left out of every training stage.
I get the assert error "write at least five different Q/A lines".
prepare.py needs at least five different lines to split them into training and validation. Blank lines and duplicate lines are not counted. If the format assert fails, check that every line starts with Q: and contains " | A:". Even the spacing must match.
