Now, let's build your very own AI.

Only the Python standard library — modules used and what they do

XSLM installs nothing with pip. It uses only the standard library that comes with Python.

• json: saves and reads settings, the tokenizer and checkpoints as text files.
• math: math functions such as exp, log and sqrt.
• random: random numbers for initial weights, picking training windows and sampling tokens. The same seed gives the same result.
• collections.Counter: counts adjacent pairs in BPE.
• pathlib: reads and writes files.
• argparse: accepts command-line options such as --steps.
• hashlib: makes a fingerprint (SHA-256) of the training data to detect whether it changed.
• os: swaps in a checkpoint safely (os.replace).
• http.server and urllib: the Runner server and the XChat client.

With no installation, there are no version conflicts or failed installs. Instead, you must write yourself what matrix libraries (numpy, PyTorch) would do. That is exactly the purpose of this course.

Linux inside the browser — what runs where

The terminal on the right is not connected to a server. Real Linux runs inside this browser.

• v86: an emulator that imitates an x86 CPU. It is built with WebAssembly and runs in the browser.
• Alpine Linux 32bit: a small Linux that runs on that virtual CPU. It has 512MB of memory.
• bash, nano, micro, python3, curl: programs preinstalled in this Linux.

Your computer’s CPU runs the commands through the emulator. No code is sent to a server, so anything you run is safe. But because it is emulated, the same Python code runs much slower than directly on your computer. That is why XSLM’s default size and number of training steps are small.

Files in ~/xslm are saved in this browser and survive a reload. Running programs (such as the Runner) stop when you reload, so start them again.

What tools are used to build bigger models

XSLM uses only the Python standard library. Bigger models are usually built with the tools below.

• PyTorch: does number-array (tensor) math and automatic differentiation fast on the GPU. It does the job of engine.py in XSLM.
• tokenizers: a fast BPE tokenizer library. It is written in Rust and is about 100 times faster than pure Python. It does the job of tokenizer.py in XSLM.
• numpy and tqdm: number arrays and a progress bar.
• uv: manages Python projects and packages. You install with commands like uv add torch.
• Device: where the computation runs. mps for an Apple GPU, cuda for an NVIDIA GPU, otherwise cpu.

The tools differ, but the principles are the same as in XSLM. The tools only do the same calculations faster.

Why is it slow — the cost of computing one number at a time

All XSLM math is additions and multiplications of individual Value objects. Count what happens when it trains once on a 12-token window.

• Each position has seven matrix multiplications — query, key, value, project, expand, contract and head — and one matrix multiplication is tens to hundreds of multiplications and additions.
• Every operation creates one Value object that is recorded in the computation graph. With the default settings, one training step creates about 32,000 of them.
• backward walks through all these objects once more in reverse.

PyTorch groups the same math into whole matrices and processes it at once in C++ or on the GPU, so it is hundreds to thousands of times faster. We chose the slow road on purpose to see the principles. If training takes long, you can stop it with Ctrl+C; everything up to the last completed step stays saved.

We timed the default 20 training steps (python3 train.py --steps 20) in this browser Linux: about 23 seconds on a fast desktop (Apple M5 Max), about 2 min 13 s with the CPU slowed 4×, and about 4 min 5 s slowed 6×. Generation (generate.py) took about 3, 17 and 33 seconds. On a slow computer, run it in parts such as --steps 5. Training continues from the checkpoint, so the result is the same as doing it all at once.

Python and terminal basics for this course

Here is just what you need so you don’t get stuck while typing the sources.

• micro filename.py: opens a Python file to edit. Ctrl+S saves; Ctrl+Q closes. Enter carries the indentation to the next line, so press Backspace if a line goes deeper than the source.
• nano filename: opens any other file, such as settings.json or raw.txt. Ctrl+O then Enter saves; Ctrl+X closes.
• python3 filename.py: runs a program. Typing only python3 opens interactive Python (REPL) for trying one line at a time; exit() leaves it.
• Indentation: Python separates blocks with spaces. This course uses 4 spaces per level. Mixing Tab and spaces causes IndentationError.
• import: from model import GPT takes GPT from model.py in the same folder. That is why every command runs in ~/xslm.
• if __name__ == '__main__': the code under it runs only when you run the file directly. It does not run when another file imports it.
• assert condition, message: stops with that message if the condition is false. XSLM uses assert to catch wrong data or settings early.

When an error occurs, read the Traceback from the bottom line up. The last line is the error type and message, and the File … line number above it is where the problem happened.

If you want to build bigger on your own computer

If you want to train a bigger model after finishing XSLM, set up a PyTorch environment on your own computer (optional).

• Install Python 3.10 or later.
• Install uv: curl -LsSf https://astral.sh/uv/install.sh | sh (on macOS, brew install uv also works).
• Create a project with uv init my-gpt, then cd my-gpt and install with uv add torch tokenizers tqdm numpy.
• On an Apple Silicon Mac, check that uv run python -c "import torch; print(torch.backends.mps.is_available())" prints True. For an NVIDIA GPU, check torch.cuda.is_available().

Apple Silicon trains with MPS, NVIDIA GPUs on Linux or Windows with CUDA, and anything else with the CPU. You can train on the CPU too, but it is slow. The names you learned in XSLM (embedding, attention, MLP, loss, optimizer) appear unchanged in PyTorch code.