Now, let's build your very own AI.

What is XSLM — one step smaller than LLMs and SLMs

XSLM (Extra Small Language Model) is one step smaller than large language models (LLMs) and small language models (SLMs): it means a very small AI model with millions to a few hundred million parameters. It is a term coined by sonub.com.

• LLM: tens to hundreds of billions of parameters. Runs on GPU servers in data centers.
• SLM: around a few billion parameters. Runs on a powerful computer or a single GPU.
• XSLM: millions to a few hundred million parameters. Runs even on small devices such as smartphones.

XSLM has three key features.

• Optimized for on-device use: the model is so light that it runs on its own, without a network connection, even on devices with very limited computing power such as smartphones, wearables and IoT (Internet of Things) equipment.
• Very low cost: a typical SLM needs several gigabytes (GB) of VRAM just to load, but an XSLM runs in a few hundred megabytes (MB) of memory or less. So deploying and running it costs almost nothing.
• A specialist for one job (Specialized Worker): unlike an LLM that knows a lot like an encyclopedia, an XSLM is built to do just one job very well, such as summarizing sentences, classifying text, function calling or routing for an AI agent (Router).

The XSLM you build in this course is far smaller still: a learning model with 1,248 parameters. It is shrunk to the smallest size so you can follow the principles with your own eyes, but the parts are the same. Grow the same structure and train it on data for one job, and it becomes the practical XSLM described above.

What is a language model — a function that gives next-token probabilities

A language model is a function that looks at the text so far and gives a probability to each possible next piece (token). For example, after "the sun is", it might give 0.6 to warm, 0.2 to hot and 0.01 to blue, and share the remaining 0.19 among the other tokens in the vocabulary. The probabilities of the whole vocabulary add up to 1.

Writing text means repeating this function.

• Turn the text so far into token IDs.
• The model computes the probability of each next token.
• Pick one token according to those probabilities.
• Append the chosen token to the text and go back to the start.

Huge models like ChatGPT and Claude work on the same basis. What differs is their size (hundreds of billions of weights), the amount of training data, and extra training that makes them good at conversation. XSLM builds this basis from scratch at the smallest size, so you can see with your own eyes what each part does.

Building from scratch — starting from random numbers

Building from scratch means you do not download weights someone else trained. All 1,248 XSLM weights start as small random numbers drawn with random.gauss(0, 0.1). At this point the model knows nothing, so it gives almost the same probability to all 40 vocabulary tokens.

Training does only one thing. It nudges the numbers so that the next token that really appears in the data gets a higher probability. This next-token prediction alone builds grammar, format and common content into the numbers. You do not need to add separate answer labels. The text itself is both the question and the answer.

So the data format, more than the algorithm, decides what the model becomes good at. Every line of XSLM data has the form Q: question followed by A: answer, so the model learns the habit of continuing after A: with something that looks like an answer. No code adds a separate ability to chat.

The whole flow — programs hand files to each other

XSLM is a chain: each program reads the file made by the previous program and makes the next file.

• prepare.py — reads data/raw.txt and makes data/train.txt and data/val.txt.
• tokenizer.py — learns merge rules from train.txt and makes data/tokenizer.json.
• engine.py and model.py — the math tools and the GPT structure. They make no files on their own; other programs import them.
• train.py — trains the GPT with the data and the tokenizer and makes checkpoints/model.json.
• generate.py — reads model.json and continues text.
• runner.py — opens generate as an HTTP API.
• xchat.py — sends questions to the Runner and shows the answers.

When an earlier file changes, everything after it must be rebuilt. For example, if you edit raw.txt, run prepare → tokenizer → train again in that order.

How small is it — a size comparison

Model size is counted as the number of learned numbers (parameters).

• XSLM: 1,248 — vocabulary 40, vector width 8, 1 block, 1 head.
• A small practice GPT (example): about 28 million — vocabulary 24,000, width 384, 6 blocks, 6 heads.
• GPT-2 Small (2019): about 120 million.
• GPT-3 (2020): about 175 billion.

There is a reason XSLM is this small. Every number is handled as one Python object (Value), and the math runs on a virtual CPU inside the browser. A bigger size would make one training run take minutes. In return, you can read the calculation of every single number. The structure is the same as GPT; only the size and the way of computing differ.

XSLMs in real use — public models and shrinking techniques

Models of this size are already published and in use.

• SmolLM2-135M (Hugging Face, 2024): about 135 million parameters. A GPT-style model that continues text, and it runs even on smartphones.
• MobileBERT (Google, 2020): about 25 million parameters. Rather than writing text, it understands sentences to classify them or to find the answer to a question inside a passage.

There are two main techniques for shrinking a big model down to XSLM size.

• Knowledge distillation: a small model (the student) learns to follow the probabilities produced by a big model (the teacher). It learns more than it would from a single correct answer alone.
• Quantization: each parameter is stored as an 8-bit or 4-bit number instead of a 32-bit decimal. Memory shrinks 4 to 8 times, and accuracy drops a little.

Both techniques build on the training loop (step 6) and saved weights (checkpoints) you make in this course. When you finish the course, you are ready to read the documents and code for these techniques.

Nine key concepts — one line each

Here is a one-line preview of each concept you will meet. Each step covers them in detail.

• Data format: decides what the model will do (step 3).
• Tokenizer: turns text into token IDs and back (step 4).
• Embedding: a table that turns token IDs into number vectors (step 5).
• Position embedding: a vector that tells which position a token is in (step 5).
• Attention: decides which earlier positions to look at and how much, and mixes their information (step 5).
• MLP (FFN): reprocesses the mixed information at each position (step 5).
• Residual connection and normalization: keep the original information and stabilize the size of numbers (step 5).
• Next-token prediction and loss: measure how well the model did by the probability of the correct token (step 6).
• Sampling: picks an actual token from the probabilities (step 7).

Four common misconceptions

Let’s correct the misconceptions people often have when building their first model.

• Sentences are stored inside the model? No. Knowledge is spread across number patterns in all the weights. Open any single number and you will find no sentence.
• It looks up the next word in a database? No. The model calculates a score for the whole vocabulary every time. It does no searching.
• Low loss means it tells the truth? No. It only means it is likely to produce text similar to the training data.
• Small models work on different principles? No. XSLM and huge models use the same parts. Only the size, data and extra training differ.

One more. What the model learned is frozen the moment training ends. To know new information, it must be trained again, or, as real services do, look up the latest documents when asked and include them (RAG, step 9).