Sign up to keep track of your learning progress.
Log inNow, let's build your very own AI.
A toy word-bigram : it looks only at the previous word. It does not learn embeddings, attention, or facts. The word splitter is a teaching example, not an tokenizer.
A Plan That Leads to Real Development
Start small like a toy, understand the principles, then grow it bigger over time.
Gather words
Collect sentences about the topic you want to learn. Just 10 sentences is enough to start.
Count next words
Make a table of how many times each word follows another word.
Compute probability scores
Count each transition, including sentence endings. A candidate's probability is its count divided by the total count after the selected word. Neural language models learn weights instead of just storing this small table.
Compare two selection methods
Choose the most likely candidate (greedy), or draw according to probabilities (sampling). Sentence-end stops generation. Sampling can produce the same result again. Generated text is not checked for truth.
Improve it your way
Add better sentences, more data, and different selection rules to grow it into your own AI.
A Tiny Probability Calculator
The table shows observed frequencies at temperature 1. Temperature changes sampling probabilities, not the learned counts or factual accuracy.
There are no candidates yet following the last word of the starting sentence.
| Previous wordNext word | |
|---|---|
Many words — showing only the first 8
Key Concept
Count each transition, including sentence endings. A candidate's probability is its count divided by the total count after the selected word. Neural language models learn instead of just storing this small table.
A neural language learns numerical from examples. At generation time, those weights and the available context determine scores for candidate tokens. The model does not simply look up a stored internet sentence or count only the word immediately before it.
This lab vs a real language model
| This lab (bigram) | Neural language model | |
|---|---|---|
| What it looks at | Only the previous word | The long context it is given |
| What it learns | A table of counts | Numeric weights |
| Embeddings and attention | No | Yes |
| Checks facts by itself | No | No |