Turn next-token scores into generated text. Write generate.py, then change , and seed one at a time and compare the results.
7
Text generation
Generation repeatedly feeds the latest to and appends one predicted . 0 picks the highest score. Positive temperatures rescale scores before random : higher values spread more broadly. keeps only the strongest candidates; 0 keeps all.
A seed repeats the random choices. It does not make answers correct. Generation stops at a or the requested token limit. Only the newly generated text is returned. Keep the Q:/A: format used during .
The generation loop — appending one token at a time (autoregressive)
generate repeats this sequence for the set number of tokens.
Take only the last context (12) tokens so far and feed them to forward.
From the returned score , use only the scores of the last position, [-1]. Those are the next-token scores.
Pick one token by the chosen method (, temperature, top_k).
Append the chosen token to ids. The next turn sees the text including this token.
Stop if the chosen token is a newline.
Because it uses its own output as input again, this is called generation. A chosen token is never taken back, so a mistake early on is followed by what comes after.
Generation only reads the . But ’s forward does the same Value calculations as in training, so it even builds a that will never be used. generates inside torch.no_grad() so no graph is built, and turns off with model.eval().
Greedy choice — temperature 0
With temperature 0, it always picks the token with the highest score (greedy, ). The same input always gives the same answer.
The advantage is predictability. It shows exactly what the finds most likely, which is good for checking whether training worked. The disadvantage is that it easily repeats the same phrase. Once it enters a loop like "is warn is warn is warn", the same token is highest every time and it cannot get out. That is why a little randomness is mixed in for conversation.
Temperature — making the distribution sharper or flatter
temperature (T) divides the scores by T before . Probability is proportional to exp(score ÷ T).
T < 1: score gaps grow and the distribution gets sharper. High tokens are picked more often.
T = 1: the probabilities exactly as the model gave them.
T > 1: score gaps shrink and the distribution gets flatter. Low tokens are also picked often, and the output gets strange.
The experiment below changes the temperature for the scores warm 2.0, hot 1.0, blue 0.2 and run −1.0. The probability of warm changes from 0.858 at T 0.5 to 0.632 at 1.0 and 0.447 at 2.0, while run rises from 0.002 to 0.1.
generate.py does the same calculation. It subtracts the largest value from the candidate scores (numerical stability), divides by T, applies exp, and passes the results as weights to random.choices. The weights need not add up to 1; choices picks in proportion. 0.7–0.9 is the most common choice; 0.1–0.5 is conservative, and 1.0 or more tends to get strange.
Check it yourself lab/temperature.py
import math
logits = {'warm': 2.0, 'hot': 1.0, 'blue': 0.2, 'run': -1.0}
for t in [0.5, 1.0, 2.0]:
weights = {w: math.exp(s / t) for w, s in logits.items()}
total = sum(weights.values())
print('T =', t, {w: round(v / total, 3) for w, v in weights.items()})
print('top_k=2 keeps', sorted(logits, key=logits.get, reverse=True)[:2])
cd ~/xslm
mkdir -p lab
micro lab/temperature.py
python3 -m lab.temperature
Even with temperature, many very low tokens can together hold a fairly large probability. With a 50,000-token , each one is tiny, but all of them added up cannot be ignored. So the candidates are narrowed before sampling.
top-k: keep only the top k scores and drop the rest. XSLM’s top_k 8 keeps only 8 of the 40 vocabulary tokens as candidates. 0 means no narrowing. top_k=2 in the experiment keeps only warm and hot.
top-p (nucleus): add probabilities from highest to lowest and keep only until the sum reaches p (for example 0.9). When the model is confident there are a few candidates, and when it is unsure there are many, so it adapts to the situation. XSLM does not have it, but real services use it a lot.
Models with vocabularies of tens of thousands often use top_k 40–60. (temperature 0.7, top_k 30) is a combination for accuracy, (0.9, 50) for balance and (1.1, 100) for creativity. For XSLM with a 40-token vocabulary, about 8 is a similar ratio.
seed — reproducing random choices
When temperature is above 0, random.choices picks tokens at random. generate makes a new generator with random.Random(seed) every time it is called, so the same checkpoint, , temperature, top_k and seed always give the same text.
The experiment below picks a, b and c 12 times with weights 5:3:2. Seed 7 gives the same row both times, and seed 8 gives a different row. a, with the biggest weight, appears most often, but decides the order.
A seed is a number for reproducing results, not a number that guarantees good answers. When comparing settings, fix the seed and change only one thing; to see whether a result is luck, try several different seeds.
Check it yourself lab/seed.py
import random
for seed in [7, 7, 8]:
rng = random.Random(seed)
print(seed, ''.join(rng.choices('abc', weights=[5, 3, 2])[0] for _ in range(12)))
cd ~/xslm
mkdir -p lab
micro lab/seed.py
python3 -m lab.seed
Output
7 aababaabaaaa
7 aababaabaaaa
8 acabaacabaaa
Stop conditions — newline and maximum tokens
Generation stops in two cases.
When a newline token is picked: in the , each answer line ended with a newline, so the model has judged that "the answer is finished". The newline is not included in the result.
When the requested number of tokens (default 24, maximum 64) is reached: this keeps it from writing forever even if the model cannot finish.
The result leaves out the prompt and returns only the newly made tokens, decoded. If a newline is picked as the very first token, the answer is empty.
Real models stop at an special token or a chosen stop marker (for example [/ANSWER]) and cut off what follows. , which sends each token to the screen as soon as it is made, is also common.
Prompt format — exactly like the training shape, spaces included
A model only continues well after shapes it saw in the training data. So the text you give it when generating (the prompt) must match the training format exactly. XSLM’s default prompt is "Q: hi" followed by "A: " on the next line.
Even one space matters. "A: " in the training data is one token ( 30), but if you drop the trailing space and give "A:", it becomes two tokens, "A" (ID 5) and ":" (ID 4). These two tokens never appeared like this in the training data, so the model starts its answer in a situation it has never seen. When we asked about sun to a model trained 3,000 times at 0.1 on a desktop, ending with "A: " gave text similar to a training sentence, like "te sun is warn is warn…", while ending with "A:" gave only fragments like "ore".
With a long format, include the beginning of the answer too. For example, if you add "Summary:" and "-" after [ANSWER], the model goes straight on to write the summary list.
When the answers are strange — causes and experiments
Here are the causes of strange answers from a small model, most common first.
Not enough training: the default 20 steps are only for checking. If the is near 3.7, the model still knows almost nothing.
Not enough : 12 pairs only let it imitate the format; it cannot answer questions it has never seen.
Format mismatch: the prompt differs from the training format, or unknown characters (�) are mixed in.
Not enough context: it cannot see anything beyond the 12-token context.
Sampling settings: if it repeats, raise temperature and top_k a little; if it is off the wall, lower them.
Adding randomness does not fix insufficient training. Here is an experiment worth trying. With a new checkpoint, change the learning rate to 0.1, train in chunks of a few hundred --steps, and ask the same question at temperature 0 each time, recording how the answer changes. The "te sun is warn…" above is a stage where it has started imitating the training sentence "the sun is warm." but has not yet even memorized it.
Type and try
Write generate.py. Compare the three while keeping the same checkpoint. Change only temperature first, then top_k and seed. The checkpoint supplies its own and model dimensions.
load rebuilds the architecture and restores numbers. generate crops context, scores candidates and samples repeatedly. A fixed seed makes comparisons .
import argparse
import json
import math
import random
from pathlib import Path
from model import GPT
from tokenizer import BPE
def load():
saved = json.loads(Path('checkpoints/model.json').read_text())
cfg = saved['settings']
tokenizer = BPE(**saved['tokenizer'])
model = GPT(len(tokenizer.vocab), cfg['width'], cfg['context'], cfg['seed'])
model.load(saved['weights'])
return model, tokenizer, cfg
def generate(model, tokenizer, prompt, tokens=24, temperature=0.8, top_k=8, seed=7):
if not prompt or tokens < 1 or temperature < 0 or not math.isfinite(temperature) or top_k < 0:
raise ValueError('invalid sampling parameters')
ids, result, rng = tokenizer.encode(prompt), [], random.Random(seed)
for _ in range(tokens):
logits = model.forward(ids[-model.context:])[-1]
ranked = sorted(range(len(logits)), key=lambda i: logits[i].data, reverse=True)
if temperature == 0:
token = ranked[0]
else:
candidates = ranked[:top_k] if top_k else ranked
peak = logits[candidates[0]].data
weights = [math.exp((logits[i].data - peak) / temperature) for i in candidates]
token = rng.choices(candidates, weights=weights)[0]
ids.append(token)
if tokenizer.vocab[token] == '\n':
break
result.append(token)
return tokenizer.decode(result)
if __name__ == '__main__':
model, tokenizer, cfg = load()
parser = argparse.ArgumentParser()
parser.add_argument('--prompt', default='Q: hi\nA: ')
parser.add_argument('--tokens', type=int, default=cfg['tokens'])
parser.add_argument('--temperature', type=float, default=0.8)
parser.add_argument('--top-k', type=int, default=8)
parser.add_argument('--seed', type=int, default=cfg['seed'])
args = parser.parse_args()
assert 1 <= args.tokens <= cfg['max_tokens']
print(generate(model, tokenizer, args.prompt, args.tokens,
args.temperature, args.top_k, args.seed))
Line-by-line walkthrough
generate.py load — bringing the model back from a checkpoint
It reads even the settings and tokenizer from the single model.json, so it builds the same model as during training even if settings.json or tokenizer.json changes later. It builds a GPT of the same size (its initial values are overwritten right away) and puts in the learned numbers with load.
generate.py Input checks and preparation
def generate(model, tokenizer, prompt, tokens=24, temperature=0.8, top_k=8, seed=7): if not prompt or tokens < 1 or temperature < 0 or not math.isfinite(temperature) or top_k < 0: raise ValueError('invalid sampling parameters') ids, result, rng = tokenizer.encode(prompt), [], random.Random(seed)
An empty prompt, a token count below 1, a negative, infinite or temperature, and a negative top_k are rejected with . math.isfinite is false for inf and nan. It turns the prompt into IDs and makes result to collect the newly made tokens and rng, a random number generator just for this generation.
generate.py Getting the next-token scores
for _ in range(tokens): logits = model.forward(ids[-model.context:])[-1] ranked = sorted(range(len(logits)), key=lambda i: logits[i].data, reverse=True)
It feeds only the last 12 tokens with ids[-model.context:] and uses only the last row [-1] of the forward result. ranked is the list of vocabulary IDs sorted from highest score to lowest, and ranked[0] is the highest token.
generate.py Picking one token
if temperature == 0: token = ranked[0] else: candidates = ranked[:top_k] if top_k else ranked peak = logits[candidates[0]].data weights = [math.exp((logits[i].data - peak) / temperature) for i in candidates] token = rng.choices(candidates, weights=weights)[0]
If temperature is 0, it picks the highest token. Otherwise it keeps top_k candidates (all of them if 0), subtracts the largest score peak, divides by temperature, applies exp, and uses the results as weights to pick one with rng.choices. choices returns a list, so it takes the item with [0].
generate.py Append, stop, return
ids.append(token) if tokenizer.vocab[token] == '\n': break result.append(token) return tokenizer.decode(result)
It appends the chosen token to ids so it becomes part of the next turn’s input. If that token is a newline, it stops without adding it to the result; otherwise it collects it in result. When the ends, it decodes and returns only result, so the prompt is not part of the result.
Run directly, it reads the checkpoint and takes the options --prompt, --tokens, --temperature, --top-k and --seed. argparse turns --top-k into args.top_k (- becomes _). After asserting that the token count is 1–max_tokens (64), it prints the generated text. To give a prompt with a newline in the , use , as in --prompt "$(printf 'Q: cat\nA: ')".
Check your understanding
The same checkpoint, prompt, seed and should the same text. Explain why loading weights differs from training them and why an empty answer can occur when newline is selected first.
If you get stuck
Strange or empty output is expected from a tiny, briefly trained model. Check data format, unknown characters and training loss. More sampling randomness is not a fix for insufficient training.
Frequently asked questions
Why do I get the same answer to the same question every time?
Because the seed is the same. generate makes a new generator with the same seed for every request. To see a different answer, change --seed or put a different integer as "seed" in the request . With temperature 0, it is always the same regardless of the seed.
Does a higher temperature make it smarter?
No. Temperature only changes how it picks from already calculated probabilities. If the model gives the right answer a low probability, no temperature will make the right answer appear often. Smartness is decided by training (data, size, number of steps).
I get an empty answer.
A newline was picked as the first token, so it stopped right away. In the training data, a line never breaks right after "A: ", so this happens when training is insufficient or the prompt format differs. Check that the prompt ends with "A: ", then train more or change the seed or temperature.
