Create your own chat and complete the entire chain. Write xchat.py and ask a question that travels through your to the you trained.
9
XChat
is the interface, is the execution service, and is the learned . xchat.py reads a question, adds the format, sends an request and displays the text. It does not import the model or generate an answer itself.
This first runs inside the . Each question is independent: it does not yet send . A useful extension is to add history, while respecting the model’s small . Fluency is not the graduation requirement; an actual request through your own trained model is.
The path a request travels — from XChat to the weights
When you type hi in XChat and press Enter, these things happen in order.
xchat.py wraps the question in the training format: "Q: hi" followed by "A: " on the next line.
It makes the {"": …, "": 24} and POSTs it to http://127.0.0.1:8765/generate.
The Runner’s do_POST reads and checks the body, then calls generate.
generate makes with the , picks tokens one by one with the in , and stops at a .
The Runner responds with {"text": …}, and xchat.py shows it after XSLM>.
If any one point on this breaks, the chat does not work. So when there is a problem, check along the path one by one: is the Runner running ( ), does generation work (curl /generate), are XChat’s address and right.
The XChat client — input loop and HTTP request
xchat.py has two parts.
ask(question): sends one request and returns the answer. It builds a request holding the address, body and headers with Request, sends it with , reads the response body with json.load and takes out text. timeout=120 means it gives up if there is no answer within 2 minutes. It is generous so it can wait for long generations in the slow .
Input : with while True, it keeps showing You> and reads one line at a time. /exit ends it, blank lines are skipped, and anything else goes to ask, with the answer printed after XSLM>.
It rereads settings.json for every question. The client adds the prompt format ("Q: " … "A: "). If you make the add the format, the client does not need to know the format. Either way, the format must be defined in only one place so it never gets out of sync.
Handling errors — never making up a fake answer
xchat.py handles errors in these groups.
EOFError and KeyboardInterrupt: you pressed (end of input) or . It ends the loop and returns to .
URLError: it could not connect to the Runner ( and the like). The Runner is off or the port is different.
HTTPError: it connected, but the Runner answered with an error like 400. It is a kind of URLError, so it is caught on the same line.
TimeoutError: no answer came within the set time.
In every case, it shows "Runner: error details" and waits for the next question. It does not hide a failed connection by showing a prepared sentence instead. That way you know right away what broke. Stop the Runner and ask a question (an error is the correct result), then start it again and check that the chat comes back.
Adding chat history — the limit of context length
Right now, each XChat question is independent. The model does not know what was said before. Real seem to remember earlier conversation because they attach the whole past conversation in front of the prompt every time. The model itself has no memory.
Try it with XSLM and you soon hit a wall. The experiment below attaches the new question cat after one exchange (Q: hi / A: hi friend.). The prompt is 21 tokens, but the model sees only the last 12, so the text it actually sees starts from "iend.". The earlier conversation is cut off.
So adding chat history requires doing these things too.
Drop the oldest conversation first to fit within the context length.
Train a new model with a larger context (the position table grows and computation slows down).
Put conversations with several turns in the too, so the model learns that shape.
Next challenge: keep a history in xchat.py and send past questions and answers along with the new one. Count the tokens and drop the front part when it goes over 12.
Check it yourself lab/context.py
import json
from pathlib import Path
from tokenizer import BPE
tok = BPE(**json.loads(Path('data/tokenizer.json').read_text()))
size = json.loads(Path('settings.json').read_text())['context']
history = 'Q: hi\nA: hi friend.\nQ: cat\nA: '
ids = tok.encode(history)
print(len(ids), 'tokens; the model sees the last', size)
print(repr(tok.decode(ids[-size:])))
cd ~/xslm
mkdir -p lab
micro lab/context.py
python3 -m lab.context
Output
21 tokens; the model sees the last 12
'iend.\nQ: cat\nA: '
How real chat models are made — the stages after pretraining
XSLM was trained from scratch on Q/A-format text. Chat models like and usually go through several stages.
: on a huge amount of text. It is the same thing XSLM’s train.py does, only at a different scale.
: further training on pairs of questions and good answers to build the habit of following instructions. This is also when it learns the chat format () that separates the user’s and assistant’s turns.
Preference training ( and others): refines it to lean toward answers people picked as better.
When up-to-date information is needed, is used instead of retraining. Documents are found by searching with the question and put into the prompt, and the model answers while looking at those documents. If the format has a document field such as [DOC] and the model learns ahead of time the shape of answering with reference to a document, search results can later be put into that field.
Next challenges — ways to grow XSLM
Here are experiments you can try with your finished XSLM. Change one thing at a time and record the and the answers.
More data: write dozens to hundreds of lines with questions in many wordings and answers in the same format.
More training: change the and number of steps and compare training and .
A bigger model: change width to 16 and context to 24. Measure how the number of and the time per step change.
Stacking blocks: wrap the + in model.py into a , make separate for each block, and pass through it twice.
Splitting heads: split the width of 8 into two heads of 4 slots each, run attention separately, then join them.
Adding : give generate an option that cuts candidates by .
and chat history: add them to the Runner and XChat.
Once you can do all this, try rewriting the same structure with on your own computer. Computing with lets you train tens of millions of parameters thousands of times faster. ’s and build-nanogpt are also good next materials.
Type and try
Leave the Runner running. Type xchat.py and launch it. At You>, type hi and press Enter. Type /exit to return to bash. Only then run the final to stop the Runner. Export your complete project.
cd ~/xslm
micro xchat.py
python3 xchat.py
hi
/exit
kill "$(cat runner.pid)"
input reads a line; Request creates the JSON POST; urlopen reads the real response. Connection errors are reported instead of replaced with a fake answer.
import json
from pathlib import Path
from urllib.request import Request, urlopen
from urllib.error import URLError
def ask(question):
cfg = json.loads(Path('settings.json').read_text())
body = {'prompt': 'Q: ' + question + '\nA: ', 'tokens': cfg['tokens']}
request = Request('http://127.0.0.1:' + str(cfg['port']) + '/generate',
data=json.dumps(body).encode(),
headers={'Content-Type': 'application/json'})
with urlopen(request, timeout=120) as response:
return json.load(response)['text']
if __name__ == '__main__':
print('XChat -> Runner -> XSLM | /exit')
while True:
try:
question = input('You> ').strip()
if question == '/exit':
break
if question:
print('XSLM>', ask(question))
except (EOFError, KeyboardInterrupt):
break
except (URLError, TimeoutError) as error:
print('Runner:', error)
For each question, it reads settings.json to get the token count and port. It wraps the question as "Q: " + question + newline + "A: " to match the training format. Giving data to Request makes it a POST, with JSON as the body and as the header. with urlopen(…) closes the connection after reading the whole response. json.load reads the response as a , and ask returns only text.
xchat.py The chat loop
if __name__ == '__main__': print('XChat -> Runner -> XSLM | /exit') while True: try: question = input('You> ').strip() if question == '/exit': break if question: print('XSLM>', ask(question)) except (EOFError, KeyboardInterrupt): break except (URLError, TimeoutError) as error: print('Runner:', error)
It prints a start message and takes questions with while True. input shows You> and reads one line until Enter, and strip removes leading and trailing spaces. /exit ends it with break, and empty questions are skipped. Ctrl+D comes in as EOFError and Ctrl+C as KeyboardInterrupt, and both also end it. Runner connection errors and timeouts are reported, and it waits for the next question.
Check your understanding
Show the chain XChat → HTTP Runner → loaded weights → generated text. Stop the Runner and try another chat: it must report a connection error. Restart it and verify chat works again. Explain each file to someone else.
If you get stuck
Do not type the kill command inside XChat; it would be sent as a question. Exit first. A successful response with poor text still demonstrates the pipeline; improve data and training as your next experiment.
Frequently asked questions
I typed kill in XChat, but the Runner did not stop.
Text typed at You> is sent to the Runner as a question, not as a command. First end XChat with /exit or Ctrl+D to return to the bash prompt ($), then type the kill command.
I get Runner: <urlopen error [Errno 111] Connection refused>.
The Runner is off or still starting. Check the startup line (XSLM Runner: …) with cat runner.log, see whether curl -s http://127.0.0.1:8765/health answers, then ask again. If you reloaded the page, you must start the Runner again.
The answer has nothing to do with the question. Is the connection wrong?
If any generated text appears after XSLM>, the connection is right. The content reflects the model’s skill. Run the same question directly with generate.py --prompt. With the same settings and seed, it gives exactly the same answer as through the Runner.
