Serve your trained through a real . Write runner.py, start it in the background and call and /generate with .
The loads the checkpoint once, accepts and calls generate. handles requests sequentially. checks the process; /generate returns generated text. This is a teaching , not a public service.
is the ’s loopback address. Run here in the ; your address bar points to a different computer. and & leave the server in the background while this tab runs. Reloading or closing the tab ends it. This environment does not require .
Running generate.py as a reads the checkpoint and rebuilds the every time. With many questions, the same preparation repeats each time. The Runner loads the model only once, keeps it in , and just calls generate whenever a request comes in.
Splitting into a server also separates roles.
As long as the agreement is kept, the client can be anything. Instead of the terminal XChat, a web page, a phone app or another server can call the same Runner. A like this, which loads trained and does the computation, is also called an . The Runner is a very small runtime; in practice, runtimes such as , and do the same job fast.
is an agreement in which a client sends a request and a server returns a response.
You can build requests yourself with curl.
The Runner opens only two addresses.
These are the fields of the /generate body.
For a bad request, the server does not crash; it answers 400 and {"error": "invalid request"}. A body over 4,096 , a body that is not JSON, a field of the wrong type or a missing prompt all give 400. The limits are there to protect the server. A single request with tokens set to 1 million could tie up the server for hours. A public server must have checks like these.
The Runner waits for requests on 8765 at 127.0.0.1.
Here, "this computer" is the virtual inside the browser. So always run curl and XChat inside the terminal on the right. Typing localhost:8765 in your browser’s address bar points to your own computer, not the virtual Linux, so it will not connect.
Two programs cannot use the same port at the same time. If the Runner is already running and you start another one, you get OSError: .
The Runner is a program that never ends and keeps waiting for requests. Run it normally and it takes over the terminal so you cannot type other commands, so run it in the background.
A real server must restart automatically when the computer reboots or the program dies. Linux does this with a systemd service (Restart=always) and with launchd. The Runner in this virtual Linux runs only while this tab is open.
XSLM’s HTTPServer handles one request at a time. A request that arrives before a generation finishes waits until the earlier request is done. For a class, this is simple and safe.
Servers used by many people do it differently.
FastAPI automatically provides request format checks (pydantic) and API documentation (Swagger) at /docs. XSLM writes the same checks by hand in runner.py.
To grow the Runner into a real service, add things like these.
Write runner.py and start it with the command shown. Save its process ID immediately. Check runner.log, then request /health and /generate. If the first health check is early, wait for the startup line and try again.
cd ~/xslm
micro runner.py
nohup python3 -u runner.py > runner.log 2>&1 &
echo $! > runner.pid
cat runner.log
curl -s http://127.0.0.1:8765/health
curl -s http://127.0.0.1:8765/generate -H 'Content-Type: application/json' -d '{"prompt":"Q: hi\nA: ","tokens":8,"temperature":0.8}'One model instance is shared by requests. Validate the body, generate, then return JSON. Binding to the guest loopback keeps this exercise to your Linux .
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from generate import load, generate
def main():
model, tokenizer, cfg = load()
class Handler(BaseHTTPRequestHandler):
def reply(self, status, body):
data = json.dumps(body, ensure_ascii=False).encode('utf-8')
self.send_response(status)
self.send_header('Content-Type', 'application/json; charset=utf-8')
self.send_header('Content-Length', str(len(data)))
self.end_headers()
self.wfile.write(data)
def do_GET(self):
if self.path == '/health':
self.reply(200, {'status': 'ok', 'model': 'XSLM'})
else:
self.reply(404, {'error': 'not found'})
def do_POST(self):
if self.path != '/generate':
return self.reply(404, {'error': 'not found'})
try:
length = int(self.headers.get('Content-Length', '0'))
if not 0 < length <= 4096:
raise ValueError('request size')
data = json.loads(self.rfile.read(length))
prompt = data['prompt']
tokens = data.get('tokens', cfg['tokens'])
temperature = data.get('temperature', 0.8)
top_k = data.get('top_k', 8)
seed = data.get('seed', cfg['seed'])
if not isinstance(prompt, str) or not 0 < len(prompt) <= 512:
raise ValueError('prompt')
if type(tokens) is not int or not 1 <= tokens <= cfg['max_tokens']:
raise ValueError('tokens')
if type(top_k) is not int or top_k < 0 or type(seed) is not int:
raise ValueError('top_k/seed')
if type(temperature) not in (int, float):
raise ValueError('temperature')
result = generate(model, tokenizer, prompt, tokens, temperature, top_k, seed)
self.reply(200, {'text': result, 'model': 'XSLM'})
except (ValueError, KeyError, TypeError):
self.reply(400, {'error': 'invalid request'})
address = ('127.0.0.1', cfg['port'])
print('XSLM Runner:', address, flush=True)
HTTPServer(address, Handler).serve_forever()
if __name__ == '__main__':
main()
def main():
model, tokenizer, cfg = load()When main starts, load() reads the model, and settings once. The below is defined inside main and uses the outer model, tokenizer and cfg directly (a closure). It does not rebuild the model for each request.
class Handler(BaseHTTPRequestHandler):
def reply(self, status, body):
data = json.dumps(body, ensure_ascii=False).encode('utf-8')
self.send_response(status)
self.send_header('Content-Type', 'application/json; charset=utf-8')
self.send_header('Content-Length', str(len(data)))
self.end_headers()
self.wfile.write(data)Handler inherits from BaseHTTPRequestHandler and handles one request. reply turns a into JSON and then into UTF-8 bytes (with ensure_ascii=False, Korean and other non-ASCII text stays as it is), and sends it in the order status code → headers (Content-Type, Content-Length) → blank line (end_headers) → body. Content-Length is the number of bytes, not characters, so it uses the length of the encoded .
def do_GET(self):
if self.path == '/health':
self.reply(200, {'status': 'ok', 'model': 'XSLM'})
else:
self.reply(404, {'error': 'not found'})When a GET request comes in, do_GET is called. If self.path is /health, it sends 200 and the status; otherwise 404. The health-check address answers right away without any model computation, so it is used as a light check that the server is alive.
def do_POST(self):
if self.path != '/generate':
return self.reply(404, {'error': 'not found'})
try:
length = int(self.headers.get('Content-Length', '0'))
if not 0 < length <= 4096:
raise ValueError('request size')
data = json.loads(self.rfile.read(length))
prompt = data['prompt']
tokens = data.get('tokens', cfg['tokens'])
temperature = data.get('temperature', 0.8)
top_k = data.get('top_k', 8)
seed = data.get('seed', cfg['seed'])
if not isinstance(prompt, str) or not 0 < len(prompt) <= 512:
raise ValueError('prompt')
if type(tokens) is not int or not 1 <= tokens <= cfg['max_tokens']:
raise ValueError('tokens')
if type(top_k) is not int or top_k < 0 or type(seed) is not int:
raise ValueError('top_k/seed')
if type(temperature) not in (int, float):
raise ValueError('temperature')If the path is not /generate, it is 404. Then inside try it checks things one by one, and if any is wrong, it raises .
result = generate(model, tokenizer, prompt, tokens, temperature, top_k, seed)
self.reply(200, {'text': result, 'model': 'XSLM'})
except (ValueError, KeyError, TypeError):
self.reply(400, {'error': 'invalid request'})If the checks pass, it calls generate and answers 200 with {"text": …}. ValueError, KeyError and TypeError all mean a bad request, so it answers 400 and the server keeps running. If the server died on every error, one request could stop the whole service.
address = ('127.0.0.1', cfg['port'])
print('XSLM Runner:', address, flush=True)
HTTPServer(address, Handler).serve_forever()The address is 127.0.0.1 and the port from the settings. It prints the startup line right away with =True so you can tell from runner.log that it started. HTTPServer(…).serve_forever() endlessly repeats taking one request at a time and passing it to Handler. Stop it with kill or .
Expect status ok from health and a JSON text field from generation. Send an empty JSON object: the API should return 400 instead of crashing. The response must come from your checkpoint, not a canned answer.
: inspect runner.log and confirm a checkpoint exists. Address already in use: keep the existing Runner or stop its recorded PID before restarting. After new weights, restart the Runner to load them.
The Runner runs in the virtual Linux inside the browser and waits only on that Linux’s 127.0.0.1. localhost in the address bar is your real computer, and there is no Runner there. Run curl in the terminal on the right.
Another Runner is already using port 8765. Keep using the running Runner, or stop it with kill "$(cat runner.pid)" and start it again. If there is no PID file, find the ID of python3 runner.py with ps and kill it.
The Runner reads checkpoints/model.json only once when it starts and keeps it in memory. It does not notice when the file changes. Stop it and start it again to use the new weights. Real services switch without stopping by starting the new model alongside and moving requests over to it.