Sign up to keep track of your learning progress.
Log inThe road your request travels
One common path for a cloud text assistant with optional tools. Local models, multimodal models, and providers can use different paths; the numbered steps simplify that variety.
The big picture — five places
- Your computer
- The network
- Remote machines
The program on my computer
A chatbot, a code editor, an app. It takes your request and decides what to send.
The network
The road across the internet. Nothing but text and files travels on it.
The distant calculating machine
A runtime loads model files and executes inference. A warm worker often reuses resident weights across requests; cold starts, eviction, offloading, or restarts can load them again. Models may span several devices or servers.
The calculating model
A neural language model learns numerical weights from training examples. At generation time, those weights and the available context determine scores for candidate tokens. The model does not simply look up a stored internet sentence or count only the word immediately before it.
Back on my computer
The program shows the answer, and — only if you allow it — does the work.
Zones 1 and 5 are the same computer: yours
Opening up place three
-
A big file of numbers, sitting asleep on a disk. That file is the model.
-
A runtime reads it and puts it on a chip built for calculating, then keeps it there.
-
A doorkeeper checks the key and the request waits its turn.
-
The calculating runs against the numbers that are already up there.
Step by step
Eight steps, in order. Step eight is where the road forks.
You send it
A speech-to-text system can transcribe audio before a text model receives it. Other models accept audio directly. Microphone permission and the selected service determine where processing happens.
The program packs a bag
The application selects messages, instructions, and permitted attachments within the context limit. It may truncate, summarize, or retrieve older material. Omitted details are not automatically remembered, and summaries can lose information.
It crosses the internet
HTTPS encrypts the connection. HTTP/1.1 and HTTP/2 commonly use TCP plus TLS; HTTP/3 uses QUIC. Live services may use WebSocket or WebRTC. This diagram shows the purpose of the connection, not one mandatory transport.
The machine receives it
A runtime loads model files and executes inference. A warm worker often reuses resident weights across requests; cold starts, eviction, offloading, or restarts can load them again. Models may span several devices or servers.
It draws the next token
Sampling can produce different answers, and it can repeat an answer. A low positive temperature concentrates the distribution; it does not make facts truer. Even temperature zero does not promise identical output across real systems: context, model versions, tools, and numerical execution can also change results.
back to this step, token by token
It sends back as it goes
Text can stream in chunks, often through SSE. Audio and some image or video systems can stream too; others return completed files. Chunk size and timing are service choices, not proof of a human typing or thinking.
The program puts it on screen
Text can stream in chunks, often through SSE. Audio and some image or video systems can stream too; others return completed files. Chunk size and timing are service choices, not proof of a human typing or thinking.
Here the road forks
If the answer is only an answer, you are done. If it contains "please do this", follow the second road.
Two roads, not oneTwo roads, not one
Drawing these as a single road is where the frightening misunderstanding comes from. They are not the same.
Talking only
-
Text on screen
-
Ends here
Read more
Letters flow onto the screen and that is the end of it. Nothing on your computer is touched. Lyen, the study helper on this site, works this way.
Borrowing hands
-
Spotting a tool request
-
Asking permission
-
Actually doing it
-
Packing the result
The result goes into the bag at step two and out again at step three
Read more
A model can propose a structured tool call. An application validates it and, within its permissions, executes it locally or on a remote service. The result may return to the model. Limits, cancellation, and failures can stop the loop.
Permissions, isolation, validation, and any required approval must be enforced by the application or service. A system prompt alone is not a security boundary. Previously authorized actions may run without another prompt; controls can fail if poorly implemented.
If we must compare it to something
-
One body swaps its calculating part out. The same code editor can be wired to a model from a different company tomorrow.
-
One calculating part serves crowds of people who do not know each other, all at the same time. Nothing alive could do that.
Read more
A model can propose a structured tool call. An application validates it and, within its permissions, executes it locally or on a remote service. The result may return to the model. Limits, cancellation, and failures can stop the loop.
Two more things
The machine is not always far away
This very site is that picture
| 1 · Your app | 2 · The network | 3 · Remote machine | 4 · The calculating model | 5 · Your screen | |
|---|---|---|---|---|---|
| Inside my own browser | Yes | No | No | Yes | Yes |
| By way of our server | Yes | Yes | Yes | Yes | Yes |
| On a distant company’s machine | Yes | Yes | Yes | Yes | Yes |
Even on the middle road, the calculating happens on the third one. What our server does is check and carry.
Read more
PES's Supertonic voice synthesizes locally after loading its model. Browser speech services may use local or remote voices. This does not make the chat private to the device: chat requests still go through PES to the language-model service.
Open the study helper and ask it something. Your browser sends it to our server, and our server hands it to a distant calculating machine. The letters coming back in pieces is step six; the letters growing on your screen is step seven. The helper stays on the talking-only road: it never touches a file on your computer.
The study helper. Our server passes the request along; it does not run a model.
This is where the calculating happens. Which runtime they use in there is not published.
Notes for grown-ups
Slightly more exact wording, for anyone who wants it.
Token
A tokenizer maps text pieces to IDs in its own vocabulary. Pieces can be words, subwords, characters, or bytes. The same text can have different IDs and token counts with different tokenizers.
Streaming
Partial output is pushed out as it is produced, usually over SSE. How big each piece is differs by product; some send a sentence at a time rather than a token.
Kinds of model
LLM (large language model), SLM (small language model), VLM (reads pictures as well), and diffusion models for images and video. They are built differently, and they do not all produce text.
The pick
Sampling can produce different answers, and it can repeat an answer. A low positive temperature concentrates the distribution; it does not make facts truer. Even temperature zero does not promise identical output across real systems: context, model versions, tools, and numerical execution can also change results.
RAG, MCP, skills
Tools can connect directly through application APIs or through MCP. MCP can expose tools, resources, and prompts on local or remote servers. Availability comes from application configuration and permissions, not from the model alone.
Programs that run models
What this site actually uses is ONNX Runtime (for speech) and sherpa-onnx (for dictation). For large language models on a server, programs with the same job — vLLM, TensorRT-LLM, llama.cpp and others — are used instead. Which company runs which is not published.
Numbers, precision, memory — and the runtime that puts them to work
-
Model file Sheet music
-
Runtime Conductor
-
CPU · GPU Performers
| Best at | |
|---|---|
| ONNX Runtime | Many kinds of models on many devices |
| llama.cpp | Running LLMs on your own machine (GGUF) |
| FreeToken | Huge MoE models on personal PCs |
Read more
An AI model is a learned arrangement of vast numbers and computation. A runtime is the engine that loads that arrangement into memory and executes it on a CPU, GPU, or another accelerator.
Think of the model file as a score, the CPU and GPU as musicians, and the runtime as the conductor that reads the score and coordinates the performance.
ONNX Runtime — A general-purpose, cross-platform engine that can run models for images, speech, language, and more through hardware-specific execution providers.
llama.cpp — An engine focused on running LLMs and VLMs with minimal setup across local and cloud hardware, commonly using GGUF model files.
FreeToken — A new, edge-native serving engine specialized for very large MoE models. It coordinates NVIDIA GPUs, CPUs, host RAM, and the links between them as one inference platform.
What is a parameter?
- Tensor
- Matrix
- Vector
- Scalar = 1 parameter
- Vector
- Matrix
[0.2, −0.7, 1.3]
1 vector · 3 scalar numbers · 3 parameters
In 8B, B means billion—not byte. An 8B model has about eight billion scalar parameters, not eight billion vectors.
Parameters and weights
- Weights
- Biases
- Embeddings
- Normalization scales
y = w₁x₁ + w₂x₂ + bEvery weight is a parameter, but not every parameter is a weight.
Read more
A parameter is one scalar number learned or adjusted by the model. A vector, matrix, or tensor is a container; every scalar number inside it counts as one parameter.
A weight is a major kind of parameter: it controls how strongly an input affects a result. Biases, embeddings, and normalization coefficients may also be parameters.
A parameter is not a box holding a whole fact. The tensor or matrix is the cabinet, and each parameter is one numbered slot inside it. Bit width says how precisely that number is recorded.
How many bits does one parameter use?
Read more
A parameter is one number, but it can be stored at different precision. Fewer bits usually reduce memory use, while quantization metadata and implementation details add some overhead.
Why can an 8B model be about 8 GB?
weight size ≈ parameter count × bits per parameter ÷ 8These are simplified decimal estimates for weights only. Real quantized files can be larger because they also store scales, metadata, and other structures.
Read more
At 8-bit precision, each parameter takes about one byte. Eight billion parameters therefore need roughly 8 GB for weights alone. The same parameter count takes a different amount at a different precision.
Running the model needs more memory
- Weights
- KV cache
- Compute buffers
- Runtime overhead
The KV cache grows with longer context
working memory = weights + KV cache + compute buffers + runtime overheadRead more
The weight file is only the starting point. Inference also needs a KV cache for earlier tokens, temporary computation buffers, and runtime overhead. A theoretically 4 GB model may therefore need 5–8 GB or more; longer context usually grows the KV cache.
How can small numbers express so much knowledge?
Read more
Eight bits can represent 256 levels, but no single parameter contains a sentence such as “Seoul is the capital of South Korea.” Knowledge is distributed across patterns formed by billions of parameters interacting through many layers.
One pixel is only a color value. Arrange millions of pixels and a face or landscape appears. Model knowledge emerges from distributed numerical patterns in a similar way.
The model is a vast pattern of scalar parameters grouped into vectors, matrices, and tensors. Precision determines the theoretical weight size; the runtime loads those weights and uses additional working memory to execute them on the available hardware.
The same road, step by step
- 1 From my hands to the program
- 2 Packing what gets sent
- 3 Crossing the internet
- 4 At the server door
- 5 Inside the server — the machine and the runtime
- 6 The model calculates
- 7 One token at a time, over and over
- 8 The way back
- 9 The program takes it in
- 10 Borrowing hands, over and over
1 From my hands to the program place 1 4 steps
-
A key goes down
A tiny computer inside the keyboard notices which key moved, turns it into a number, and sends it down the cable.
-
The operating system catches it
A driver in the kernel turns that number into an input event and queues it for whichever window is in front.
-
The program pulls it off the queue
Every program runs a loop that takes one event at a time and deals with it. The letter appearing in the box is this step.
-
The microphone takes another route
A speech-to-text system can transcribe audio before a text model receives it. Other models accept audio directly. Microphone permission and the selected service determine where processing happens.
2 Packing what gets sent place 1 7 steps
-
The system prompt
Rules written in advance by whoever built the program go in first — what it is, and how it should answer.
-
The conversation so far
The application selects messages, instructions, and permitted attachments within the context limit. It may truncate, summarize, or retrieve older material. Omitted details are not automatically remembered, and summaries can lose information.
-
Whatever you attached
An API may accept image bytes, a file ID, a URL, or base64. PDFs may yield text, page images, or OCR results. File support, extraction quality, and size limits vary by service.
-
Searching your documents (RAG)
RAG retrieves material and supplies it as context for generation. Retrieval may use keywords such as BM25, embeddings, or both. It usually does not retrain model weights, and retrieved text still needs source and relevance checks.
-
The list of tools
Tools can connect directly through application APIs or through MCP. MCP can expose tools, resources, and prompts on local or remote servers. Availability comes from application configuration and permissions, not from the model alone.
-
Skills and commands
A skill packages instructions, and may include scripts and reference files. A compatible agent can load relevant parts when needed. The Agent Skills website defines the format; it is not a guarantee that a skill is safe or correct.
-
Making it fit
The application selects messages, instructions, and permitted attachments within the context limit. It may truncate, summarize, or retrieve older material. Omitted details are not automatically remembered, and summaries can lose information.
3 Crossing the internet place 2 4 steps
-
Finding the address
The server’s name is looked up and turned into a numeric address (DNS).
-
Opening the road and locking it
HTTPS encrypts the connection. HTTP/1.1 and HTTP/2 commonly use TCP plus TLS; HTTP/3 uses QUIC. Live services may use WebSocket or WebRTC. This diagram shows the purpose of the connection, not one mandatory transport.
-
Saying who you are
A key or a token is attached to the head of the request. This is what decides whose account gets charged.
-
Sending the request
HTTPS encrypts the connection. HTTP/1.1 and HTTP/2 commonly use TCP plus TLS; HTTP/3 uses QUIC. Live services may use WebSocket or WebRTC. This diagram shows the purpose of the connection, not one mandatory transport.
4 At the server door place 3 3 steps
-
The doorkeeper
The key is checked, the call rate is checked, and the request is handed to a machine that is free.
-
The queue
At the door, keys are checked and requests wait briefly in line. Later, when the calculating happens, the runtime works through several waiting questions against the one set of numbers it already holds.
-
Checking what came in
The request is looked over against the rules before anything is calculated.
5 Inside the server — the machine and the runtime place 3 3 steps
-
A file of numbers lies asleep
A model that has finished training is a large file on a disk. By itself it calculates nothing. The speech model on this site is four files, about 398 MB together.
-
A runtime loads it, once
A runtime loads model files and executes inference. A warm worker often reuses resident weights across requests; cold starts, eviction, offloading, or restarts can load them again. Models may span several devices or servers.
-
An accelerator takes the multiplying
The same multiplication has to happen an enormous number of times, so it is handed to a chip built for exactly that, with fast memory beside it where the numbers sit. An ordinary CPU can do it too — it is only slower.
The runtime does not live inside the accelerator. It is a separate program that reads the file and puts the numbers into the memory next to that chip.
Grouping several requests is step 17, and keeping the finished work is step 27 — this same program does both.
6 The model calculates place 4 7 steps
-
Cutting into tokens
A tokenizer maps text pieces to IDs in its own vocabulary. Pieces can be words, subwords, characters, or bytes. The same text can have different IDs and token counts with different tokenizers.
-
Numbers into lists of numbers
A token ID selects a learned embedding vector. The ID's numeric size has no semantic meaning. Useful similarities can emerge in learned vectors, but depend on the model and task; closeness is not a guarantee of identical meaning.
-
Marking the order
Position information lets the model distinguish token order. Some architectures add position vectors; others use methods such as rotary position embeddings. Position handling is not identical in every Transformer.
-
Looking at each other (attention)
An autoregressive Transformer applies attention and other transformations through its layers. Causal attention uses only allowed earlier positions, sometimes within a limited window. Architecture and layer count differ by model.
-
Layer after layer
An autoregressive Transformer applies attention and other transformations through its layers. Causal attention uses only allowed earlier positions, sometimes within a limited window. Architecture and layer count differ by model.
-
Scoring everything
The last layer gives a score to every token it knows — tens of thousands of scores at once.
-
Picking one
Softmax converts logits into probabilities. Temperature rescales logits before sampling. Top-k keeps a fixed number of candidates; top-p keeps a probability-mass set. Greedy chooses a highest score. Provider handling of temperature zero varies.
7 One token at a time, over and over place 4 3 steps
-
Adding it to the end
-
Keeping the work already done
A KV cache reuses past attention keys and values during autoregressive decoding. It saves recomputation but consumes memory. Latency also depends on context length, batching, hardware, and network conditions; later tokens are not always faster.
-
Back to step 22
A model can propose a structured tool call. An application validates it and, within its permissions, executes it locally or on a remote service. The result may return to the model. Limits, cancellation, and failures can stop the loop.
This is the loop that writes the answer — one token at a time, back to step 22.
8 The way back place 2 4 steps
-
Turning token numbers back into readable text
-
Letting it flow as it comes
Rather than waiting for the whole answer, small chunks are pushed out as they are produced (SSE).
-
Checking what goes out
The outgoing text is looked over against the rules as well.
-
Counting what was used
Tokens in and tokens out are counted. This is where the bill comes from.
9 The program takes it in place 5 4 steps
-
Stitching the arriving chunks in order
-
Drawing it on screen
Markdown is read and turned into headings, lists and code boxes. This is the part that looks like typing.
-
Spotting a tool request
The program watches for a tool call in the agreed format sitting inside the answer.
-
Checking the arguments
Whether that tool exists, and whether the arguments have the right shape, is checked before anything runs.
10 Borrowing hands, over and over place 5 4 steps
-
Asking permission
Permissions, isolation, validation, and any required approval must be enforced by the application or service. A system prompt alone is not a security boundary. Previously authorized actions may run without another prompt; controls can fail if poorly implemented.
-
Actually doing it
A model can propose a structured tool call. An application validates it and, within its permissions, executes it locally or on a remote service. The result may return to the model. Limits, cancellation, and failures can stop the loop.
-
Packing the result
What came out — or why it failed — is added to the conversation from step 6 as a new line.
-
Back to step 15
A model can propose a structured tool call. An application validates it and, within its permissions, executes it locally or on a remote service. The result may return to the model. Limits, cancellation, and failures can stop the loop.
This is the loop people call an agent — back to step 15, again and again.