SEPARATE VISUAL AID / NEXT-TOKEN PREDICTION
One token inside a model
Watch a small pretrained story model run, then slow down one of its calculations. The left column names the steps, the center shows a calculation, and the right side shows the weights and input values used.
Select Load pretrained model, then Process prompt / reset. After a prediction, use Play calculation to replay its arithmetic. Select Generate next token to continue the story. The live status and replay status identify which activity you are viewing.
Watch the original Qwen reference video
The original 60-second recording, compressed for playback. This is recorded Qwen output, not live inference on this page. It shows a layer list, one calculation, weights, and progression toward a selected token. The interactive viewer below follows that layout using TinyStories values.
Run TinyStories-1M, a small pretrained story-completion transformer. The browser version uses your CPU in a background worker. No prompt is sent to an inference server.
Load the weights to begin. The download is about 15 MiB plus the tokenizer.
Load the model and process a story opening.
NEXT TOKEN —
No prediction captured yet
MLP · up projection
Row 1 of 256 · term 1 of 64
partial sum = previous sum + weight × input
Process a prompt to inspect a measured calculation.
The layout follows the reference video. Values come from TinyStories, a different, much smaller model. This is a replay of captured arithmetic. At faster speeds the cursor skips terms between frames. The display clock is an illustration, not measured inference time.
More views and data: neural field, activations, and saved traces
TinyStories neural field
The text being processed will appear here.
zj = bj + Σ Wj,i xi → GELU(zj)
Select a computed unit to inspect its actual inputs and weights.
The term slider inspects the calculation. It does not rerun the model.
Drag the field to rotate it; click a unit to inspect it. Spatial layout groups units by layer, not brain anatomy. Glow shows measured signed activation magnitude (saturated at |activation| = 1). Lines show layer order, not individual learned connections. Updates arrive from the running calculation; observation pauses are intentional.
Text already processed
Next selected token:
Next-token candidates
Softmax over all 50,257 vocabulary scores, before any sampling filters. Greedy decoding chooses the highest score. These probabilities do not measure factual accuracy.
Memory payloads
Stable allocated weight storage does not mean zero repeated reads. Storage capacity and memory traffic are different quantities.
Actual layer activations
Blue = negative; orange = positive; white = near zero. All cells use the same scale for this frame. Brightness shows magnitude, not importance or a named concept.
Point at a cell or focus a layer and use the arrow keys to inspect exact values.
Read all activation values as numbers
About this visual aid
The instrument layout follows the supplied Qwen video. This viewer runs the pinned TinyStories GPT-Neo model, not Qwen. Its eight layers each have 256 feed-forward units. The MLP up-projection replay uses the captured normalized input, actual learned weights and bias, partial sums, and measured GELU output. Its 131,072 displayed terms cover those eight projections, not every operation in the model. JavaScript sums and the worker’s float32 storage can differ slightly at rounding.
During inference, completed attention and feed-forward stages arrive from the browser worker. The slower arithmetic display is a replay after the prediction. Pausing that display does not pause a scalar instruction on the CPU. The one-term-per-frame clock describes display time at 60 frames per second, not GPU speed or hardware traffic. Our CPU timings include observation work; intentional pauses are excluded.
The model has fixed pretrained weights and chooses the highest-scoring next token. Colors indicate signed numeric values, not ideas or importance. The optional neural field groups units by layer, not semantic clusters. The model is dense. The separate MoE lesson uses a labeled untrained sparse network.
Optional advanced resources: Python observer notebook · Python source. These are extensions, not required class assignments. Model source.