01 / THE BASIC JOB
Next-token prediction
A large language model (LLM) is a neural network trained on large amounts of data. For the text-generating models in this lesson, the basic job is to predict the next token from the tokens already in context. Append a prediction, feed the longer sequence back in, and repeat. That loop can produce a sentence, a conversation, or code.
From text to the next token
The model produces a distribution over its vocabulary. A decoding rule selects a token to append to the sequence.
The selected token becomes part of the input for the next prediction. The model also uses position information to represent token order. The small teaching model below omits it.
Take “The cat sat on the …” You can probably suggest a few endings. The model assigns scores to many possible next tokens, turns those scores into probabilities, and a decoding rule chooses one.
Training changes the model’s weights by comparing predictions with training examples. Inference uses those weights to produce outputs. Typing a new prompt changes the input to inference; it does not automatically retrain the model.
QUESTION Would “The player stepped onto the …” produce the same next-token scores as the cat example? Write an ending for each.
02 / LOOK UNDER THE HOOD
Which earlier tokens matter?
Tokens first become lists of numbers called embeddings. In a transformer layer, attention lets each position mix information from other positions. A query is compared with keys, and those scores weight the values to mix. In a causal text generator, a position can use itself and earlier tokens, never future ones.
Each token produces a query, a key, and a value
Each layer makes three numeric vectors for each token. Query (Q) is used to score the available keys. Key (K) is what each query is compared against. Value (V) supplies information for the weighted sum. Three learned matrices produce these vectors; the same matrices are used at every position. X contains the layer’s input vectors.
Swipe or scroll sideways to view the whole diagram.
A dot product multiplies matching entries of two vectors and adds the products. Attention scales these scores, then uses softmax to turn the set of scores into weights that sum to 1. Those weights are used to mix the value vectors.
A feed-forward neural network then transforms each position’s numbers. Models stack many layers of attention and feed-forward computation. A neuron’s activation is a number in that computation; it is not a tiny person thinking a word.
What does the last position attend to?
Each bar is an attention weight. Together they total 100% before rounding. Earlier positions cannot see future tokens.
What token comes next?
These are next-token probabilities over four toy candidates. A high score says how likely a continuation is in this model; it does not measure truth.
Neural network activations
Query–key scores determine how the values are mixed
This diagram uses the same computed values as the example above and updates with its controls. The teaching model uses identity projections, so each token’s Q, K, and V contain the same numbers. Real models learn separate projections.
Swipe or scroll sideways to view the whole diagram.
Attention weights total 1 before rounding. They describe the mixture of value vectors for one position. The next-token probabilities are a separate distribution produced later in the network.
What is actually running here?
A four-number embedding for each toy token, one causal attention head with identity query/key/value projections, a residual connection, two dense layers with ReLU between them, another residual connection, and an output softmax. Circles show actual forward-pass values for the last position. Brighter circles mean larger values; every value is also printed.
The weights are hand-chosen to make this small example readable. It has no training, layer normalization, positional embeddings, or full language vocabulary. The chips use a word/punctuation tokenizer; production tokenizers often split words into smaller pieces. This is an inspectable teaching model, not a view inside the classroom LLM. It always appends the highest-probability candidate, so changing temperature changes the distribution but not its ranking.
Optional math: attention = softmax(QKᵀ / √4)V. The mask removes future positions. The feed-forward step is h = ReLU(0.8x − 0.1), then output = x + 0.5h. Output scores become probabilities with softmax.
- Compare all three contexts before appending anything. Find the token with the strongest attention weight and the most likely continuation.
- Append one token. Notice that the context grew and the next prediction changed. Append again to finish the toy sequence.
- Reset. Change temperature from 0.2 to 2.0. Notice how the probabilities spread out. The demo still picks the largest one.
- Choose one printed neuron value. Explain which stage produced it and whether it changed when you changed the context.
03 / FROM PREDICTION TO CONVERSATION
How does it become a chatbot?
The next-token mechanism stays. The training and input format teach it what a helpful conversation looks like.
- Pretraining: learn patterns from large datasets by predicting tokens.
- Instruction/chat tuning: train further on instruction-and-response or conversation examples. Preference training can further shape which responses it tends to give.
- Chat formatting: the app packages system, user, assistant, and sometimes tool messages into the model’s expected token sequence.
- Generate the assistant turn: the model continues that formatted sequence until a stopping condition or output limit.
The user role contains your request. The assistant role marks the model’s replies. The system role carries app-level instructions, such as how to teach or format an answer. The app usually sends selected conversation history again on later turns.
Chat messages become a token sequence
The app keeps track of message roles and formats them using the model’s chat template.
The role markers used by the formatter below are illustrative. Exact chat templates and token IDs depend on the model.
The messages the app prepares
A possible serialized token stream
Role markers below are illustrative. Each model has its own chat template and special token IDs. The assistant marker tells the model whose turn it is to continue.
This formatter calls no LLM. The example answer is fixed; edit the user message to inspect the input, then test real behavior in VCC.
04 / TRY IT IN VCC
System instructions
A system prompt can describe the assistant’s role, response style, and tool rules. A classroom tutor might be told to ask one guiding question before giving an answer. These instructions influence generation because of the model’s training. They can fail, and they do not give it new tools or make its claims automatically true.
In this VCC project, make a small Node script that sends the same user question twice to the existing classroom model gateway. First use the system instruction "Explain tokens in two short sentences for a high school student." Then use "Ask one guiding question about tokens before giving an explanation." Keep the model and user question the same. Print the role-labeled requests, returned text, and provider token usage when available. Use the existing VCC model configuration; add no packages or personal API keys. Do not print credentials. Explain how to run it in Bash. Keep the starter app and database working.
Open VCC Projects ↗, create a project, and stay in VCC sandbox. Use Ask agent in Build & activity to create the experiment. Open the coding workspace, start it, then run the command it supplies in the Bash tab.
Record the two instructions, one difference in the replies, and one claim you checked. Model answers can vary across runs even when a prompt stays the same.
05 / REVIEW
Review questions
Review: Trace one message from user text → tokens → embeddings → attention and feed-forward layers → next-token probabilities → assistant reply. Then explain why a chatbot still needs extra software to read a file or run code.
Show your instructor: one context comparison from the demo, your two VCC system prompts and replies, and your training-versus-inference explanation.
Instructor notes & sources
Allow about 5–10 minutes for the opening model, 15 for the browser investigation, 20–25 for the VCC activity, and 5 for the review questions. Students can work in pairs and compare their answers. Provision VCC access before class. The demos need no sign-in, API key, or paid inference. If class AI is unavailable, students can inspect demos, run the offline harness, and write expected traces; the live assignment still needs a later model run.
Assess the prediction, trace, and explanation along with the result. The small models, invented player data, fixed chat replies, and scripted browser requests are labeled at the point of use.
Further reading: Attention Is All You Need · Chat templates · Agent workflows and routing · Switch Transformers.