01 / EXPERT NETWORKS
Several clusters, one model
Imagine a school with many workshop rooms. A student visits two rooms for a project instead of completing an activity in every room. The school can offer more rooms without requiring every student to visit them all.
A Mixture of Experts, or MoE, uses a similar idea inside a neural network. A router scores expert networks for a token and selects a small group. Each expert is a cluster of neurons doing a numerical calculation. The selected outputs are combined before the model continues.
Real routers learn their weights during training. “Expert” does not mean a separate chatbot, a person, or a guaranteed specialist in a school subject. Lesson 6’s harness could choose a calculator or an agent outside the model. Here the choice happens inside a model layer.
02 / NEURON ACTIVITY
Watch the selected clusters
Each card below is one expert containing 12 neurons. The token chips move through a fixed example sentence. Their numerical features stand in for representations from earlier layers; this little network does not generate the sentence.
Which neurons do the work?
Select a neuron in a used cluster to inspect its calculation.
Expert visits this sentence
Counts include the current token and earlier visited positions, once each. Reset or change a setting to start a new count.
Read the calculated values
The token features and weights are hand-chosen. Each transition is a slowed visual replay of a local calculation. No language model or hardware measurements are being observed here.
- Keep 8 clusters and 2 selected. Select Play sentence. Notice which cards receive the token and which stay marked Skipped.
- Pause on a token. Click a neuron in a selected cluster to read its actual activation. A value of zero means the neuron was calculated and its ReLU function returned zero.
- Select Next token. Do the same experts always receive the next token?
- Switch to All clusters. Compare the number of expert calculations. This comparison runs every expert in our teaching network; it is not a speed benchmark.
03 / CAPACITY AND WORK
What happens when we add more experts?
A parameter is a stored model number, such as a weight or bias. More expert networks add parameters, giving a trained model more capacity to learn patterns. MoE can increase that pool without using every expert for each token.
- Choose Selected clusters (MoE), 2 selected, and 4 clusters. Record stored parameters, expert parameters used, and shared/router parameters used.
- Increase the pool to 8, then 12 clusters, keeping 2 selected. Which number stays the same? Which numbers grow?
- With 12 clusters, compare 2 selected and 4 selected. Explain why selecting more experts increases this token’s expert work.
Each expert here contains 112 parameters. Choosing two uses 224 expert parameters whether the pool has 4, 8, or 12 experts. The shared layers still run, and the router scores every expert, so the total active count grows slightly as the pool grows.
These counters count parameters used by this small calculation. They do not measure elapsed time, electrical power, GPU memory traffic, or how good a trained model’s answers would be.
04 / SHARING THE WORK
What if everyone picks the same room?
Return to the workshop analogy. Ten rooms will not prevent a queue if nearly everyone needs room 1. Real MoE systems also need to manage uneven routing. Some experts may receive many tokens while others receive very few.
Select Play sentence for each example and inspect the Expert visits this sentence chart. It counts the selections for the tokens you have actually stepped through. A new sentence or a network-setting change starts a fresh count. Does adding experts guarantee equal visits?
Engineers can train routers to spread work more evenly and can place experts across computing devices. Several tokens can be processed together, and different tokens can choose different experts. That helps share resources, but moving information between devices has a cost.
Classroom activity
Make eight expert-room cards and give six students token cards. Each token names two rooms to visit. Count the visits per room. Repeat after giving every token the same two rooms. Both rounds use two rooms per token; compare the busiest room in each round. Propose a way to reduce the queue and explain what might make your proposal difficult.
05 / REVIEW
Design a small MoE
Turn in: a three-row table for 4, 8, and 12 clusters with two selected, one inspected neuron value, and a sketch showing which clusters a token uses. Explain one benefit of MoE and one reason a bigger model might still need more resources.
Optional VCC activity: Ask the coding agent to build a workshop queue simulation with eight rooms and six token cards, each visiting two rooms. Include one evenly spread case and one crowded case. Check the total of 12 visits in each, and compare the busiest room. Label the simulation as an analogy for routing. Save your work and stop the workspace.
Instructor notes & sources
Suggested pacing: 7 minutes for experts and the analogy, 13 for the neuron activity, 12 for the scale comparison, 10 for room queues, and 5–8 for review. Students need only multiplication and comparisons. The queue activity can run without computers.
The demo uses hand-chosen weights, four input values, 12 ReLU units per expert, two shared projections, and a linear router. Only selected expert functions execute. It is an untrained neural calculation, not a pretrained MoE language model. All-clusters mode uses the same network with every expert selected. The displayed output can change when the selected set changes. More capacity is not a measured quality improvement here.
Source: Switch Transformers explains sparse computation, scaling, and routing constraints. Our top-two teaching gate differs from Switch’s top-one gate. The Qwen3 model release gives real examples with different total and active parameter counts.