You've met the core of the engine: what a forward pass is, why a linear layer's cost flips between memory- and compute-bound, what the activation buys you, how a transformer block stacks, where the parameters and the KV cache actually live, and how MoE decouples capacity from cost. Prove it stuck before the modalities.
Eight questions. Score 80% or better to unlock the image-gen, video, and bottleneck lessons. Every option explains itself, and you can take another pass any time.
01 · Checkpoint
Transformers & attention — the core.
Neural-net basics, matmul FLOPs, activations, transformer architecture and parameter counts, attention with the KV cache and GQA, and mixture-of-experts.
Check your understanding0 / 8
Question 1
Running a trained neural network to produce an output — inference — consists of:
Question 2
A linear layer maps a length-d_in vector to a length-d_out vector. Roughly how many FLOPs does it take per token?
Question 3
The exact same linear-layer matmul is memory-bound at batch size 1 but compute-bound at large batch. Why?
Question 4
Why does a transformer need a nonlinear activation (e.g. SwiGLU) between its linear layers?
Question 5
In a modern decoder-only block, which part holds most of the parameters (and is why decode is memory-bound)?
Question 6
A modern (Llama-style) transformer block applies, in order:
Question 7
Grouped-Query Attention (GQA) shrinks the KV cache versus full multi-head attention by:
Question 8
A Mixture-of-Experts layer replaces the dense FFN with N experts plus a router that fires top-k of them per token. The effect is: