You've worked the two levers that shrink the cost of every token: quantisation — splitting a bit budget between range and precision, then climbing the PTQ-to-QAT ladder only as far as your quality bar demands — and speculative decoding — proposing tokens cheaply and verifying them for free, from draft-target to Medusa, EAGLE, and the draft-free n-gram and lookahead tricks. Prove it stuck before the serving lessons.
Eight questions. Score 80% or better to unlock the serving half of the chapter — prefix caching, KV storage and routing, long context, tensor and expert parallelism, multi-node, and disaggregation. Every option explains itself, and you can take another pass any time.
01 · Checkpoint
Cheaper tokens — quantisation & speculation.
The range/precision bit budget and FP8 scaling, the quantisation ladder, AWQ vs SmoothQuant, why perplexity is necessary but not sufficient, the output-equivalence of speculation and the accept-rate lever, Medusa vs EAGLE, and the draft-free n-gram / lookahead methods.
Check your understanding0 / 8
Question 1
BF16 and FP16 are both 16-bit floats. Why does BF16 dominate for inference weights despite being less precise than FP16?
Question 2
Why does FP8 inference need a per-tensor or per-channel scaling factor that FP16 can usually skip?
Question 3
The lesson frames quantisation as a five-rung ladder (PTQ → GPTQ → AWQ → SmoothQuant → QAT). What's the operating rule?
Question 4
AWQ and SmoothQuant both fight the same enemy. What is it, and how do they differ?
Question 5
You quantised a model and perplexity barely moved. What does the lesson say you can conclude?
Question 6
Draft-target speculative decoding runs a small model to propose tokens. What is the single most load-bearing property, and what decides the speedup?
Question 7
Medusa and EAGLE both drop the separate draft model. How do they differ, and which tends to accept more?
Question 8
You have no draft model at all. What do n-gram speculation and lookahead decoding each do, and how are they verified?