Skip to content

Loading

“All non-trivial abstractions, to some degree, are leaky.” — Joel Spolsky

IInference/Engineering/Lesson 98 of 101

Inference Engineering

8 chapters · 101 lessons

What inference actually is
What this course covers (and what it doesn't)
Who this is for and what Larry already knows

Scale and specialization
About your app — the three axes Larry picks on
AI-native applications
Online vs offline inference
Consumer vs B2B
How Larry picks a model
Evaluation — what 'good' means here
Fine-tuning for domain-specific quality
Distillation — when a smaller student wins
Measuring latency and throughput

Neural networks in 800 words
Linear layers and matmul
Activation functions
The autoregressive loop
LLM architecture in one diagram
Transformer blocks, line by line
Attention — Q, K, V from the inside
Mixture of experts
Image-generation model architecture
Few-step image generation
Video generation — the temporal axis
Ops:byte ratio and arithmetic intensity
LLM inference bottlenecks
Image-gen bottlenecks
Optimizing attention — the standard moves

GPU compute — SMs, warps, tensor cores
GPU memory and caches
Hopper GPUs (H100, H200)
Ada Lovelace (L4, L40, RTX 40xx)
Blackwell (B100, B200, GB200)
Rubin GPUs — what's coming
Grace + Vera CPUs — the host side
Instances — multi-GPU and MIG
Other accelerators
Local inference — desktop and mobile

CUDA in one chapter
CUDA kernels for inference
Kernel selection — heuristics and autotuning
Reducing memory accesses with fusion
PyTorch as an inference runtime
Model file formats
ONNX Runtime and TensorRT
HF transformers and diffusers
vLLM — what it actually does
SGLang — and where it diverges
TensorRT-LLM — the NVIDIA opinion
NVIDIA Dynamo
Benchmarking and profiling

Number formats
Quantization approaches
Measuring quality impact
Draft-target speculative decoding
Medusa — multi-head speculation
EAGLE — the cheaper proposal
N-gram speculation and lookahead
Prefix caching and KV reuse
Where to store the KV cache
Cache-aware routing
Long-context handling
Tensor parallelism — lower latency
Expert parallelism — higher throughput
Multi-node inference — the bandwidth math
How disaggregation works
When to disaggregate (and when not to)
Dynamic disaggregation with NVIDIA Dynamo

Vision language models
Video processing for VLMs
Omni-modal models — all inputs, one model
Embedding model architecture
Embedding model inference at scale
ASR — single-chunk latency
ASR — long-file latency
ASR — diarization
Streaming real-time TTS
Speech-to-speech models
Image generation kernel optimization
One weird trick for faster image generation
Video generation — attention opt and quant
Context parallelism

Containerization — what a serving container needs
Dependency management — pinning the right way
NIMs — NVIDIA's pre-baked containers
Concurrency and batch sizing
Cold starts
Routing, load balancing, queueing
Scale to zero — economics and UX
Independent component scaling
GPU procurement — where to find them
Geo-aware load balancing
Building for reliability
Security and compliance
Zero-downtime deployment
Cost estimation — from FLOPs to dollars
Observability
Client-side latency overhead
Asynchronous inference
Streaming and protocol support
Production inference with Baseten — or roll your own

Chapter 7 · Production

Client-side latency overhead

The latency Larry didn't measure, and his user did.

Lesson 98 / 1016 min readUpdated May 2026

Free · sign in to continue

Read the rest of this lesson — free.

The whole course is free. Create an account (GitHub or Google) and read every lesson, top to bottom. You'll land right back here.

← PreviousObservability Next →Asynchronous inference