How LLMs Generate Text — Inference Pipeline

The full LLM inference pipeline in one sketchnote — tokenizer, embeddings, transformer blocks, KV cache, LM head, sampling, speculative decoding, and streaming output. Prefill vs decode explained.

1 · Input

User prompt enters the runtime

2 · Tokenizer

BPE splits text → token IDs (not word-by-word)

3 · Embedding Layer

Step 3: IDs → embeddings

4 · Transformer Block × N (×96 Gpt-4 Class)

Attention + KV cache + FFN × N layers (×96 GPT-4 class)