Kernel Fusion

Kernel fusion combines what would otherwise be a sequence of GPU kernel launches into a single fused kernel that does all of the work without round-tripping intermediate tensors through HBM. It is the single most important class of GPU performance optimization for LLM workloads, and the principle that explains why FlashAttention is fast, why torch.compile beats eager-mode PyTorch, and why hand-rolled CUDA / Triton kernels still earn their keep in production inference engines.

The premise is mechanical: each kernel launch on a GPU loads its inputs from HBM (GPU main memory), does its compute, and writes its output back to HBM. HBM is roughly an order of magnitude slower than on-chip SRAM and registers. Modern GPUs deliver several hundred FLOPS for every byte loaded from HBM — meaning any compute pattern that produces small intermediate tensors (elementwise ops, small reductions, layer-by-layer activations) is memory-bound, not compute-bound. The compute units sit idle waiting on memory.

Fusion fixes that by keeping intermediate values on-chip across multiple “logical” operations.

What fusion looks like in practice

Take the unfused sequence for a SwiGLU-style FFN block:

1. gate = linear_gate(x) — matmul, write to HBM.
2. up = linear_up(x) — matmul, write to HBM.
3. act = silu(gate) — elementwise, read gate from HBM, write to HBM.
4. mul = act * up — elementwise, read both from HBM, write to HBM.
5. out = linear_down(mul) — matmul, read mul from HBM, write to HBM.

Five kernels, five round-trips. Three of those — the elementwise SiLU, the multiply, and (if you are clever) the bias add inside the down-projection — are pure memory traffic with negligible compute.

A fused SwiGLU kernel does this:

1. Compute gate = linear_gate(x) and up = linear_up(x) into SRAM tiles.
2. While the tile is still in SRAM, apply silu(gate) * up.
3. Stream the result directly into linear_down.

One kernel, one round-trip. Same math, 1.5-2x faster, and the intermediates never touch HBM. Production transformer kernels — Triton-based fused FFNs, fused RMSNorm-residual-attention blocks, NVIDIA TransformerEngine — are all variations on this idea.

Why it matters in inference

In LLM serving, where every microsecond of latency matters, fusion is the table-stakes optimization. vLLM , TensorRT-LLM, and SGLang all ship fused attention, fused FFN, fused norm-and-residual kernels as their defaults. Naive PyTorch attention vs FlashAttention is the headline 2-4x; fused vs unfused FFN is another 1.5x; fused RMSNorm-residual-attention boundaries are another 10-20%. These compose multiplicatively.

Where fusion wins (and where it does not)

Fusion has limits. You cannot fuse across two large matmuls if the intermediate is itself big — there is not enough SRAM to hold it. You cannot fuse if the second op needs the full output of the first (e.g., a softmax needs the full row sum). FlashAttention’s online softmax is exactly the trick that made attention fusable — and it took years of research to find.

How it composes with graph compilation

Hand-writing fused kernels is expensive — Triton, CUDA, careful tile-size tuning. Most teams do not. The dominant production path is to lean on graph compilation (torch.compile, TensorRT-LLM, JAX jit) to fuse for you, and write hand-rolled kernels only for the few hot paths where the compiler leaves significant performance on the table. That covers ~80% of the throughput wins in practice; the remaining ~20% is what FlashAttention / vLLM / TensorRT-LLM make their living on.

Kernel fusion is the lens through which “performance engineering” becomes a coherent discipline rather than a bag of tricks. Every other optimization in the perf-eng topic — graph compilation, mixed precision, FlashAttention, fused optimizers — is, structurally, a different way to fuse more work into fewer HBM round-trips.