Pipeline Parallelism

Pipeline parallelism splits a model along its layer dimension, assigns groups of consecutive layers to separate GPU stages, and feeds micro-batches through the stages in an assembly-line schedule.

Where tensor parallelism shards a single matmul across GPUs, pipeline parallelism shards entire layers — a much coarser cut, with much lighter communication (point-to-point activation sends between adjacent stages, not all-reduces). It scales beyond the NVLink domain comfortably and is the standard tool for spanning a model across nodes when tensor parallelism alone isn’t enough.

How an assembly line maps onto a transformer

A 96-layer model on 8 pipeline stages assigns 12 layers to each stage. A training step splits the global batch into micro-batches that flow through the stages.

• Stage 0 runs micro-batch 1’s forward, sends activations to stage 1, then starts micro-batch 2’s forward.
• Stage 1 runs micro-batch 1 once stage 0’s output arrives.
• After the pipe fills (after steps), every stage works on a different micro-batch concurrently.
• Backward passes run in reverse order; gradients flow stage-by-stage from the loss back to the inputs.

The communication between stages is one activation tensor send forward (and one gradient send back) per micro-batch boundary — typically a few hundred MB on cross-node InfiniBand at 50-100 GB/s. The volume is roughly of what tensor parallelism’s all-reduces would move at the same scale, which is why pipeline parallelism is cross-node-friendly where TP isn’t.

The bubble

The cost is pipeline filling and draining. With stages and micro-batches per step, the bubble fraction is

At and that’s about 10 percent. At and it’s about 50 percent — half your GPUs idle. The fix is more micro-batches (or fewer stages), which means a larger global batch size. This is why pipeline parallelism pairs with large batches and why it doesn’t compose with the small-batch regime — small models trained on a single node should not be using pipeline parallelism.

Stacking with tensor and data parallelism

Modern frontier training composes three parallelism axes simultaneously — “3D parallelism”:

• Tensor parallelism within a node (TP = 8 typical), splitting matmuls across NVLink-connected GPUs.
• Pipeline parallelism across nodes (PP = 4-16), splitting layer groups across InfiniBand-connected nodes.
• Data parallelism across the resulting “model replicas,” replicating the (TP x PP)-sharded model to scale batch size.

For a 405B Llama-class model on 1024 H100s: TP=8 within node, PP=8 across nodes, DP=16 replicas. Each DP replica is one pipeline of 8 stages, each stage is 8 NVLink-coupled GPUs, each layer is split across those 8 via tensor parallelism. Megatron-LM, DeepSpeed, and NeMo all express this composition; the sharding plan is a 3D mapping from logical model parameters to physical (DP-rank, PP-stage, TP-rank) coordinates.

Where the limits are

Pipeline parallelism’s bandwidth requirements are mild — point-to-point activation sends, not all-reduces — so it scales well across slow interconnect. Its memory savings are linear in (each stage holds of layers and of optimizer state), which is what makes 1T+ parameter training tractable on commodity InfiniBand topologies. Its bubble is the binding overhead, and it shrinks the right way as global batch grows. MFU for well-tuned 3D parallelism — TP + PP + DP combined — sits in the 40-50 percent range on H100 clusters in 2026, with bubble accounting for 5-10 percent of the lost ceiling.