Tensor parallelism · TP
Splitting a single layer's math across multiple GPUs so they jointly compute one forward/backward pass.
Current numbers
~18×NVLink5 scale-up (1.8 TB/s/GPU bidirectional, ~900 GB/s/dir; 130 TB/s NVL72 rack) over ~400G scale-out NIC (~50 GB/s/dir) — keep TP/EP inside scale-up