Fit one decode replica before buying more HBM — conditional result
About 180 GB resident; 40 ms memory lower bound. At most 9 sequences fit; 10 fail. Below about 3.2 TB/s the 50 ms memory budget fails.derived
| Value kind | derived — Derived values are guide calculations; their result is only as fixed as the stated inputs, scope, and method. |
|---|---|
| Scope | Weights = 70×10⁹ × 2 B = 140 GB. Define K = 2 × 80 × 8 × 128 × 2 B × 8,192, the exact KV bytes per sequence. Eight sequences need about 21 GB of KV; with 20 GB reserve, the resident set is about 180 GB. Capacity screen: floor((186−140−20)×10⁹/K) = 9 sequences. At nine, total residence is below 185 GB; at ten it is above 186 GB (about 187 GB), so the integer decision uses exact tensor counts rather than rounded totals. Reading weights and KV at 4.0 TB/s takes about 40 ms before compute and collectives. The 50 ms bandwidth boundary is (140×10⁹+8K)/0.050, about 3.2 TB/s; the unrounded inequality governs. At 3.0 TB/s memory alone takes about 54 ms and fails. |
| As of | 2026-09 |
| Source | NVIDIA GB200 specifications and CUDA performance memory model — method reference; guide-authored hypothetical scenario September 8, 2026. No supplier quote or test measurement. |
| Derivation | Weights = 70×10⁹ × 2 B = 140 GB. Define K = 2 × 80 × 8 × 128 × 2 B × 8,192, the exact KV bytes per sequence. Eight sequences need about 21 GB of KV; with 20 GB reserve, the resident set is about 180 GB. Capacity screen: floor((186−140−20)×10⁹/K) = 9 sequences. At nine, total residence is below 185 GB; at ten it is above 186 GB (about 187 GB), so the integer decision uses exact tensor counts rather than rounded totals. Reading weights and KV at 4.0 TB/s takes about 40 ms before compute and collectives. The 50 ms bandwidth boundary is (140×10⁹+8K)/0.050, about 3.2 TB/s; the unrounded inequality governs. At 3.0 TB/s memory alone takes about 54 ms and fails. |
| Review | checking…review by 2027-03-08 · standard cadence |
| Recorded changes | last 2026-09-16 |
| Claim id | guide2-t29-7-6-result |
Where the guide uses it
← Full numbers register — every date-stamped figure in the guide, with revision history.