Cluster sizing — model → MW
Size an inference fleet end to end: model parameters, quantization, KV cache and context length → VRAM per replica → active GPUs → installed racks, stranded capacity, facility megawatts, and fractional rental cost at a contract underwriting rate (GB300 NVL72 ~$5.00/GPU-hr, the SemiAnalysis InferenceX 'retail' tier; the $18.00 on-demand list is context only).
Cluster sizing — model to megawatts
The MW figure is for energy and cost, at nominal TDP. Electrical design uses your rack's facility design power and transient peak (GB300 NVL72: 135 kW nominal, up to 142 kW facility basis, 155 kW peak) — three different numbers, never interchangeable → Ch 7.2. VRAM sets only a minimum. Enter a replica size you have tested for your model, runtime, hardware, context and batch size; it need not be a power of two. Purchase quantum is the smallest unit you can buy or contract, and does not follow from rack density. Facility design power follows installed units, while rental cost follows the active fractional fleet. At long context the KV cache can dominate VRAM → Ch 9.7.
Worked decision — stated assumptions
All ledger inputs are assumed teaching values chosen to exercise this chapter’s method; they are not market quotes, OEM performance claims or operating records. Arithmetic uses the full entered precision.
| Input | Value |
|---|---|
| Model size | 70 B params |
| Bytes / param | 1 bytes |
| KV cache / token | 330 KB |
| Context length | 32000 tok |
| Concurrent requests / replica | 64 requests |
| Activation + framework overhead | 15 % |
| HBM per GPU | 288 GB |
| Peak demand | 50000 tok/s |
| Throughput / replica | 11200 tok/s |
| Serving capacity | 70 % |
| GPUs / replica | 8 GPUs |
| GPUs / rack | 72 GPUs |
| Purchase quantum | 72 GPUs |
| GPU TDP | 1400 W |
| Host overhead | 1.3393 × GPU TDP |
| PUE | 1.15 ratio |
| Underwriting rental rate | 18 $/GPU-hr |
Illustrative purchase decision: assume a 70-billion-parameter model at one byte per parameter, 330 KB of KV cache per token, 32,000-token context, 64 concurrent requests and 15% framework overhead. Memory is (70 + 330 × 32,000 × 64 / 1,000,000) × 1.15 = about 858 GB per replica. Assume 288 GB per GPU and an eight-GPU replica you have tested: the three-GPU memory floor does not replace that qualification.
Assume 50,000 output tokens/s peak demand, 11,200 tokens/s per replica and 70% serving capacity. Round 50,000 / (11,200 × 0.70) up to seven replicas: 56 active GPUs. A 72-GPU purchase unit gives 72 installed GPUs, 16 stranded slots and one 72-GPU rack. At an assumed 135 kW nominal rack input and PUE 1.15, the nominal facility screen is about 155 kW. Fractional rental at an assumed $18/GPU-hour costs $1,008/hour for the active fleet.
Assume a $1,100/hour active-fleet rental allowance. The seven-replica case fits. Above 7 × 11,200 × 0.70 = 54,880 output tokens/s, demand needs an eighth replica: at 60,000 tokens/s, 64 active GPUs cost $1,152/hour and exceed the allowance, while the same 72-GPU purchase unit still fits. Acquire the exact serving qualification before committing the purchase.
Guide derivation. Method and handoff: Chapter 1.7 — The Requirements-and-Consequences Matrix.
These are transparent screening estimates, not final designs, financial advice, or project approvals. Replace every default with current project data and have the responsible project authorities validate the result. You can save, share a permalink, or export to CSV; see the full calculator suite.