Skip to content
MUSEBOARD

GPU / H100-SXM

NVIDIA H100 SXM 80GB

Large-scale training and high-throughput serving

Plan on this GPU Compare All GPUs
Spec sheet
NVIDIA
Vendor
NVIDIA
Architecture
Hopper
Memory
80 GB HBM3
Bandwidth
3.35 TB/s
Power
700 W
Class
Datacenter
Form factor
SXM5
Interconnect
NVLink 4 · 900 GB/s
Released
2022
Workloads
Inference · Fine-tuning · Training
Model fit / Inference
8K ctx · batch 1 · ≥10% headroom
GPUs required per model and precision for NVIDIA H100 SXM 80GB
ModelFP16INT8INT4
Llama 3.2 1B1×3.0 GB1×1.6 GB1×0.95 GB
Llama 3.2 3B1×8.0 GB1×4.5 GB1×2.7 GB
Llama 3.1 8B1×18.7 GB1×9.9 GB1×5.5 GB
Llama 3.1 70B4×158.0 GB2×80.3 GB1×41.5 GB
Llama 3.3 70B4×158.0 GB2×80.3 GB1×41.5 GB
Llama 3.1 405B16×897.2 GB8×450.7 GB4×227.5 GB
Qwen2.5 1.5B1×3.6 GB1×1.9 GB1×1.1 GB
Qwen2.5 3B1×7.1 GB1×3.7 GB1×2.0 GB
Qwen2.5 7B1×17.2 GB1×8.9 GB1×4.7 GB
Qwen2.5 14B1×34.0 GB1×17.8 GB1×9.7 GB
Qwen2.5 32B2×73.6 GB1×37.9 GB1×20.0 GB
Qwen2.5 72B4×162.6 GB2×82.7 GB1×42.7 GB
Mistral 7B1×17.0 GB1×9.0 GB1×5.1 GB
Mistral NeMo 12B1×28.2 GB1×14.8 GB1×8.1 GB
Mistral Small 24B1×53.3 GB1×27.3 GB1×14.3 GB
Mixtral 8x7B2×103.8 GB1×52.4 GB1×26.8 GB
Mixtral 8x22B8×312.1 GB4×157.0 GB2×79.4 GB
Gemma 2 9B1×23.1 GB1×13.0 GB1×7.9 GB
Gemma 2 27B1×62.9 GB1×33.0 GB1×18.0 GB
Phi-3 Mini 3.8B1×11.6 GB1×7.4 GB1×5.3 GB
Phi-3 Medium 14B1×32.5 GB1×17.1 GB1×9.4 GB
DeepSeek R1 Distill Qwen 32B2×74.3 GB1×38.2 GB1×20.2 GB
DeepSeek R1 Distill Llama 70B4×158.0 GB2×80.3 GB1×41.5 GB
DeepSeek V3 671B24×1,477 GB16×738.7 GB8×369.6 GB