Skip to content
MUSEBOARD

GPU / A100-80GB

NVIDIA A100 80GB

Training and serving of 7B–70B models

Plan on this GPU Compare All GPUs
Spec sheet
NVIDIA
Vendor
NVIDIA
Architecture
Ampere
Memory
80 GB HBM2e
Bandwidth
2.04 TB/s
Power
400 W
Class
Datacenter
Form factor
SXM4
Interconnect
NVLink 3 · 600 GB/s
Released
2020
Workloads
Inference · Fine-tuning · Training
Model fit / Inference
8K ctx · batch 1 · ≥10% headroom
GPUs required per model and precision for NVIDIA A100 80GB
ModelFP16INT8INT4
Llama 3.2 1B1×3.0 GB1×1.6 GB1×0.95 GB
Llama 3.2 3B1×8.0 GB1×4.5 GB1×2.7 GB
Llama 3.1 8B1×18.7 GB1×9.9 GB1×5.5 GB
Llama 3.1 70B4×158.0 GB2×80.3 GB1×41.5 GB
Llama 3.3 70B4×158.0 GB2×80.3 GB1×41.5 GB
Llama 3.1 405B16×897.2 GB8×450.7 GB4×227.5 GB
Qwen2.5 1.5B1×3.6 GB1×1.9 GB1×1.1 GB
Qwen2.5 3B1×7.1 GB1×3.7 GB1×2.0 GB
Qwen2.5 7B1×17.2 GB1×8.9 GB1×4.7 GB
Qwen2.5 14B1×34.0 GB1×17.8 GB1×9.7 GB
Qwen2.5 32B2×73.6 GB1×37.9 GB1×20.0 GB
Qwen2.5 72B4×162.6 GB2×82.7 GB1×42.7 GB
Mistral 7B1×17.0 GB1×9.0 GB1×5.1 GB
Mistral NeMo 12B1×28.2 GB1×14.8 GB1×8.1 GB
Mistral Small 24B1×53.3 GB1×27.3 GB1×14.3 GB
Mixtral 8x7B2×103.8 GB1×52.4 GB1×26.8 GB
Mixtral 8x22B8×312.1 GB4×157.0 GB2×79.4 GB
Gemma 2 9B1×23.1 GB1×13.0 GB1×7.9 GB
Gemma 2 27B1×62.9 GB1×33.0 GB1×18.0 GB
Phi-3 Mini 3.8B1×11.6 GB1×7.4 GB1×5.3 GB
Phi-3 Medium 14B1×32.5 GB1×17.1 GB1×9.4 GB
DeepSeek R1 Distill Qwen 32B2×74.3 GB1×38.2 GB1×20.2 GB
DeepSeek R1 Distill Llama 70B4×158.0 GB2×80.3 GB1×41.5 GB
DeepSeek V3 671B24×1,477 GB16×738.7 GB8×369.6 GB