Llama 3.1 Nemotron Safety Guard 8B NIM Performance
This page is for version 1.14.0 (default).
For other versions, use one of these documentation indexes:
- 1.14.0 (default): https://docs.nvidia.com/nim/llama-3-1-nemotron-safety-guard-8b/1.14.0/llms.txt
- 1.10.1: https://docs.nvidia.com/nim/llama-3-1-nemotron-safety-guard-8b/1.10.1/llms.txt
NVIDIA used the genai-perf tool to benchmark the microservice. See NIM Benchmarking for details.
Some TensorRT-LLM profiles can experience lower throughput under high load compared to generic profiles that use vLLM. See the 1.14.0 known issues for affected profiles.
GH200 480GB
GB200
GH200 144G HBM3E
B200
H200 NVL
H200
H100 80GB HBM3
H100 NVL
A100-SXM4-40GB
A100-SXM4-80GB
A10G
L40S
RTX PRO 6000 BLACKWELL SERVER EDITION
1000_1000
20000_2000
500_2000
5000_500