Skip to navigation

Llama 3.1 Nemotron Safety Guard 8B NIM Performance

View as Markdown

NVIDIA used the genai-perf tool to benchmark the microservice. See NIM Benchmarking for details.

Some TensorRT-LLM profiles can experience lower throughput under high load compared to generic profiles that use vLLM. See the 1.14.0 known issues for affected profiles.

ConcurrencyTTFT (ms)Throughput (inputs/s)
115.6545895165.8306125
100178.78854337590.530806
250982.9970637152.400699