Support Matrix | Llama 3.1 Nemotron Safety Guard 8B NIM

Software

CUDA 12.9 is configured in the microservice container.
NVIDIA GPU Driver version 525 or higher. NVIDIA verified the documentation using 570.133.20.
A container runtime environment such as Docker or Kubernetes. For Docker, see the Docker installation instructions.
NVIDIA Container Toolkit installed and configured. See the installation guide.
An active NVIDIA AI Enterprise subscription or membership in the NVIDIA Developer Program. Access to the container and model is restricted.
An NGC API key to download the container and model.

About Model Profiles

NVIDIA NIM microservices use model engines tuned for specific GPU models, GPU counts, precision, and other settings. These combinations are called model profiles. Each profile has a unique 64-character hexadecimal profile ID. The NIM can select a profile automatically based on available GPUs or use a profile ID that you specify.

The model manifest is stored at /opt/nim/etc/default/model_manifest.yaml in the container.

NVIDIA Llama 3.1 Nemotron Safety Guard 8B NIM Model Profiles

The model requires 48 GB of GPU memory. NVIDIA developed and tested it with these GPUs:

B200 SXM
H200 SXM
H100 SXM
A100 SXM 40 GB and 80 GB
A100 PCIe 40 GB and 80 GB (generic profiles only)
A10G
L40S PCIe
H100 NVL
H200 NVL
GB200 NVL72
GH200 NVL2
GH200 96 GB
RTX PRO 6000 Blackwell Server Edition

You can use one GPU with the required memory capacity or multiple GPUs that meet the capacity. For details about locally buildable and generic profiles, see NIM for LLMs model profiles.

The table lists the model profiles in the 1.14.0 manifest. Profile IDs select an exact profile.

GPU	Precision	Engine	Profile	GPUs	Tensor parallelism	Profile ID
B200	FP8	TensorRT-LLM	throughput	1	1	`8258db301f5fb0a5b5a5b9c263af9334ffc0714c6ae82d2a0687b95c11f5834e`
B200	FP8	TensorRT-LLM	latency	2	2	`1c67491281ac66f32ca917bc566808bf4657ad20dec137f2b40c78d95b3a40dd`
B200	BF16	TensorRT-LLM	throughput	1	1	`d9bab56236ea000a0e0760e76d2530d7e460dde85cb6487b496009cbbb9ea7b2`
B200	BF16	TensorRT-LLM	latency	2	2	`7b2460a744467de931be74543f3b1be6a4540edd8f5d3523c31aca253d3ee714`
GB200	FP8	TensorRT-LLM	throughput	1	1	`45999aa05eca849f11beb20bbf70dd48f2970a01dcf4c2617302b8038fca1662`
GB200	FP8	TensorRT-LLM	latency	2	2	`cfca3a90be399e2fc6b91dfe19aa141fe7db0ad114df365cf43d77c675711049`
GB200	BF16	TensorRT-LLM	throughput	1	1	`20ac8ad6d3cb6b083facc320669292ab06972015f432839c64c841494beb4177`
GB200	BF16	TensorRT-LLM	latency	2	2	`3b86e1c4eafac6dd59612ed5cea6878177867773d567fcc0e0127ad5b2b1221b`
GH200 144GB	FP8	TensorRT-LLM	throughput	1	1	`c53b2566c86bf5ca53d25883b3f305f7fcfa1b8a854a52bf21d6901fec27fd3f`
GH200 144GB	FP8	TensorRT-LLM	latency	2	2	`052a14156d375521d64c584a0197a00ab3c54ae742b55145f1df091072656de7`
GH200 144GB	BF16	TensorRT-LLM	throughput	1	1	`8f0e0665f4f53a02e2e68ab690ba990d5d9a3a6b49cdcd4ff679031757de30ed`
GH200 144GB	BF16	TensorRT-LLM	latency	2	2	`542edd6068b3fee7bb9431ba988f167dfc9f6e5b6dbf005b2e208d42bd17d705`
GH200 480GB	FP8	TensorRT-LLM	throughput	1	1	`8534f70d957abfcbb4353276dcb754f0a0820e90f555afe233be835f8d5c9743`
GH200 480GB	FP8	TensorRT-LLM	latency	1	1	`9784974c7d8bd2e2081fc751d02a70bd619b1b16d535d3ff764622c1b7e73ff2`
GH200 480GB	BF16	TensorRT-LLM	throughput	1	1	`076e5a3eec39670d90ef75725966fabc895074660b42a0ba3edce65c0eb15756`
GH200 480GB	BF16	TensorRT-LLM	latency	1	1	`f41c136a67469ae0bda89f96d78cb8c2b9c01c27d0ac618112248025320817c3`
H200 NVL	FP8	TensorRT-LLM	throughput	1	1	`7a4c46a6911fff7e11e99df87161eeae73b08c48cc75f1928e8289fbacc188fc`
H200 NVL	FP8	TensorRT-LLM	latency	2	2	`373b366ba1d0bf15e9c7087967fadcc00db30c079943feee2ae0e2489aee9d66`
H200 NVL	BF16	TensorRT-LLM	throughput	1	1	`58041a8daba4de24b75c86456d256fbc34e881de58ebdab6135aad18466d8130`
H200 NVL	BF16	TensorRT-LLM	latency	2	2	`dbb457d9b5a45d0a6976c0ba1a8ee6072deb8fe64c49a12e47ba9c71863618d2`
H200	FP8	TensorRT-LLM	throughput	1	1	`908a7d5dbabe5e1a7956b66d90747b34d44edb357546ab098474efc28a1f95f2`
H200	FP8	TensorRT-LLM	latency	2	2	`5a708fe91514e2aa44438158af955b35d51fab4ca1fb7268e35930e67fce6e08`
H200	BF16	TensorRT-LLM	throughput	1	1	`1b953829736de1f2ffc00c8d25316a24d43ed398b5f8599957c9547007fd5471`
H200	BF16	TensorRT-LLM	latency	2	2	`158a13eff79873eb73689daf87c365fa06946f74856646e54edc69728ef59a8e`
H100 NVL	FP8	TensorRT-LLM	throughput	1	1	`f3fd46462fe31ea44e5207d9f97ea190d8928b5688f5752445bb7dfd8afc999c`
H100 NVL	FP8	TensorRT-LLM	latency	2	2	`5e8a78e4d0c9e2e513466ec23ac181ae8d75ce05bda5c4653eddf8f3a99f2d58`
H100 NVL	BF16	TensorRT-LLM	throughput	1	1	`dbf1e0e5f43bcabfdd0791b0ffa83036fd7cf6be28fd5266990cf94bb8ffc338`
H100 NVL	BF16	TensorRT-LLM	latency	2	2	`a91cc87e8e98c7e88967319c273392e447fab72dd22aa8231630b573284525b2`
H100	FP8	TensorRT-LLM	throughput	1	1	`21879fb81cb016c598cb4324a7cfa93acac1904ebc8cbfd9fff42d79a929b19d`
H100	FP8	TensorRT-LLM	latency	2	2	`3a8d53337b395483663f3d96e4528a759d1474c5afa8b68d74a39e99fad7aebf`
H100	BF16	TensorRT-LLM	throughput	1	1	`dfe2b2518ce82cb404a639f9039d956ec1b8f35703e7594d066ef080d1172156`
H100	BF16	TensorRT-LLM	latency	2	2	`2f69689bf8fef4118bb018bb07869fc2d4b6eb3185115b2117ad62150f5d0006`
A100	BF16	TensorRT-LLM	throughput	1	1	`8125bb2032b5649b5152192498543b77543de174d39f18dfb7473a6332c9b3fd`
A100	BF16	TensorRT-LLM	latency	2	2	`f5a41523d4dd3de5340276b18dfcbce17bb047b2457a36c1310e61589cc28935`
A100 SXM4 40GB	BF16	TensorRT-LLM	throughput	1	1	`3f24e5c0ba016486c057ff9997caa9341f1f974f5dbf53e96fdbf9a0558bc9ab`
A100 SXM4 40GB	BF16	TensorRT-LLM	latency	2	2	`bd225e66799ed1e826ae5e2d7db017e1ba9a73855807e4f362967f22b2d7fecb`
A10G	BF16	TensorRT-LLM	throughput	4	4	`105f6654f47c50206ac6dc9662aa774e1d8bad4b7268e1db3358ac0e446d9abc`
A10G	BF16	TensorRT-LLM	latency	8	8	`80b75e35feecbcc4f3cc33ea634318bff503afcea1924a140f9fe9c3a848b99c`
L40S	FP8	TensorRT-LLM	throughput	1	1	`032ae037e7cddf2e43f37341027d8060e5eb5082797ed3e39d803f1c6af4eb84`
L40S	FP8	TensorRT-LLM	latency	2	2	`f282d4039fc42e3ab8a69854daf1a3a9e0fdce7974d06c3924969e3196e4ac08`
L40S	BF16	TensorRT-LLM	throughput	1	1	`7dedab87edfe691a1eb95d7bdc3fa74cc3e7fb6eeb6b56ab933f2ea3561d4c8d`
L40S	BF16	TensorRT-LLM	latency	2	2	`260a963986f4b26d55e2af38ef19fd45341314564cc55f6b3f968e0323a8daea`
RTX PRO 6000 Blackwell Server Edition	FP8	TensorRT-LLM	latency	1	1	`9325fdaeebf47da133de2bffcbfdcfb7537de5f0cf72adb091fb5bb8429047ad`
RTX PRO 6000 Blackwell Server Edition	FP8	TensorRT-LLM	throughput	1	1	`b5938999f25208fbf5ef64ad1ce0d3978ac52d5b18d32f9b7e1551cbc97737db`
RTX PRO 6000 Blackwell Server Edition	BF16	TensorRT-LLM	throughput	1	1	`86edc198bd7a53981bd95f612e07e204727e116bdf779f1405e0c8ff92e0f067`
RTX PRO 6000 Blackwell Server Edition	BF16	TensorRT-LLM	latency	2	2	`5f2aea268787b1aea54e98d370f99685f2d6066310615ff946cf4f76242dcbfc`
Generic	BF16	vLLM	Generic	1	1	`47804570e8db67fdc60d9cb518b0f60316ac4eb0f3a55af81aafefe124505a8f`
Generic	BF16	vLLM	Generic	2	2	`60cc64256f69a6242aa2b7320a1070ae84e88fe5416674cbbe9a2cb6ecb8e992`
Generic	BF16	vLLM	Generic	4	4	`511022fb9a93b2c8b1eb84f4c472f05833e6904e51de7fd676be168a0aa9d63d`
Generic	BF16	vLLM	Generic	8	8	`a98cb8c244d2fd95e65a102758e4b039c94864f322c5b64bc9ee5749ddac59ab`