> This page is for NIM on GKE, version 1.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 1.0.0 (Latest) (default): https://docs.nvidia.com/nim/cloud/gke/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/cloud/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/cloud/_mcp/server.

# Hardware Support for NVIDIA NIM on Google Kubernetes Engine (GKE)

The following are the supported optimized profiles for specific hardware configurations for [NVIDIA NIM](https://www.nvidia.com/en-us/ai/) on Google Kubernetes Engine (GKE).

<table>
<thead>
   <tr>
     <th rowSpan="2">NIM</th>
     <th rowSpan="2">Version</th>
     <th rowSpan="2">Min #GPUs required for NIM</th>
     <th rowSpan="2">GPU</th>
     <th colSpan="4">Supported compute on GCP</th>
   </tr>
   <tr>
     <th>Compute name in config page</th>
     <th>#GPU on instance</th>
     <th>Precision</th>
     <th>Profile</th>
   </tr>
</thead>
<tbody>
   <tr>
     <td>meta/llama3.1-405b-instruct</td>
     <td>1.1.2</td>
     <td>8</td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP8 (trt-llm)</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td rowSpan="5">meta/llama3.1-8b-instruct</td>
     <td rowSpan="5">1.1.2</td>
     <td>1</td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP8 (trt-llm)</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td>2</td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP8 (trt-llm)</td>
     <td>Latency</td>
   </tr>
   <tr>
     <td>1</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-1g</td>
     <td>1</td>
     <td>BF16 (trt-llm)</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td>2</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-2g</td>
     <td>2</td>
     <td>BF16 (trt-llm)</td>
     <td>Latency</td>
   </tr>
   <tr>
     <td>2</td>
     <td>L4</td>
     <td>L4-&lt;region&gt;-g2-standard-24</td>
     <td>2</td>
     <td>FP16 (vllm)</td>
     <td>Non-optimized</td>
   </tr>
   <tr>
     <td rowSpan="5">meta/llama3.1-70b-instruct</td>
     <td rowSpan="5">1.1.2</td>
     <td>4</td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP8 (trt-llm)</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td>8</td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP8 (trt-llm)</td>
     <td>Latency</td>
   </tr>
   <tr>
     <td>4</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-4g</td>
     <td>4</td>
     <td>BF16 (trt-llm)</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td>8</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-8g</td>
     <td>8</td>
     <td>BF16 (trt-llm)</td>
     <td>Latency</td>
   </tr>
   <tr>
     <td>8</td>
     <td>L4</td>
     <td>L4-&lt;region&gt;-g2-standard-96</td>
     <td>8</td>
     <td>FP16 (vllm)</td>
     <td>Non-optimized</td>
   </tr>
   <tr>
     <td rowSpan="4">meta/llama3-70b-instruct</td>
     <td rowSpan="4">1.0.3</td>
     <td>4 </td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP8 (trt-llm)</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td>8</td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP8 (trt-llm)</td>
     <td>Latency</td>
   </tr>
   <tr>
     <td>4</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-4g</td>
     <td>4</td>
     <td>FP16</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td>8</td>
     <td>L4</td>
     <td>L4-&lt;region&gt;-g2-standard-96</td>
     <td>8</td>
     <td>FP16 (vllm)</td>
     <td>Non-optimized</td>
   </tr>
   <tr>
     <td rowSpan="5">meta/llama3-8b-instruct</td>
     <td rowSpan="5">1.0.3</td>
     <td>1</td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP16</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td>2</td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP16</td>
     <td>Latency</td>
   </tr>
   <tr>
     <td>1</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-1g</td>
     <td>1</td>
     <td>FP16</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td>2</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-2g</td>
     <td>2</td>
     <td>FP16</td>
     <td>Latency</td>
   </tr>
   <tr>
     <td>2</td>
     <td>L4</td>
     <td>L4-&lt;region&gt;-g2-standard-24</td>
     <td>2</td>
     <td>FP16 (vllm)</td>
     <td>Non-optimized</td>
   </tr>
   <tr>
     <td rowSpan="5">mistralai/mistral-7b-instruct-v.03</td>
     <td rowSpan="5">1.0.3</td>
     <td>1</td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP8 (trt-llm)</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td>2</td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP8 (trt-llm)</td>
     <td>Latency</td>
   </tr>
   <tr>
     <td>1</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-1g</td>
     <td>1</td>
     <td>FP16 (trt-llm)</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td>2</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-2g</td>
     <td>2</td>
     <td>FP16 (trt-llm)</td>
     <td>Latency</td>
   </tr>
   <tr>
     <td>4</td>
     <td>L4</td>
     <td>L4-&lt;region&gt;-g2-standard-48</td>
     <td>4</td>
     <td>FP16 (vllm)</td>
     <td>Non-optimized</td>
   </tr>
   <tr>
     <td rowSpan="4">mistralai/mixtral-8x7b-instruct-v0.1</td>
     <td rowSpan="4">1.0.0</td>
     <td>2 </td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8 </td>
     <td>8</td>
     <td>FP8 (trt-llm)</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td>4</td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP8 (trt-llm)</td>
     <td>Latency</td>
   </tr>
   <tr>
     <td>2</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-2&nbsp;&nbsp;&nbsp;g</td>
     <td>2</td>
     <td>FP16 (trt-llm)</td>
     <td>Throughput</td>
   </tr>
   <tr>
     <td>4</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-4g</td>
     <td>4</td>
     <td>FP16 (trt-llm)</td>
     <td>Latency</td>
   </tr>
   <tr>
     <td rowSpan="3">nvidia/nv-rerankqa-mistral-4b-v3</td>
     <td rowSpan="3">1.0.2</td>
     <td>1 </td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP16</td>
     <td> </td>
   </tr>
   <tr>
     <td>1</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-1g</td>
     <td>1</td>
     <td>FP16</td>
     <td> </td>
   </tr>
   <tr>
     <td>2</td>
     <td>L4</td>
     <td>L4-&lt;region&gt;-g2-standard-24</td>
     <td>2</td>
     <td>FP16</td>
     <td> <br /></td>
   </tr>
   <tr>
     <td rowSpan="3">nvidia/nv-embedqa-e5-v5</td>
     <td rowSpan="3">1.0.1</td>
     <td>1 </td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP16</td>
     <td> </td>
   </tr>
   <tr>
     <td>1</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-1g</td>
     <td>1</td>
     <td>FP16</td>
     <td> </td>
   </tr>
   <tr>
     <td>2</td>
     <td>L4</td>
     <td>L4-&lt;region&gt;-g2-standard-24</td>
     <td>2</td>
     <td>FP16</td>
     <td> <br /></td>
   </tr>
   <tr>
     <td rowSpan="3">nvidia/nv-embedqa-mistral-7b-v2</td>
     <td rowSpan="3">1.0.1</td>
     <td>1 </td>
     <td>H100 (80GB)</td>
     <td>H100(80GB)-&lt;region&gt;-a3-highgpu-8g</td>
     <td>8</td>
     <td>FP8</td>
     <td> </td>
   </tr>
   <tr>
     <td>1</td>
     <td>A100 (80GB)</td>
     <td>A100(80GB)-&lt;region&gt;-a2-ultragpu-1g</td>
     <td>1</td>
     <td>FP16</td>
     <td> </td>
   </tr>
   <tr>
     <td>2</td>
     <td>L4</td>
     <td>L4-&lt;region&gt;-g2-standard-24</td>
     <td>2</td>
     <td>FP16</td>
     <td> &nbsp;&nbsp;&nbsp;<br /></td>
   </tr>
</tbody>
</table>