> This page is for LLM, version 1.0.0. > For other versions, use one of these documentation indexes: > - 2.0.0 (Latest) (default): https://docs.nvidia.com/nim/benchmarking/llm/2.0.0/llms.txt > - 1.0.0: https://docs.nvidia.com/nim/benchmarking/llm/1.0.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nim/benchmarking/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/benchmarking/_mcp/server. # Performance This section shows the latency and throughput numbers for Llama models powered by NVIDIA NIM. Please see [Using AIPerf to Benchmark](/step-by-step) for the benchmark process. For specifications of the hardware on which these measurements were collected, see the [Hardware Specifications](#hardware-specifications) section. ## Llama-3.3-70b-instruct Results * NIM Container: [Llama-3.3-70b-Instruct](https://catalog.ngc.nvidia.com/orgs/nim/teams/meta/containers/llama-3.3-70b-instruct/tags) ### Version: 1.8.0 #### 2 x H100 80G #### fp8 TP2 #### 5000 Input Tokens / 500 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 406.77 | 18.58 | 51.67 | | 5 | 546.7 | 21.83 | 218.48 | | 25 | 688.69 | 38.75 | 623.4 | | 50 | 834.51 | 59.73 | 814.37 | | 100 | 7996.25 | 92.91 | 917.26 | | 150 | 34794.22 | 92.86 | 920.9 | | 200 | 61680.22 | 93.06 | 920.54 | | 250 | 88605.32 | 93.2 | 920.12 | #### 500 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 47.77 | 18.91 | 52.85 | | 5 | 126.53 | 19.04 | 261.83 | | 25 | 147.63 | 21.69 | 1149.05 | | 50 | 181.64 | 23.97 | 2078.43 | | 100 | 229.77 | 29.98 | 3322.87 | | 150 | 296.74 | 37.85 | 3948.26 | | 200 | 4867.89 | 40.72 | 4624.14 | #### 200 Input Tokens / 200 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 31.22 | 18.8 | 53 | | 5 | 88.7 | 18.83 | 260.67 | | 25 | 138.98 | 20.61 | 1178.68 | | 50 | 156.28 | 23.33 | 2081.24 | | 100 | 176.8 | 29.87 | 3263.31 | | 150 | 195.51 | 37.75 | 3883.1 | | 200 | 253.36 | 42.95 | 4534.54 | | 250 | 264.22 | 48.16 | 5067.05 | #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 82.46 | 18.67 | 53.37 | | 5 | 145.62 | 19.07 | 260.48 | | 25 | 250.49 | 22.45 | 1102.23 | | 50 | 278.62 | 25.87 | 1912.93 | | 100 | 334.23 | 34.21 | 2895.49 | | 150 | 428.54 | 44.44 | 3342.67 | | 200 | 558.23 | 50.66 | 3901.74 | | 250 | 3499.83 | 56.3 | 4167.39 | #### 20000 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 1833.01 | 19.58 | 48.8 | | 5 | 2910.33 | 25.24 | 186.8 | | 25 | 18781.26 | 48.26 | 415.33 | | 50 | 123969.71 | 49.22 | 418.65 | | 100 | 333564.85 | 49.81 | 425.1 | | 150 | 543226.4 | 50.01 | 423.95 | | 200 | 753417.03 | 50.06 | 425.74 | | 250 | 963222.96 | 50.13 | 427.06 | #### 2 x H200 141GB #### fp8 TP2 #### 5000 Input Tokens / 500 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 418.61 | 16.72 | 57.05 | | 5 | 567.86 | 19.15 | 246.86 | | 25 | 708 | 35.86 | 671.12 | | 50 | 841.35 | 56.65 | 857.45 | | 100 | 5701.4 | 91.4 | 972.2 | | 150 | 31108.04 | 91.64 | 972.92 | | 200 | 56556.61 | 91.81 | 972.7 | | 250 | 82015.69 | 91.91 | 972.59 | #### 500 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 46.38 | 16.79 | 59.51 | | 5 | 121.67 | 16.36 | 304.63 | | 25 | 169.85 | 18.64 | 1335.79 | | 50 | 190.84 | 19.79 | 2515.16 | | 100 | 238.18 | 26.06 | 3820.26 | | 150 | 302.88 | 33.98 | 4395.18 | | 200 | 357.1 | 37.06 | 5370.23 | | 250 | 576.85 | 39.99 | 6202.02 | #### 200 Input Tokens / 200 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 29.58 | 16.85 | 59.12 | | 5 | 96.02 | 16.41 | 297.51 | | 25 | 144.09 | 18.03 | 1339.22 | | 50 | 157.84 | 20.17 | 2391.63 | | 100 | 187.07 | 27.57 | 3520.02 | | 150 | 201.75 | 36.76 | 3983.17 | | 200 | 258.88 | 42.13 | 4617.01 | | 250 | 294.21 | 47 | 5172.59 | #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 81.45 | 16.72 | 59.56 | | 5 | 147.1 | 16.48 | 301.06 | | 25 | 252.61 | 19.38 | 1274.05 | | 50 | 278.5 | 21.84 | 2261.37 | | 100 | 332.18 | 30.22 | 3273.32 | | 150 | 423.88 | 41.08 | 3614.04 | | 200 | 582.43 | 46.43 | 4251.5 | | 250 | 666.7 | 52.75 | 4676.16 | #### 20000 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 1892.93 | 17.44 | 54.38 | | 5 | 2981.43 | 21.61 | 215.84 | | 25 | 7173.02 | 44.36 | 516.05 | | 50 | 12866.29 | 72.84 | 621.2 | | 100 | 145188.49 | 81.99 | 615.1 | | 150 | 288488.94 | 82.99 | 621.73 | | 200 | 432638.5 | 83.58 | 620.62 | | 250 | 576444.85 | 84.03 | 620.09 | #### bf16 TP2 #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 81.66 | 16.12 | 61.78 | | 5 | 306.56 | 15.63 | 314.03 | | 25 | 894.46 | 18.35 | 1299.94 | | 50 | 1067.36 | 20.75 | 2293.14 | | 100 | 1158.71 | 29.61 | 3252.32 | | 150 | 1341.05 | 41.04 | 3541.71 | | 200 | 1283.61 | 46.78 | 4164.44 | | 250 | 1808.22 | 51.77 | 4669.86 | #### 4 x H100 80GB #### fp8 TP4 #### 5000 Input Tokens / 500 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 214.63 | 13.89 | 69.96 | | 5 | 640.52 | 14.34 | 320.72 | | 25 | 592.2 | 24.78 | 964.48 | | 50 | 661.04 | 36.53 | 1322.8 | | 100 | 857.04 | 61.36 | 1587.86 | | 150 | 1098.38 | 88 | 1665.56 | | 200 | 1149.56 | 111.46 | 1760.61 | | 250 | 6176.07 | 127.61 | 1785.89 | #### 500 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 32.41 | 13.85 | 72.17 | | 5 | 93.74 | 12.93 | 385.51 | | 25 | 356.42 | 14.19 | 1740.17 | | 50 | 492.05 | 15.62 | 3153.24 | | 100 | 566.05 | 19.44 | 5071.9 | | 150 | 707 | 24.7 | 5988.23 | | 200 | 799.72 | 26.52 | 7430.1 | | 250 | 1152.75 | 28.57 | 8579.87 | #### 200 Input Tokens / 200 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 22.73 | 13.8 | 72.22 | | 5 | 56.59 | 13.44 | 366.12 | | 25 | 160.64 | 13.83 | 1716.65 | | 50 | 266.81 | 14.83 | 3104.67 | | 100 | 370.92 | 18.82 | 4849.98 | | 150 | 423.34 | 24.16 | 5718.54 | | 200 | 639.02 | 25.32 | 7031.28 | | 250 | 890.53 | 27.6 | 7803.07 | #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 51.01 | 13.99 | 71.26 | | 5 | 170.81 | 13.06 | 378.3 | | 25 | 503.84 | 14.76 | 1639.43 | | 50 | 608.55 | 16.86 | 2864.92 | | 100 | 613.39 | 22.02 | 4421.55 | | 150 | 743.73 | 28.79 | 5082.9 | | 200 | 955.06 | 31.74 | 6120.83 | | 250 | 1405.48 | 34.75 | 6918.63 | #### 20000 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 941.17 | 14.1 | 68.68 | | 5 | 2191.86 | 16.22 | 288.76 | | 25 | 4157.12 | 31.48 | 744.05 | | 50 | 6205.6 | 50.33 | 933.97 | | 100 | 84784.81 | 57.29 | 958.23 | | 150 | 176553.42 | 58.23 | 966.23 | | 200 | 269826.81 | 58.42 | 957.89 | | 250 | 362546.32 | 58.52 | 963.45 | #### bf16 TP4 #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 78.59 | 15.49 | 64.3 | | 5 | 292.54 | 16.74 | 293.91 | | 25 | 893.19 | 20.19 | 1186.59 | | 50 | 996.37 | 24.49 | 1963.32 | | 100 | 1091.68 | 33.11 | 2926.44 | | 150 | 1204.74 | 41.27 | 3534.23 | | 200 | 1249.37 | 48.75 | 4002.72 | | 250 | 5689.49 | 54.54 | 4133.74 | #### 8 x A100 80GB #### bf16 TP8 #### 5000 Input Tokens / 500 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 554.81 | 18.54 | 50.99 | | 5 | 1005.5 | 23.9 | 193.29 | | 25 | 1362.44 | 52 | 457.61 | | 50 | 1723.18 | 82.19 | 584.77 | | 100 | 2257.5 | 146.67 | 662.51 | | 150 | 2644.06 | 213.3 | 687.31 | #### 500 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 79.98 | 18.57 | 53.75 | | 5 | 259.87 | 19.88 | 249.99 | | 25 | 947.31 | 24.89 | 985.94 | | 50 | 1167.07 | 28.75 | 1705.2 | | 100 | 1458.45 | 39.83 | 2465.98 | | 150 | 1513.77 | 53.21 | 2780.56 | #### 200 Input Tokens / 200 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 47.73 | 18.11 | 54.77 | | 5 | 135.15 | 19.63 | 247.48 | | 25 | 528.55 | 23.47 | 961.89 | | 50 | 848.5 | 26.59 | 1629.45 | | 100 | 1208.62 | 38.47 | 2257.04 | | 150 | 1381.12 | 53.42 | 2498.15 | | 200 | 1415.91 | 61.44 | 2932.47 | | 250 | 1491.94 | 69.03 | 3283.87 | #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 131.21 | 18.44 | 53.91 | | 5 | 460.71 | 19.83 | 246.66 | | 25 | 1345.99 | 26.36 | 903 | | 50 | 1498.8 | 31.6 | 1512.02 | | 100 | 1625.47 | 46.47 | 2080.95 | | 150 | 1729.4 | 63.28 | 2309.02 | | 200 | 1680.2 | 72.79 | 2687.61 | | 250 | 1884.04 | 82.1 | 2979.05 | #### 20000 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 2521.02 | 19.46 | 48.27 | | 5 | 5331.09 | 26.89 | 169.17 | | 25 | 10457.52 | 64.46 | 358.42 | | 50 | 16794.16 | 107.78 | 429.76 | | 100 | 178225.46 | 127.17 | 441.63 | | 150 | 379534.42 | 128.79 | 444.7 | | 200 | 578423.34 | 129.63 | 446.76 | | 250 | 778536.81 | 130.36 | 447.46 | #### 8 x H100 80GB #### bf16 TP8 #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 57.63 | 10.25 | 97.06 | | 5 | 175.85 | 11.3 | 436.08 | | 25 | 536.06 | 14.51 | 1663.36 | | 50 | 602.94 | 18.33 | 2642.87 | | 100 | 655.12 | 25.67 | 3801.4 | | 150 | 687.9 | 31.68 | 4637.51 | | 200 | 688.09 | 36.23 | 5421.19 | | 250 | 795.95 | 40.31 | 6085.75 | ### Version: 1.5.0 #### 4 x H100 80G #### fp16 TP4 #### 5000 Input Tokens / 500 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 368.11 | 19.94 | 48.44 | | 5 | 446.68 | 23.89 | 199.17 | | 25 | 751.74 | 42.12 | 564.5 | | 50 | 1083.13 | 63.72 | 738.78 | | 100 | 17754.93 | 87.55 | 777.31 | | 150 | 46381.93 | 87.39 | 777.94 | | 200 | 73326.6 | 87.64 | 777.02 | | | | | | #### 500 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 71.82 | 19.26 | 51.84 | | 5 | 109.62 | 20.04 | 237.72 | | 25 | 159.62 | 23.28 | 1032.56 | | 50 | 232.46 | 25.58 | 1880.75 | | 100 | 293.19 | 32.83 | 2914.64 | | 150 | 448.16 | 45.57 | 3096.87 | | | | | | | | | | | #### 200 Input Tokens / 200 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 32.78 | 19.11 | 52.15 | | 5 | 81.17 | 19.9 | 247.26 | | 25 | 247.81 | 21.99 | 1080.83 | | 50 | 377.42 | 23.94 | 1936.88 | | 100 | 583.54 | 31.79 | 2886.57 | | 150 | 831.88 | 44.81 | 3067.43 | | 200 | 925.3 | 45.92 | 3947.82 | | 250 | 1105.17 | 49.83 | 4521.68 | #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 103.2 | 19.31 | 51.54 | | 5 | 125.56 | 20.52 | 234.76 | | 25 | 195.68 | 24.76 | 979.58 | | 50 | 243.19 | 28.55 | 1706.26 | | 100 | 344.86 | 38.65 | 2507.94 | | 150 | 568.88 | 54.06 | 2663.03 | | 200 | 1384.9 | 58.11 | 3238.57 | | | | | | #### 2500 Input Tokens / 300 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 190.36 | 19.6 | 49.57 | | 5 | 430.84 | 21.64 | 216.96 | | 25 | 870.9 | 35 | 654.15 | | 50 | 1066.1 | 51.56 | 903.17 | | 100 | 1332.94 | 88.91 | 1062.3 | | 150 | 3853.57 | 124.93 | 1070.78 | | 200 | 17105.6 | 125.13 | 1064.79 | | 250 | 30309.49 | 125.27 | 1073.5 | #### 2000 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 1637.36 | 20.5 | 46.92 | | 5 | 3126.01 | 27 | 175.65 | | 25 | 33539.02 | 52.4 | 323.8 | | 50 | 127678.39 | 51.08 | 335.75 | | 100 | 277353.39 | 48.51 | 345.84 | | 150 | 339462.09 | 48.51 | 342.77 | | 200 | 336890.58 | 51.03 | 338.3 | | 250 | 515613.11 | 49.4 | 349.04 | ## Llama-3.1-8b-instruct Results * NIM Container: [Llama-3.1-8b-Instruct](https://catalog.ngc.nvidia.com/orgs/nim/teams/meta/containers/llama-3.1-8b-instruct) ### Version: 1.8.0 #### 1 x H100 80G #### fp8 TP1 #### 200 Input Tokens / 200 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 10.26 | 4.63 | 214.56 | | 5 | 16.81 | 4.58 | 1076.88 | | 25 | 36.34 | 5.19 | 4678.88 | | 50 | 67.41 | 5.91 | 8018.23 | | 100 | 119.97 | 7.57 | 12214.97 | | 150 | 346.53 | 9.81 | 12988.89 | | 200 | 465.18 | 12.65 | 13348.4 | | 250 | 605.6 | 16.14 | 12964.72 | #### 500 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 13.84 | 4.56 | 218.86 | | 5 | 30.88 | 4.81 | 1035.85 | | 25 | 97.07 | 5.82 | 4263.49 | | 50 | 135.26 | 7.06 | 7009.01 | | 100 | 239.92 | 9.51 | 10379.22 | | 150 | 376.95 | 13.34 | 11053.75 | | 200 | 1664.34 | 14.96 | 12549.74 | | 250 | 6550.46 | 16.97 | 12268.82 | #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 19.03 | 4.53 | 220.1 | | 5 | 55.87 | 4.8 | 1031.3 | | 25 | 268.42 | 5.84 | 4098.01 | | 50 | 414.87 | 7.26 | 6514.66 | | 100 | 615.16 | 10.04 | 9392.43 | | 150 | 885.53 | 14.14 | 9982.74 | | 200 | 1258.79 | 16.02 | 11553.34 | | 250 | 1885.04 | 19.68 | 11527.61 | #### 5000 Input Tokens / 500 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 77.45 | 4.62 | 209.96 | | 5 | 279.92 | 5.42 | 838.03 | | 25 | 724.14 | 9.67 | 2251.44 | | 50 | 817.73 | 16.47 | 2765.71 | | 100 | 915.15 | 29.67 | 3180.16 | | 150 | 1080 | 43.35 | 3301.68 | | 200 | 4361.35 | 50.41 | 3378.48 | | 250 | 11717.42 | 50.53 | 3372 | #### 20000 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 403.2 | 5.06 | 189.96 | | 5 | 1141.22 | 7.39 | 627.76 | | 25 | 2371.46 | 19.35 | 1217.04 | | 50 | 11071.41 | 30.73 | 1330.57 | | 100 | 78187.07 | 31.08 | 1336.09 | | 150 | 145290.54 | 31.31 | 1337 | | 200 | 211735.71 | 31.46 | 1338.24 | | 250 | 278580.85 | 31.62 | 1337.66 | #### bf16 TP1 #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 27.36 | 6.52 | 152.84 | | 5 | 95.23 | 7.55 | 654.5 | | 25 | 332.13 | 9.34 | 2585.01 | | 50 | 589.89 | 11.46 | 4152.55 | | 100 | 922.41 | 15.99 | 5914.93 | | 150 | 1151.54 | 20.92 | 6800.02 | | 200 | 1273.42 | 25.68 | 7424.87 | | 250 | 5362.51 | 28.09 | 7435.16 | #### 1 x H200 141GB #### fp8 TP1 #### 200 Input Tokens / 200 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 9.97 | 4.35 | 228.55 | | 5 | 19.02 | 4.11 | 1194.26 | | 25 | 48.56 | 4.61 | 5170.73 | | 50 | 69.42 | 5.32 | 8823.15 | | 100 | 124.16 | 7.08 | 12949.5 | | 150 | 318.5 | 9.67 | 13298.39 | | 200 | 432.64 | 12.69 | 13456.29 | | 250 | 553.89 | 15.82 | 13392.14 | #### 500 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 12.88 | 4.25 | 235.17 | | 5 | 30.26 | 4.3 | 1160.07 | | 25 | 106.08 | 5.12 | 4832.39 | | 50 | 146.47 | 6.04 | 8175.67 | | 100 | 305.31 | 8.3 | 11824.76 | | 150 | 474.13 | 11.71 | 12530.29 | | 200 | 3858.17 | 12.7 | 13534.56 | | 250 | 8119.64 | 15.01 | 12973.67 | #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 19.79 | 4.24 | 234.95 | | 5 | 57.78 | 4.31 | 1144.52 | | 25 | 194.5 | 5.25 | 4598.71 | | 50 | 254.85 | 6.4 | 7513.85 | | 100 | 389.08 | 9.11 | 10533.71 | | 150 | 713.87 | 13.01 | 10930.77 | | 200 | 1362.27 | 14.27 | 12625.72 | | 250 | 2313.26 | 16.82 | 12826.05 | #### 5000 Input Tokens / 500 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 80.16 | 4.36 | 221.63 | | 5 | 249.53 | 4.92 | 924.18 | | 25 | 443.73 | 9.3 | 2458.55 | | 50 | 431.31 | 15.69 | 3024.47 | | 100 | 455.52 | 28.05 | 3458.25 | | 150 | 643.97 | 40.83 | 3566.03 | | 200 | 730.31 | 51.75 | 3763.37 | | 250 | 804.18 | 63.22 | 3860.48 | #### 20000 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 408.66 | 4.65 | 205.97 | | 5 | 1118.18 | 6.38 | 720.3 | | 25 | 1890.26 | 16.21 | 1455.56 | | 50 | 2911.51 | 30.64 | 1555.38 | | 100 | 19767.13 | 46.32 | 1736.48 | | 150 | 71795.92 | 46.88 | 1735.44 | | 200 | 123202.74 | 47.16 | 1734.91 | | 250 | 175197.57 | 47.31 | 1735.35 | #### bf16 TP1 #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 28.82 | 7.11 | 140.14 | | 5 | 97.14 | 7.27 | 679.46 | | 25 | 420.1 | 8.81 | 2711.49 | | 50 | 738.5 | 9.57 | 4854.67 | | 100 | 1125.65 | 13.45 | 6864.22 | | 150 | 1075.56 | 22.17 | 6456.17 | | 200 | 1399.1 | 24.22 | 7811.53 | | 250 | 2004.62 | 26.37 | 8813.31 | #### 1 x L40s 48GB #### fp8 TP1 #### 200 Input Tokens / 200 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 19.42 | 13.75 | 72.58 | | 5 | 49.22 | 14.12 | 349.76 | | 25 | 160.89 | 15.56 | 1535.29 | | 50 | 348.94 | 16.93 | 2688.88 | | 100 | 486 | 21.19 | 4250.73 | | 150 | 627.64 | 25.65 | 5232.81 | | 200 | 678.66 | 31.16 | 5812.48 | | 250 | 706.39 | 36.66 | 6245.55 | #### 500 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 26.98 | 14.06 | 71.1 | | 5 | 94.1 | 14.87 | 335.23 | | 25 | 381.83 | 18.2 | 1360.15 | #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 44.61 | 13.97 | 71.42 | | 5 | 173.24 | 14.91 | 331.7 | | 25 | 617.75 | 18.59 | 1302.3 | | 50 | 704.53 | 23.59 | 2060.1 | | 100 | 777.75 | 34.1 | 2869.49 | | 150 | 770.03 | 44.38 | 3324.96 | | 200 | 894.28 | 55.33 | 3559.97 | | 250 | 5047.2 | 62.7 | 3676.16 | #### 5000 Input Tokens / 500 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 243.92 | 14.72 | 65.86 | | 5 | 455.05 | 18.14 | 262.89 | | 25 | 644.27 | 36.93 | 655.22 | | 50 | 830.47 | 60.47 | 806 | | 100 | 8988.57 | 94.33 | 889.32 | | 150 | 36691.18 | 94.4 | 890.18 | | 200 | 64263.2 | 94.29 | 893.23 | #### 20000 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | ---------- | -------- | -------------------- | | 1 | 1299.33 | 17 | 56.67 | | 5 | 2691.78 | 26.37 | 180.37 | | 25 | 27548.98 | 70.21 | 289.26 | | 50 | 182179.84 | 70.65 | 289.79 | | 100 | 492175.92 | 70.71 | 291.05 | | 150 | 798991.51 | 70.76 | 291.4 | | 200 | 1106990.55 | 70.85 | 291.4 | #### bf16 TP1 #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 80.75 | 22.05 | 45.23 | | 5 | 152.54 | 23.58 | 210.91 | | 25 | 269.84 | 31.04 | 799 | | 50 | 312.86 | 40.48 | 1226.46 | | 100 | 2752.07 | 58.84 | 1613.38 | | 150 | 33137.67 | 58.88 | 1616.93 | | 200 | 63646.49 | 58.9 | 1618.87 | ### Version: 1.3.0 #### 1 x H100 80G #### fp8 TP1 #### 200 Input Tokens / 200 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 8.98 | 4.37 | 227.44 | | 5 | 15.31 | 5.1 | 971.34 | | 25 | 21.08 | 6.02 | 4100.33 | | 50 | 50.68 | 7.19 | 6745.78 | | 100 | 209.66 | 8.62 | 10369.56 | | 150 | 398.54 | 12.49 | 10383.16 | | 200 | 501.44 | 17.76 | 9884.77 | | | | | | #### 500 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 13.89 | 4.77 | 209.52 | | 5 | 27.22 | 5.45 | 915.96 | | 25 | 110.61 | 6.32 | 3924.39 | | 50 | 112.02 | 8.3 | 5979.38 | | 100 | 245.7 | 11.12 | 8878.96 | | 150 | 5217.57 | 13.78 | 8955.7 | | 200 | 9112.64 | 18.27 | 8443.89 | | 250 | 21097.73 | 19.26 | 8256.58 | #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 19.97 | 4.77 | 208.88 | | 5 | 57.87 | 5.45 | 908.9 | | 25 | 170.91 | 6.57 | 3710.08 | | 50 | 214.79 | 8.74 | 5582.41 | | 100 | 373.73 | 12.41 | 7819.41 | | 150 | 890.01 | 15.84 | 8935.06 | | 200 | 3231.57 | 18.31 | 9151.55 | | 250 | 6469.67 | 21.75 | 8725.87 | #### 5000 Input Tokens / 500 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 82.87 | 6.04 | 161.51 | | 5 | 249.06 | 7.04 | 664.39 | | 25 | 372.59 | 11.61 | 2024.67 | | 50 | 415.72 | 19.99 | 2398.11 | | 100 | 501.51 | 33.71 | 2844.33 | | 150 | 646.08 | 47.58 | 3047.56 | | 200 | 6830.5 | 51.83 | 3008.29 | | 250 | 14795.72 | 51.74 | 3030.76 | #### 20000 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 443.88 | 11.3 | 86.79 | | 5 | 1011.39 | 12.71 | 377.88 | | 25 | 1432.91 | 21.41 | 1099.79 | | 50 | 16505.76 | 32.16 | 1116.23 | | 100 | 89292.14 | 32.21 | 1190.64 | | 150 | 153371.08 | 32.17 | 1195.86 | | 200 | 208680.03 | 32.14 | 1200.69 | | 250 | 281213.01 | 32.19 | 1213.79 | #### fp16 TP1 #### 200 Input Tokens / 200 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 11.96 | 6.54 | 152.24 | | 5 | 27.18 | 7.39 | 667.60 | | 25 | 34.67 | 8.60 | 2858.90 | | 50 | 56.65 | 10.04 | 4858.86 | | 100 | 168.60 | 12.57 | 7478.74 | | 150 | 479.48 | 14.66 | 8811.28 | | 200 | 755.88 | 18.81 | 8864.97 | | 250 | 957.59 | 24.77 | 8439.91 | #### 500 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 19.89 | 6.94 | 143.93 | | 5 | 51.19 | 7.70 | 647.12 | | 25 | 202.87 | 9.42 | 2625.41 | | 50 | 329.35 | 11.62 | 4242.00 | | 100 | 552.68 | 15.67 | 6268.04 | | 150 | 782.65 | 20.32 | 7232.18 | | 200 | 7996.68 | 21.76 | 7666.88 | | 250 | 20815.08 | 21.96 | 7610.47 | #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 29.73 | 6.93 | 143.87 | | 5 | 96.74 | 7.72 | 640.26 | | 25 | 386.45 | 9.50 | 2530.05 | | 50 | 625.88 | 12.03 | 3951.34 | | 100 | 724.55 | 17.79 | 5326.15 | | 150 | 1126.07 | 22.88 | 6241.50 | | 200 | 1691.82 | 27.89 | 6548.23 | | 250 | 5664.63 | 29.12 | 7150.32 | #### 5000 Input Tokens / 500 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 128.32 | 8.14 | 119.30 | | 5 | 503.39 | 8.88 | 506.79 | | 25 | 1243.85 | 17.36 | 1259.32 | | 50 | 1273.57 | 29.38 | 1561.19 | | 100 | 6941.19 | 44.67 | 1689.99 | | 150 | 21179.86 | 44.66 | 1690.45 | | 200 | 35001.67 | 44.54 | 1694.93 | | 250 | 49204.54 | 44.48 | 1703.87 | #### 20000 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 637.84 | 13.16 | 74.21 | | 5 | 1881.45 | 13.50 | 346.33 | | 25 | 19476.78 | 31.76 | 579.50 | | 50 | 92969.97 | 31.81 | 558.51 | | 100 | 216580.75 | 31.83 | 582.54 | | 150 | 301171.32 | 31.79 | 583.14 | | 200 | 348383.77 | 31.81 | 582.88 | | 250 | 484376.95 | 31.85 | 584.58 | ## Llama-3.1-70b-instruct Results * NIM Container: [llama-3.1-70b-instruct](https://catalog.ngc.nvidia.com/orgs/nim/teams/meta/containers/llama-3.1-70b-instruct/tags) ### Version: 1.3.0 #### 4 x H100 80G #### fp8 TP4 #### 200 Input Tokens / 200 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 25.05 | 13.21 | 75.36 | | 5 | 64.82 | 12.64 | 387.55 | | 25 | 146.11 | 15.31 | 1564.9 | | 50 | 169.61 | 17.42 | 2743.6 | | 100 | 235.17 | 23.28 | 4092.34 | | 150 | 511.21 | 27.73 | 4962.78 | | 200 | 582.07 | 31.22 | 5855.9 | | 250 | 718.98 | 35.25 | 6429.95 | #### 500 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 38.64 | 14.24 | 70.14 | | 5 | 109.06 | 13.68 | 364.24 | | 25 | 393.58 | 15.86 | 1557.21 | | 50 | 564.17 | 17.14 | 2868.22 | | 100 | 640.42 | 22.64 | 4350.2 | | 150 | 792.84 | 28.49 | 5183.08 | | 200 | 1417.35 | 31.62 | 6168.81 | | 250 | 2045.69 | 35.75 | 6511.88 | #### 1000 Input Tokens / 1000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 59.89 | 14.24 | 69.98 | | 5 | 180.35 | 13.72 | 360.12 | | 25 | 571.16 | 16.38 | 1475.28 | | 50 | 843.45 | 18.15 | 2631.13 | | 100 | 740 | 25.55 | 3794.76 | | 150 | 952.07 | 32.31 | 4493.57 | | 200 | 980.89 | 37.92 | 5114.09 | | 250 | 2632.73 | 41.92 | 5497.21 | #### 5000 Input Tokens / 500 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 236.58 | 17.35 | 56.2 | | 5 | 432.23 | 18.16 | 263.04 | | 25 | 657.73 | 28.69 | 830.42 | | 50 | 952.43 | 40.13 | 1169.61 | | 100 | 1312.92 | 68.86 | 1378 | | 150 | 1973.14 | 98.88 | 1428.94 | | 200 | 2850.08 | 126.51 | 1455.09 | | 250 | 11283.63 | 139.94 | 1488.41 | #### 20000 Input Tokens / 2000 Output Tokens | Concurrency | TTFT (ms) | ITL (ms) | Throughput(Tokens/s) | | ----------- | --------- | -------- | -------------------- | | 1 | 1018.5 | 30.32 | 32.45 | | 5 | 2116.84 | 31.29 | 153.71 | | 25 | 3987.46 | 43.16 | 541.65 | | 50 | 7012.87 | 56.95 | 793.95 | | 100 | 93862.88 | 61.72 | 815.75 | | 150 | 175864.67 | 61.73 | 815.38 | | 200 | 240432.76 | 61.73 | 815.52 | | 250 | 335844.98 | 61.75 | 828.23 | ## Hardware Specifications ### NVIDIA H100 | Property | Value | | ----------------------------- | ------------------------------------------------------ | | Motherboard Model | NVIDIA DGX H100 | | Server Model | NVIDIA DGX H100 | | Number of Nodes | 1 | | CPU Information | Platinum 8480CL @ 3.8GHz Turbo (Sapphire Rapids) HT On | | Number of CPU sockets enabled | 2 | | Number of CPU threads enabled | 224 | | GPU Information | H100 80GB HBM3(GH100) | | Driver Information | 570.124.06 (r570\_00) | | GPU Core Clock (MHz) | 1980 | | GPU Boost Clock (MHz) | 1980 | | GPU Memory Clock (MHz) | 2619 | ### NVIDIA H200 | Property | Value | | ----------------------------- | ------------------------------------------------------ | | Motherboard Model | NVIDIA DGX H200 | | Server Model | NVIDIA DGX H200 | | Number of Nodes | 1 | | CPU Information | Platinum 8480CL @ 3.8GHz Turbo (Sapphire Rapids) HT On | | Number of CPU sockets enabled | 2 | | Number of CPU threads enabled | 224 | | GPU Information | H200 141GB HBM3e(GH100) | | Driver Information | 570.124.06 (r570\_00) | | GPU Core Clock (MHz) | 1980 | | GPU Boost Clock (MHz) | 1980 | | GPU Memory Clock (MHz) | 2619 | ### NVIDIA A100 | Property | Value | | ----------------------------- | ------------------------------------------------ | | Motherboard Model | NVIDIA DGX A100 | | Server Model | NVIDIA DGX A100 | | Number of Nodes | 1 | | CPU Information | AMD EPYC 7742 @ 2.2GHz 3.4GHz Turbo (Rome) HT On | | Number of CPU sockets enabled | 2 | | Number of CPU threads enabled | 256 | | GPU Information | A100 SXM4 80GB(GA100) | | Driver Information | 570.124.06 (r570\_00) | | GPU Core Clock (MHz) | 1155 | | GPU Boost Clock (MHz) | 1401 | | GPU Memory Clock (MHz) | 1593 | ### NVIDIA L40s | Property | Value | | ----------------------------- | ---------------------------------------------------------- | | Motherboard Model | Supermicro X13DEG-OA | | Server Model | Supermicro SYS-521GE-TNRT | | Number of Nodes | 1 | | CPU Information | Platinum 8570 @ 2.1GHz 4.0GHz Turbo (Emerald Rapids) HT On | | Number of CPU sockets enabled | 2 | | Number of CPU threads enabled | 224 | | GPU Information | L40S(AD102) | | Driver Information | 570.124.06 (r570\_00) | | GPU Core Clock (MHz) | 2520 | | GPU Boost Clock (MHz) | 2520 | | GPU Memory Clock (MHz) | 9001 |