Reference

Self-Managed NVCF gRPC Load Test

View as Markdown

Prerequisites

Self-hosted CLI

You need a working nvcf-cli configured against your self-managed cluster. If you have not set this up yet, follow the self-hosted-cli guide to install the binary and the cli-configuration section to point it at your gateway.

Verify the CLI can reach the cluster before continuing:

$./nvcf-cli init
$./nvcf-cli api-key generate

Deploy the load test function

Use the load_tester_supreme container for load testing. It is purpose-built for high-throughput benchmarking and includes:

  • gRPC + HTTP + SSE endpoints in a single image
  • 500 gRPC worker threads by default (configurable via WORKER_COUNT), compared to 10 in the simpler grpc_echo_sample
  • Tunable repeats, delay, and size fields to shape request/response profiles
  • Built-in OpenTelemetry tracing

The grpc_echo_sample from the same repository shares the Echo/EchoMessage proto and will work for a quick smoke test, but its 10-thread pool will saturate under moderate concurrency. Always use load_tester_supreme for real load tests.

The source, build instructions, and registry push examples are in the NVCF examples repository. Build and push the image to whichever container registry your cluster has credentials for:

$git clone https://github.com/NVIDIA/nvcf.git
$cd nvcf/examples/function-samples/load-tester-supreme
$
$# Build
$docker build --platform linux/amd64 -t load_tester_supreme .
$
$# Tag and push (replace with your registry -- NGC, ECR, etc.)
$docker tag load_tester_supreme nvcr.io/<your-org>/load_tester_supreme:latest
$docker push nvcr.io/<your-org>/load_tester_supreme:latest

To check which registries your cluster recognises, run ./nvcf-cli registry list.

Then create the function and deploy it using the CLI:

$# Create the function (gRPC -- set inference-url to a placeholder)
$./nvcf-cli function create \
>  --name "load-tester-supreme" \
>  --image "nvcr.io/<your-org>/load_tester_supreme:latest" \
>  --inference-url "/grpc" \
>  --inference-port 8001 \
>  --health-protocol GRPC \
>  --health-uri "/" \
>  --health-port 8001 \
>  --health-timeout PT30S \
>
$# Deploy (adjust GPU type and instance type for your cluster)
$./nvcf-cli function deploy create \
>  --gpu H100 \
>  --instance-type NCP.GPU.H100_1x \
>  --min-instances 1 \
>  --max-instances 1 \
>  --max-request-concurrency 1000 \
>  --function-id <function id> \
>  --version-id <version id>
$
$# Generate an API key for invocations
$./nvcf-cli api-key generate

Once deployed, note the following — you will need them for the run script:

  • Function ID — the UUID returned by function create
  • Function Version ID — the UUID of the specific deployed version
  • gRPC endpoint — your gateway address on port 10081 (see below)
  • API key — the key from api-key generate (begins with nvapi-)

Your gateway address is the external address of the Envoy Gateway deployed with the control plane. To retrieve it:

$kubectl get gateway nvcf-gateway -n envoy-gateway \
>  -o jsonpath='{.status.addresses[0].value}'

On AWS EKS this is an ELB hostname (e.g. a1b2c3d4.us-east-1.elb.amazonaws.com). For a local deployment (Kind, k3d, Docker Desktop) it is typically localhost or 127.0.0.1.

The CLI saves the function and version IDs automatically. Run ./nvcf-cli status to view them at any time.

Clone the load test scripts

$git clone https://github.com/NVIDIA/nvcf.git
$cd nvcf/examples/load-tests

Install k6

Install k6 if you don’t have it:

$# macOS
$brew install k6
$# Linux (Debian/Ubuntu)
$sudo gpg -k
$sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg \
>  --keyserver hkp://keyserver.ubuntu.com:80 \
>  --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
$echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" \
>  | sudo tee /etc/apt/sources.list.d/k6.list
$sudo apt-get update && sudo apt-get install k6

Create your run script

The run*.sh scripts are gitignored, so each user creates their own locally. Create run_grpc_self_managed_test.sh in the load-tests directory:

$#!/bin/bash
$
$set -e
$
$# From "nvcf-cli api-key show"
$export TOKEN=<your-nvapi-token>
$
$# Gateway address + gRPC port (see "kubectl get gateway" above)
$# Examples:
$#   AWS EKS:  a1b2c3d4.us-east-1.elb.amazonaws.com:10081
$#   Local:    localhost:10081
$export NVCF_GRPC_URL=<your-gateway-address>:10081
$# From "nvcf-cli status" or the function create output
$export GRPC_SUPREME_FUNCTION_ID=<your-function-id>
$export GRPC_SUPREME_FUNCTION_VERSION_ID=<your-function-version-id>
$
$export GRPC_PLAINTEXT=true
$
$export SENT_MESSAGE_SIZE=2048
$export RESPONSE_COUNT=1
$
$k6 run functions/supreme_grpc_test.js \
>  --vus 10 --duration 60s \
>  --env TOKEN=${TOKEN} \
>  --env NVCF_GRPC_URL=${NVCF_GRPC_URL} \
>  --env GRPC_SUPREME_FUNCTION_ID=${GRPC_SUPREME_FUNCTION_ID} \
>  --env GRPC_SUPREME_FUNCTION_VERSION_ID=${GRPC_SUPREME_FUNCTION_VERSION_ID} \
>  --env GRPC_PLAINTEXT=${GRPC_PLAINTEXT} \
>  --env SENT_MESSAGE_SIZE=${SENT_MESSAGE_SIZE} \
>  --env RESPONSE_COUNT=${RESPONSE_COUNT}

Make it executable and run:

$chmod +x run_grpc_self_managed_test.sh
$./run_grpc_self_managed_test.sh

Tune the load

Virtual users (VUs)

Each VU simulates a single concurrent client holding an open gRPC connection and sending requests in a loop. The number of VUs directly controls the concurrency hitting your endpoint.

VUsSimulates
1—5Smoke test — verify the endpoint works under minimal load
10—50Light load — a small team or service calling the function
100—500Moderate load — multiple services or a rollout with real traffic
1000+Stress test — find the breaking point or max throughput

Default control plane sizing: The default resource sizing that ships with the self-managed stack is designed to handle roughly 100 concurrent users. If you need to test beyond that, you will need to scale the control plane components first. Starting with --vus 100 or the scratch config is a good baseline for validating a default self-managed deployment.

Start low and increase gradually. If you see rising error rates or latency, you have found the saturation point.

Fixed VUs for a set duration (simplest approach):

$# 10 concurrent users for 1 minute
$k6 run functions/supreme_grpc_test.js --vus 10 --duration 60s ...
$
$# 200 concurrent users for 10 minutes
$k6 run functions/supreme_grpc_test.js --vus 200 --duration 10m ...

Ramping VUs with a config file (recommended for real load tests):

Config files let you gradually ramp users up, hold steady, and ramp down. This avoids slamming the endpoint all at once and gives more realistic results.

The k6_long_scaling_test_config.json ramps to 100 VUs over 5 minutes, holds for 15 minutes, then steps through higher concurrency levels:

$k6 run functions/supreme_grpc_test.js \
>  --config functions/test_configs/k6_long_scaling_test_config.json \
>  --env TOKEN=${TOKEN} \
>  --env NVCF_GRPC_URL=${NVCF_GRPC_URL} \
>  --env GRPC_SUPREME_FUNCTION_ID=${GRPC_SUPREME_FUNCTION_ID} \
>  --env GRPC_SUPREME_FUNCTION_VERSION_ID=${GRPC_SUPREME_FUNCTION_VERSION_ID} \
>  --env GRPC_PLAINTEXT=${GRPC_PLAINTEXT} \
>  --env SENT_MESSAGE_SIZE=${SENT_MESSAGE_SIZE} \
>  --env RESPONSE_COUNT=${RESPONSE_COUNT}

The k6_hammer_test_config.json uses a ramping-arrival-rate executor that ramps up to 100,000 requests/second — use this only when you want to push the endpoint to its limit:

$k6 run functions/supreme_grpc_test.js \
>  --config functions/test_configs/k6_hammer_test_config.json \
>  --env TOKEN=${TOKEN} \
>  --env NVCF_GRPC_URL=${NVCF_GRPC_URL} \
>  --env GRPC_SUPREME_FUNCTION_ID=${GRPC_SUPREME_FUNCTION_ID} \
>  --env GRPC_SUPREME_FUNCTION_VERSION_ID=${GRPC_SUPREME_FUNCTION_VERSION_ID} \
>  --env GRPC_PLAINTEXT=${GRPC_PLAINTEXT} \
>  --env SENT_MESSAGE_SIZE=${SENT_MESSAGE_SIZE} \
>  --env RESPONSE_COUNT=${RESPONSE_COUNT}

Other tuning parameters

ParameterPurpose
SENT_MESSAGE_SIZEPayload size in bytes (e.g. 2048 for 2 KB)
RESPONSE_COUNTNumber of echoed responses per request

Environment variables reference

VariablePurpose
TOKENYour nvapi-* bearer token
NVCF_GRPC_URLgRPC endpoint — your gateway address with port 10081 (e.g. a1b2c3d4.us-east-1.elb.amazonaws.com:10081 or localhost:10081)
GRPC_SUPREME_FUNCTION_IDFunction ID from NVCF
GRPC_SUPREME_FUNCTION_VERSION_IDFunction version ID (required for self-managed deployments)
GRPC_PLAINTEXTSet to true for non-TLS connections
SENT_MESSAGE_SIZESize of the test payload in bytes
RESPONSE_COUNTNumber of responses the server should return

Verifying your endpoint manually

Before running a load test, you can verify the endpoint works with grpcurl:

$grpcurl -plaintext \
>  -H "function-id: <your-function-id>" \
>  -H "function-version-id: <your-function-version-id>" \
>  -H "authorization: Bearer <your-api-key>" \
>  -d '{"message": "hello from grpc"}' \
>  <your-gateway-address>:10081 Echo/EchoMessage