Self-Managed NVCF HTTP Soak Test

This guide walks through running a sustained HTTP soak test against a self-managed NVCF cluster using k6. The test sends a constant arrival-rate load to one or more deployed functions and reports success rate, latency percentiles, and throughput over an extended period (default 48 hours).

Prerequisites

Self-hosted CLI

You need a working nvcf-cli configured against your self-managed cluster. If you have not set this up yet, follow the self-hosted-cli guide to install the binary and the cli-configuration section to point it at your gateway.

Verify the CLI can reach the cluster before continuing:

$
./nvcf-cli init

Deploy the load test function

Use the load_tester_supreme container for soak testing. It is purpose-built for high-throughput benchmarking and includes:

• gRPC + HTTP + SSE endpoints in a single image
• Tunable repeats, delay, and size fields to shape request/response profiles
• Built-in OpenTelemetry tracing

The source, build instructions, and registry push examples are in the nv-cloud-function-helpers repository. Build and push the image to whichever container registry your cluster has credentials for:

$
git clone https://github.com/NVIDIA/nv-cloud-function-helpers.git
$
cd nv-cloud-function-helpers/examples/function_samples/load_tester_supreme
$
$
# Build (multi-arch)
$
docker buildx build --platform linux/amd64,linux/arm64 -t load_tester_supreme .
$
$
# Tag and push (replace with your registry -- ECR, NGC, Docker Hub, etc.)
$
docker tag load_tester_supreme <your-registry>/load_tester_supreme:latest
$
docker push <your-registry>/load_tester_supreme:latest

Then create the function and deploy it using the CLI. For HTTP soak testing you can create multiple functions to simulate broader load:

$
# Create the function (HTTP)
$
./nvcf-cli function create \
>
  --name "load-tester-supreme" \
>
  --image "<your-registry>/load_tester_supreme:latest" \
>
  --inference-url "/echo" \
>
  --inference-port 8000 \
>
  --health-uri "/health" \
>
  --health-port 8000 \
>
  --health-timeout PT30S
$
$
# Deploy (adjust GPU type and instance type for your cluster)
$
./nvcf-cli function deploy create \
>
  --gpu L40S \
>
  --instance-type NCP.GPU.L40S_1x \
>
  --min-instances 1 \
>
  --max-instances 1
$
$
# Generate an API key for invocations (default expiry: 24h)
$
./nvcf-cli api-key generate
$
$
# For soak tests longer than 24 hours, set a longer expiry:
$
./nvcf-cli api-key generate --expires-in "7d"

Export the key so it can be passed to k6:

$
export API_KEY=<your-nvapi-key>

Repeat the function create and function deploy create steps to create additional functions if you want to distribute load across multiple function endpoints.

Once deployed, note the following — you will need them for the k6 script:

• Function ID — the UUID returned by function create
• Invocation domain — the domain after invocation. in the function invocation hostname
• API key — from ./nvcf-cli api-key generate (begins with nvapi-); export it as $API_KEY

Obtain the gateway address

Your gateway address is the external address of the Envoy Gateway deployed with the control plane. To retrieve it:

$
export GATEWAY_ADDR=$(kubectl get gateway nvcf-gateway -n envoy-gateway \
>
  -o jsonpath='{.status.addresses[0].value}')
$
echo "Gateway Address: $GATEWAY_ADDR"

On AWS EKS this is an ELB hostname (e.g. a1b2c3d4.us-east-1.elb.amazonaws.com). For a local deployment (Kind, k3d, Docker Desktop) it is typically localhost or 127.0.0.1.

The gateway routes invocation traffic through function-specific hostnames:

For the soak test, set INVOKE_DOMAIN to the domain after invocation. in the function invocation hostname. For a self-managed gateway that uses <function-id>.invocation.$GATEWAY_ADDR, set INVOKE_DOMAIN to $GATEWAY_ADDR.

Install k6

Install k6 if you don’t have it:

$
# macOS
$
brew install k6

$
# Linux (Debian/Ubuntu)
$
sudo gpg -k
$
sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg \
>
  --keyserver hkp://keyserver.ubuntu.com:80 \
>
  --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
$
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" \
>
  | sudo tee /etc/apt/sources.list.d/k6.list
$
sudo apt-get update && sudo apt-get install k6

The k6 test script

Save the following script as k6-nvcf-http-soak.js. The script uses the constant-arrival-rate executor to guarantee an exact request rate per second regardless of response time, which is critical for soak testing where you want a steady, predictable load.

1
/**
2
 * NVCF HTTP soak test script.
3
 *
4
 * Pass a FUNCTIONS JSON array with one entry per function you want to
5
 * load-test (one function per GPU node is typical). There is no limit
6
 * on the number of functions -- each iteration sends one request to
7
 * every function via http.batch().
8
 *
9
 * Total TPS = TPS_PER_FUNC × number of functions.
10
 *
11
 * Required env vars: INVOKE_DOMAIN, FUNCTIONS
12
 * Optional: API_KEY, INVOKE_SCHEME, TPS_PER_FUNC, PRE_VUS, MAX_VUS, REPEATS, DURATION
13
 *
14
 * Example:
15
 *   k6 run -e INVOKE_DOMAIN=$GATEWAY_ADDR \
16
 *          -e 'FUNCTIONS=[{"funcId":"uuid1"}]' \
17
 *          -e DURATION=48h \
18
 *          -e API_KEY=$API_KEY \
19
 *          k6-nvcf-http-soak.js
20
 */
21
import http from 'k6/http';
22
import { check } from 'k6';
23
import { Rate, Trend, Counter } from 'k6/metrics';
24
25
const invokeSuccess = new Rate('invoke_success');
26
const invokeLatency = new Trend('invoke_latency_ms');
27
const totalRequests = new Counter('total_requests');
28
29
// ---------- Required config ----------
30
const DURATION      = __ENV.DURATION      || '48h';
31
const INVOKE_DOMAIN = __ENV.INVOKE_DOMAIN || '';
32
33
const FUNCTIONS = (() => {
34
  const raw = __ENV.FUNCTIONS || '';
35
  if (!raw) return [];
36
  try {
37
    const arr = JSON.parse(raw);
38
    if (!Array.isArray(arr) || arr.length === 0) return [];
39
    return arr;
40
  } catch (e) {
41
    console.warn('FUNCTIONS JSON parse failed: ' + e.message);
42
    return [];
43
  }
44
})();
45
46
// ---------- Optional parameters ----------
47
const API_KEY       = __ENV.API_KEY       || '';
48
const INVOKE_SCHEME = __ENV.INVOKE_SCHEME || 'https';
49
const REPEATS       = parseInt(__ENV.REPEATS      || '100', 10);
50
const TPS_PER_FUNC  = parseInt(__ENV.TPS_PER_FUNC || '125', 10);
51
const PRE_VUS       = parseInt(__ENV.PRE_VUS      || '250', 10);
52
const MAX_VUS       = parseInt(__ENV.MAX_VUS      || '500', 10);
53
54
export const options = {
55
  scenarios: {
56
    invoke_all_functions: {
57
      executor: 'constant-arrival-rate',
58
      rate: TPS_PER_FUNC,
59
      timeUnit: '1s',
60
      duration: DURATION,
61
      preAllocatedVUs: PRE_VUS,
62
      maxVUs: MAX_VUS,
63
      exec: 'invoke_all_functions',
64
    },
65
  },
66
  thresholds: {
67
    http_req_duration: ['p(95)<5000'],
68
    invoke_success:    ['rate>0.999'],
69
  },
70
};
71
72
export function invoke_all_functions() {
73
  const payload = JSON.stringify({ message: 'randomString', repeats: REPEATS });
74
75
  const requests = FUNCTIONS.map((f, i) => ({
76
    method: 'POST',
77
    url: `${INVOKE_SCHEME}://${f.funcId}.invocation.${INVOKE_DOMAIN}/echo`,
78
    body: payload,
79
    params: {
80
      headers: {
81
        'Content-Type':      'application/json',
82
        'Authorization':     `Bearer ${API_KEY}`,
83
        'Nvcf-Poll-Seconds': '5',
84
      },
85
      tags: { func: `func_${i}` },
86
    },
87
  }));
88
89
  const responses = http.batch(requests);
90
  responses.forEach((res, i) => {
91
    totalRequests.add(1);
92
    invokeSuccess.add(res.status === 200);
93
    if (res.status === 200) invokeLatency.add(res.timings.duration);
94
95
    check(res, {
96
      [`func_${i} status 200`]:     (r) => r.status === 200,
97
      [`func_${i} fulfilled`]:      (r) => r.headers['Nvcf-Status'] === 'fulfilled',
98
      [`func_${i} body not empty`]: (r) => r.body && r.body.length > 0,
99
    });
100
  });
101
}

Running the soak test

A typical soak test uses multiple functions at a sustained TPS for an extended period. For example, multiple functions at 125 TPS each for 48 hours:

$
k6 run k6-nvcf-http-soak.js \
>
  -e INVOKE_SCHEME=https \
>
  -e INVOKE_DOMAIN=$GATEWAY_ADDR \
>
  -e 'FUNCTIONS=[{"funcId":"<id-1>"},{"funcId":"<id-2>"},{"funcId":"<id-3>"},{"funcId":"<id-4>"}]' \
>
  -e DURATION=48h \
>
  -e TPS_PER_FUNC=125 \
>
  -e API_KEY=$API_KEY

$
tmux new -s soak
$
# run k6 command above
$
# Ctrl-B D to detach, tmux attach -t soak to re-attach

Tune the load

TPS per function

The constant-arrival-rate executor guarantees a fixed number of requests per second. The total TPS is TPS_PER_FUNC × number of functions.

VU allocation

Each VU is a virtual user that can hold one in-flight request. The executor creates new VUs if existing ones are busy. If you see insufficient VUs, consider increasing maxVUs warnings, increase PRE_VUS and MAX_VUS:

$
-e PRE_VUS=500 -e MAX_VUS=1000

A good rule of thumb: PRE_VUS ≈ TPS_PER_FUNC × avg_latency_seconds × 2.

Environment variables reference

Interpreting results

k6 prints a summary to stdout at the end of each run. Key metrics to monitor during a soak test:

To save results for offline analysis:

$
# Plain text log
$
k6 run k6-nvcf-http-soak.js ... 2>&1 | tee soak-run.log
$
$
# JSON output for Grafana / post-processing
$
k6 run --out json=soak-results.json k6-nvcf-http-soak.js ...

Verifying the endpoint manually

Before running the soak test, verify the endpoint works with curl:

$
export FUNCTION_ID=<your-function-id>
$
$
curl --request POST \
>
  --url "https://${FUNCTION_ID}.invocation.${GATEWAY_ADDR}/echo" \
>
  --header "Authorization: Bearer ${API_KEY}" \
>
  --header "Content-Type: application/json" \
>
  --data '{"message": "hello"}'

You should receive a 200 OK response with the Nvcf-Status: fulfilled header and the message repeated three times.