Is this page helpful?

Observability#

Alpamayo1.5 NIM exposes standard NIM health, metadata, and metrics endpoints over HTTP. For endpoint details, refer to API Reference.

Poll /v1/health/ready before routing requests. Input validation errors do not change readiness. If CUDA reports a device-side assertion, illegal memory access, or launch failure, the inference server stops because its CUDA context cannot be recovered safely. Readiness then becomes non-200; restart the container before sending more traffic.

Health and Prometheus Metrics#

Check liveness and readiness independently:

curl --fail http://localhost:8000/v1/health/live
curl --fail http://localhost:8000/v1/health/ready

Scrape Prometheus-format metrics from /v1/metrics:

curl --fail --silent http://localhost:8000/v1/metrics \
   --output /tmp/alpamayo-metrics.prom
grep -E '^(gpu_|process_|python_)' /tmp/alpamayo-metrics.prom | head

Configure your Prometheus deployment to scrape the same path and HTTP port. Metric names can change as the serving runtime evolves; discover the names from the exposition rather than hard-coding the preceding example filter.

In this release, /v1/metrics reports GPU and process/runtime telemetry. It does not emit per-request counters, request-latency histograms, or an Alpamayo queue-depth gauge. Use the response inference_time field for model execution time, the following client-side benchmark for end-to-end latency, and request profiling for detailed stage timings. HTTP requests can be buffered or serialized by upstream serving layers, so this release does not guarantee a queue-full status at a fixed client concurrency. The direct gRPC API reports RESOURCE_EXHAUSTED if the bounded backend queue rejects a request.

CUDA JPEG Decode Fallback#

JPEG requests normally use CUDA-accelerated decoding through torchvision and nvJPEG. If that path raises an exception, the NIM retries the same payload with the regular image decoder. A valid request can therefore still succeed, but that request’s image preprocessing is slower.

An isolated CUDA JPEG failure does not permanently disable acceleration. The next JPEG request retries CUDA decoding. A successful retry resets the failure counter and logs that GPU decoding recovered. Malformed JPEG input that also fails regular decoding returns a client error and does not count against the CUDA failure counter.

After three consecutive CUDA failures on payloads that successfully decode with the regular decoder, the NIM disables CUDA JPEG decoding for the remaining lifetime of that container process. Subsequent requests continue through the regular decoder. Restart the container to re-enable and retry CUDA JPEG decoding. The transition is reported with a bounded, single-line log message; the underlying C++ stack trace is not emitted.

This fallback affects image preprocessing performance, not the response schema or model selection. If CUDA JPEG decoding repeatedly becomes disabled, compare request latency before and after the transition and retain the one-line error, container version, GPU model, driver version, and the JPEG properties for diagnosis.

Request Profiling#

To enable request timings, add these options to the docker run command in Quickstart Guide:

-e NIM_ALPAMAYO_ENABLE_TIMINGS=1 \
-e NIM_ALPAMAYO_TIMINGS_DIR=/opt/nim/.cache/alpamayo_timings

While timing is enabled, the server logs one [alpamayo-trtllm] timings ... JSON payload per trajectory request and writes timing artifacts under NIM_ALPAMAYO_TIMINGS_DIR. Enable request profiling only while collecting diagnostics, then restart the container without these variables for normal serving.

Minimal Warm-Path Benchmark#

After copying build_http_payload.py and sample_data as described in API Reference, create a full-layout request and warm it once:

python3 build_http_payload.py \
   --endpoint infer --sample-dir sample_data > /tmp/alpamayo-infer.json

curl --fail --silent --output /dev/null \
   -H 'Content-Type: application/json' \
   --data-binary @/tmp/alpamayo-infer.json \
   http://localhost:8000/v1/infer

Then record ten single-client warm-path request times:

for run in $(seq 1 10); do
   curl --fail --silent --output /dev/null \
      --write-out "run=$run seconds=%{time_total}\n" \
      -H 'Content-Type: application/json' \
      --data-binary @/tmp/alpamayo-infer.json \
      http://localhost:8000/v1/infer
done

This recipe measures developer-facing, single-client end-to-end latency. It is not a production throughput or concurrency benchmark. Keep the image layout, profile, GPU, K value, sampling parameters, and warmup state fixed when comparing results.