Release Notes

Release 1.10.0

Summary

This release adds optional admission control for saturated deployments, and corrects health reporting so that a container with a failed GPU worker no longer reports itself as ready.

Key Features

• Optional admission control and queue metrics. A saturated NIM can now shed load instead of queueing without bound. Set NIM_ENABLE_QUEUE_METRICS to true, together with NIM_MAX_CONCURRENT_REQUESTS, to enable it. Requests beyond the limit wait, and then receive 503 with a Retry-After header instead of blocking until an unrelated timeout expires.

  This NIM serves one request per GPU, so NIM_MAX_CONCURRENT_REQUESTS must equal the number of GPUs that the container can use. The container refuses to start when the two values disagree. A larger value admits requests that the GPU pool cannot run, and the queue depth reported on /v1/metrics then does not describe where those requests are waiting. Health and documentation endpoints bypass admission control. The /v1/models endpoint and the predict endpoint do not. Refer to Configuring the Boltz-2 NIM.

  While admission control is enabled, the following Prometheus series are published on /v1/metrics. The names match the vLLM-based NIMs, so autoscaling thresholds transfer between products.

  Metric	Description
  nim_request_queue_depth	Number of requests waiting for a GPU.
  nim_requests_running	Number of requests running on a GPU.
  nim_request_queue_time_seconds	Histogram of time spent waiting for a GPU.
  nim_requests_shed_total	Requests rejected with 503, labeled queue_full or timeout.

• nimlib upgrades from 0.19.5 to 0.22.0, which provides the admission control middleware and the queue metrics.

Fixed Issues

• A NIM with a failed GPU worker no longer reports itself as ready. When a GPU worker failed while loading models, /v1/health/ready continued to return true, and each request then blocked for 600 seconds before failing. Startup now fails with the error reported by the worker, and the container exits. Readiness continues to track worker liveness afterward, so a worker lost later to a CUDA fault or an out-of-memory condition also removes the NIM from rotation. A request that reaches a failed worker now fails in about five seconds instead of 600.

• A fatal startup error no longer leaves the container running. Fatal conditions raised while the inference interface is built, such as a missing checkpoint under NIM_FAIL_WITHOUT_MODEL or an admission control limit that disagrees with the GPU count, previously left the process alive with no server listening. The container now exits with a non-zero status and reports the reason on standard error.

• Addresses CVE findings across container dependencies.

Notes

• This release does not change prediction output. Response fields, their types, and predicted structures are unchanged.
• Admission control is disabled unless you set NIM_ENABLE_QUEUE_METRICS. When it is unset, request handling and timeouts behave as they do in 1.9.0, and none of the queue metric series are registered. An uninstrumented NIM does not advertise metric names that it never populates.