Is this page helpful?

Release Notes#

Release 2.0.10-variant#

This release contains model updates outlined in the following sections.

Gemma 4 31B IT#

This is an updated release of Gemma 4 31B IT. This NIM is part of the NIM Certified offering. For more information on this model, refer to the model card.

For GPU support, refer to the support matrix for Gemma 4 31B IT.

Note the following limitation:

  • 8K MP4 video inputs are not supported and can be rejected. Use lower-resolution MP4 inputs for video requests.

Inkling#

This is an updated release of Inkling. This NIM is part of the NIM Certified offering. For more information on this model, refer to the model card.

For GPU support, refer to the support matrix for Inkling.

Note the following limitations:

  • Structured output is not supported for non-streaming requests. Requests that use response_format or structured_outputs, covering JSON schema, JSON object, choice, regex, and grammar constraints, return HTTP 500. Streaming requests return constrained content, except that requests using regex or grammar constraints can return empty content.

  • On /v1/completions, structured_outputs.choice can be accepted without being enforced, producing unconstrained output.

  • tool_choice: "required" is not enforced. Non-streaming requests return HTTP 500. Streaming requests return HTTP 200 without a tool call.

  • tool_choice: "none" does not reliably suppress tool invocation, and tool_choice: "auto" does not reliably invoke a relevant tool from the supplied list.

  • Named tool choice, tool_choice: {"type": "function", "function": {"name": "..."}}, is not enforced. The model can call a different function from the supplied tool list.

  • The echo parameter has no effect on /v1/chat/completions. The user message is not prepended to the response.

  • An empty audio_url.url returns HTTP 500 rather than a 4xx client error.

  • Reasoning tokens are not reported in usage accounting. Responses that contain reasoning_content report usage.reasoning_tokens as zero.

  • The thinking_token_budget parameter is not available and returns HTTP 400. It requires the V1 model runner (VLLM_USE_V2_MODEL_RUNNER=0). This release uses the V2 runner, which does not implement it.

  • On /v1/messages, thinking blocks are returned even when the thinking parameter is omitted. Reasoning cannot be disabled through the Anthropic Messages API.

  • On /v1/messages, a request can return HTTP 200 whose content contains only a thinking block with no text block. This is deterministic when stop_sequences contains a token that also occurs during reasoning, and has been observed intermittently otherwise. Increasing max_tokens does not recover the text.

  • The documents extension to /v1/chat/completions has no effect. The field is accepted, but the Inkling chat template does not consume it, so document content does not reach the prompt. Include document text in the message body instead.

  • Chat batch requests, where messages is a list of conversations, return choices with content and reasoning set to null. Send batched conversations as individual requests.

  • Responses API storage is disabled. Requests using store: true with previous_response_id return HTTP 404. GET /v1/responses/{response_id} and requests that cancel a response also return HTTP 404. background: true returns HTTP 400. Storage is gated behind VLLM_ENABLE_RESPONSES_API_STORE, which the upstream engine leaves disabled by default because the response store is unbounded.

  • Inkling generates reasoning content before its final answer, and reasoning consumes the completion token budget. If max_tokens is exhausted during reasoning, the response returns finish_reason: "length" with an empty content field. Size max_tokens to accommodate both the reasoning and the answer.

  • MTP speculative decoding is enabled only on the single-node, 8-GPU NVFP4 profiles. The 16-GPU BF16 configurations span two nodes with pipeline parallelism, which the speculative decoding draft module does not support, so those profiles run without it.

  • Setting NIM_LOG_LEVEL=DEBUG causes the first inference request to return HTTP 500 and terminate the model engine. Default INFO logging is unaffected. To collect debug logs, also set VLLM_USE_BREAKABLE_CUDAGRAPH=0.

  • Request validation and error semantics are inherited from the upstream vLLM inference engine and are preserved intentionally. As a result, the API does not guarantee strict or consistent parameter validation, and clients should validate requests on their side rather than relying on the server to reject malformed input. Note the following:

    • Some invalid or out-of-range parameters, such as an empty messages array, an empty content string, top_k=0, a best_of value less than n, or repetition_penalty=3.0, are accepted and return HTTP 200 rather than a validation error.

    • On the /v1/messages endpoint, out-of-range temperature or top_p values are surfaced as an HTTP 500 internal server error rather than a 4xx client error.

    • For streaming requests, if the server detects an error after the response has begun, it delivers the error as an event within the Server-Sent Events (SSE) stream rather than as a top-level HTTP status code.

    These behaviors match conventional vLLM and are not specific to this NIM.

For information about past updates and older versions, refer to the previous release notes.