Model-Specific Deployment Considerations

Some NIM LLM and VLM containers, including the Model-Free NIMs, need extra profile selection or startup settings beyond the standard Get Started path. Use this page for those models.

Important

Check the version of the NIM you are deploying. This page only covers NIMs for the current release version. Refer to the previous versions of the documentation for NIMs that have already been released.

For generic profile selection, refer to Model Profiles and Selection. For validated GPU and precision combinations, refer to the Support Matrix for NIMs. Check the Release Notes for the requirements that apply to your image version.

Muse Glimmer

muse-glimmer reasons before it answers every request, and its container enforces the cache-directory permission step that the Get Started path lists as optional. Account for both before you follow the standard quickstart.

Cache Permissions Are Required

The muse-glimmer container checks the mounted cache directory during startup and exits if the directory is not writable by the container user. A cache created with mkdir -p alone fails that check about one second after the container starts:

ERROR: Cache directory /opt/nim/.cache is not accessible by the container user (UID=1000, GID=0).

Apply the cache-permission step from configuration before you start the container. For muse-glimmer, this step is required:

export LOCAL_NIM_CACHE=~/.cache/nim/muse-glimmer
mkdir -p "$LOCAL_NIM_CACHE"
chmod -R a+rwxt "$LOCAL_NIM_CACHE"

As an alternative that needs no chmod, run the container as your own user in the root group by adding -u $(id -u):0 to the docker run command.

Reasoning Is Always On

muse-glimmer emits a reasoning section on every request. For /v1/chat/completions, read reasoning text from message.reasoning_content, or from delta.reasoning_content when streaming, and read the answer from message.content. This container uses SGLang’s reasoning_content field instead of the reasoning field shown in vLLM examples. For the Messages response format, refer to Messages (Anthropic-compatible).

Reasoning cannot be turned off. chat_template_kwargs.enable_thinking, thinking, and reasoning_effort: "none" have no effect on this model.

To reduce the amount of reasoning, set reasoning_strength in chat_template_kwargs. The packaged chat template accepts low, medium, high, and xhigh, and defaults to high.

Budget Enough Output Tokens

Reasoning tokens count against max_tokens. A request with a small budget spends the whole budget inside the reasoning section and returns an empty content with finish_reason: "length". Chat examples elsewhere in this documentation that use 64 to 256 output tokens return no answer text on muse-glimmer for this reason.

Use a max_tokens value of at least 1024 for chat and Messages requests, or lower reasoning_strength when the budget must stay small. The following request sets both and reads the two fields separately:

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "meta/muse-glimmer",
    "messages": [{"role": "user", "content": "Hello! How are you?"}],
    "max_tokens": 1024,
    "chat_template_kwargs": {"reasoning_strength": "low"}
  }' | jq '.choices[0] | {content: .message.content, reasoning_content: .message.reasoning_content, finish_reason}'

Use the served model name that GET /v1/models reports for your image in the model field.

A stop sequence is matched inside the reasoning section as well as in the answer, so a short stop string can end the response before any answer text is produced. Constrained output has its own limitations with this model. Refer to the muse-glimmer entries in the Release Notes.

Nemotron 3.5 Bolt E4B

The nvcr.io/nim/nvidia/nemotron-3.5-bolt-e4b-4gb:2.1.4 image sets NIM_PASSTHROUGH_ARGS to the following value, which turns on tool calling, reasoning parsing, and reasoning by default:

--enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4 --default-chat-template-kwargs {\"enable_thinking\":true}

If you set NIM_PASSTHROUGH_ARGS, your value replaces this one. Include these flags in your value to keep these features. Refer to Advanced Configuration for argument precedence and Custom Parsers and Chat Templates for parser options. To turn off reasoning for a request, set "chat_template_kwargs": {"enable_thinking": false}.

The profile includes a DFlash draft model and enables speculative decoding by default. To turn it off or change its settings, refer to Speculative Decoding.

If you start the container from a model store, which you create with create-model-store and pass in NIM_MODEL_PATH, NIM 2.1.4 does not apply the DFlash and batching settings of the profile. To keep them, save the following content as runtime_config.json at the root of the model store before you start the container:

{"speculative_config": {"method": "dflash", "model": "dflash", "num_speculative_tokens": 7, "attention_backend": "flash_attn"}, "max_num_seqs": 256, "max_num_batched_tokens": 2048}

When you select the profile with NIM_MODEL_PROFILE, including from the offline cache, NIM applies these settings without this step. Refer to Air-Gap Deployment for both methods.

On NVIDIA RTX PRO 4500 Blackwell Server Edition (32 GB), keep concurrency at 16 or lower for prompts of about 8,000 tokens. At 32 concurrent requests, time to first token increases sharply.

With audio input, include a text instruction such as “Transcribe the speech”. Without one, the model responds to the spoken content itself, and the reply varies between requests. For the request format, refer to Image, Audio, and Video Input.