Is this page helpful?

Get Started with Muse Spark Open#

Muse Spark Open is a mixture-of-experts (MoE) causal language model with 42B active parameters, built for multimodal reasoning. It accepts text, image, video, and audio input through perception encoders, generates text output, and supports a 1M-token context window.

This page shows you how to deploy Muse Spark Open using the SGLang Model-Free NIM. The model-free container downloads model assets from a remote source at startup. For more information, refer to Model-Free NIM.

For the generic NIM LLM and VLM onboarding path, refer to Quickstart. For full endpoint examples, refer to API Reference. For tool calling and MCP integration, refer to Tool Calling and MCP Integration.

Support Matrix#

The following table lists the supported profile configurations for meta-models/Muse-Spark-Open-816B-A42B-NVFP4.

Precision	TP8 or EP8
NVFP4	(sglang-nvfp4-tp8-pp1)

Note that there are no packaged profile IDs for this NIM. The NIM infers the information in the preceding table.

This model has been verified on the NVIDIA-B200 GPU only.

Known Issues#

On Kubernetes, concurrent downloads of the large model weights from Hugging Face can encounter HTTP 429 responses and repeated Pod restarts. Add the following entries to the existing env list in your Helm values.yaml to limit download concurrency on each node:

env:
  - name: NIM_SDK_MAX_PARALLEL_DOWNLOAD_REQUESTS
    value: "3"
  - name: NIM_SDK_MAX_PARALLEL_FILE_REQUESTS
    value: "3"

Preserve other entries in the env list and retain download retry/backoff behavior.

Prerequisites#

Before deploying a NIM LLM and VLM container, ensure your environment meets the following requirements:

Hardware Requirements#

The following are the minimum required specifications for supported hardware components:

Requirement	Specification
CPU	AMD64, ARM64
GPU	Refer to the Support Matrix

Software Requirements#

Minimum required versions for supported software components.

Requirement	Specification
Operating System	Ubuntu 22.04 LTS or later recommended
Container Toolkit	1.14.0 or later
CUDA SDK	12.9 or later
GPU Driver	580 or later
Docker	24.0 or later

Operating System#

While other Linux distributions can be compatible with NIM, they have not been officially validated.

We recommend using Ubuntu 22.04 LTS or later for the best experience.

CUDA SDK#

Install CUDA SDK by following the CUDA installation guide for Linux.

GPU Drivers#

Install the NVIDIA GPU drivers by following the NVIDIA Driver Installation Guide.

Docker#

Docker is required to run the containerized NIM services.

Install Docker Engine for your Linux distribution by following the Docker Engine installation guide.
Verify that the Docker daemon is running and that your user can execute docker commands without sudo. Add your user to the docker group if needed:
```
sudo groupadd docker
sudo usermod -aG docker $USER
```
Log out and back in for the group change to take effect.

Container Toolkit#

The NVIDIA Container Toolkit enables Docker containers to access the host GPU.

Install the toolkit by following the NVIDIA Container Toolkit installation guide.
Configure Docker to use the NVIDIA runtime by following the Docker configuration steps.
Restart the Docker daemon after configuration:
```
sudo systemctl restart docker
```

NIM Container Access#

To download and deploy NIM containers, you need one of the following:

A free NVIDIA Developer Program membership.
An NVIDIA AI Enterprise license. To request a free 90-day evaluation license, refer to Ways to Get Started With NVIDIA AI Enterprise and Activate Your NVIDIA AI Enterprise License.

Generate Access Credentials#

An NGC Personal API key is not required to access most NVIDIA NIM containers and models hosted on NGC. An NGC API key is only required to download Production Branch (PB) models. This Get Started documentation uses a feature branch model as an example, so creating an NGC Personal API key is optional.

If you want to create an NGC Personal API key, do the following:

Generate the Personal API Key on the Setup API Keys page.
When creating the Personal API key, select at least NGC Catalog from the Services Included list. You can also include additional services if you want to use the same key for other purposes.

Warning

Legacy API keys are not supported by NIM. Always use a Personal API Key.

Verify NVIDIA Runtime Access#

To ensure that your setup is correct, run the following command:

docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

This command should produce output similar to one of the following, where you can confirm CUDA driver version, and available GPUs.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:1B:00.0 Off |                    0 |
| N/A   36C    P0            112W /  700W |   78489MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Configuration#

Use environment variables to control authentication and model caching.

Create and Export a Hugging Face Access Token#

To download models from Hugging Face, use the following steps to create and export a Hugging Face access token in your environment.

Note

If you want to serve a pre-downloaded local model or a private cloud model instead of downloading one from Hugging Face, you do not need a Hugging Face access token. Refer to Model Downloads for your workflow.

Create a Hugging Face access token with read permissions (or higher, only if your org or private repos require it). This token allows the Model-Free NIM to fetch models directly from Hugging Face.
Export the variable in your shell (temporary), replacing <token-value> with your actual token:
```
export HF_TOKEN="<token-value>"
```

Persist it (optional, for future terminals):

If using bash:

echo 'export HF_TOKEN="<token-value>"' >> ~/.bashrc

If using zsh:

echo 'export HF_TOKEN="<token-value>"' >> ~/.zshrc

Verify the variable is set:
```
echo "$HF_TOKEN"
```

Export NGC API Key#

Note

An NGC API key is only required to download Production Branch (PB) models. You can skip this section if you did not create an NGC API key.

Export the variable in your shell (temporary), replacing <VALUE> with your actual API key:
```
export NGC_API_KEY=<VALUE>
```

Persist the variable (optional):

If using bash:

echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.bashrc

If using zsh:

echo "export NGC_API_KEY=$NGC_API_KEY" >> ~/.zshrc

Verify the variable is set:
```
echo "$NGC_API_KEY"
```

Model Cache#

NIM downloads model weights to a cache on the host that you mount into the container. Artifacts persist across restarts, so you do not pull the full model on every run.

Local Cache#

An essential variable to configure on your host system is the cache path directory. This directory is mapped from the host machine to container; assets (for example, model weights) are downloaded to this host directory and persist across container restarts. Configuring a local cache is highly recommended, as it avoids re-downloading large model files upon subsequent container restarts. You can name the environment variable containing the path to the local cache whatever you want.

Create the cache directory and export an environment variable:

export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p $LOCAL_NIM_CACHE
# Optionally add sticky bit to avoid issues writing to the cache if the container is running as a different user
chmod -R a+rwxt $LOCAL_NIM_CACHE

When you start the NIM container, you must map your host machine’s local cache directory ($LOCAL_NIM_CACHE) to the container’s internal cache path (/opt/nim/.cache) using a Docker volume mount, such as -v "$LOCAL_NIM_CACHE:/opt/nim/.cache". This mapping ensures that the large model weights downloaded by the container are saved to your host machine. Because containers are ephemeral, any data stored only inside the container is lost when it stops. By using a volume mount, subsequent container runs detect the existing model files in your local cache and skip the lengthy download process, allowing the NIM to start up faster.

Cache Directory Permissions#

The NIM container runs as a non-root user with GID 0 (root group). The cache directory on your host must be writable by GID 0:

sudo chgrp -R 0 "$LOCAL_NIM_CACHE"
sudo chmod -R g+rwX "$LOCAL_NIM_CACHE"

Run the container with the cache mounted:

docker run --gpus all \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

To run as a custom user (for example, your host user), pass -u <uid>:0:

docker run --gpus all -u $(id -u):0 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  ...

Important

When using -u <uid>, you must include :0 to set GID 0 (for example, -u $(id -u):0). The container’s writable directories are group-owned by GID 0. Without it, the container will fail with PermissionError when writing to cache, config, or log paths.

Tip

To make this setting permanent across terminal sessions, you can add export LOCAL_NIM_CACHE=~/.cache/nim to your ~/.bashrc or ~/.zshrc profile.

Custom Model Source#

You can configure model-free NIM to download a model from Hugging Face or other supported sources. To do this, specify the directory where the model files should be stored and set the appropriate environment variable.

Model weights are distributed through the Hugging Face model card. The SGLang Model-Free NIM container is distributed through NGC and serves this model using the 2.1.6-variant tag.

Specify a Hugging Face model directly using the hf:// prefix.

export NIM_MODEL_PATH="hf://meta-models/Muse-Spark-Open-816B-A42B-NVFP4"

The NIM will automatically download the model from Hugging Face and cache it in your configured LOCAL_NIM_CACHE. When running the container, you will map your host machine’s local cache directory to a path inside the container using the -v flag. This ensures that the downloaded model weights persist on your host machine across ephemeral container restarts.

For example, using -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" tells Docker: “Take the folder at $LOCAL_NIM_CACHE on my host machine, and make it available inside the container at the path /opt/nim/.cache.”

Pull the Container Image#

Use the docker pull command to fetch the model-free NIM container from the NGC registry:

docker pull nvcr.io/nim/nvidia/sglang-model-free-nim-experimental:2.1.6-variant

Run NIM#

Run the container to download the model. The first run downloads about 500 GB of weights. On eight NVIDIA B200 GPUs, readiness after the download takes about 6–10 minutes, depending on the cache state.

docker run --gpus all --ipc=host --shm-size=32g \
  -e NIM_MODEL_PATH="$NIM_MODEL_PATH" \
  -e NIM_SERVED_MODEL_NAME=meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
  -e HF_TOKEN \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  nvcr.io/nim/nvidia/sglang-model-free-nim-experimental:2.1.6-variant

Multimodal Input#

For more information on including image and audio input, refer to Image, Audio, and Video Input. Inline base64 data is accepted up to 256 MiB by default. Use the NIM_MAX_REQUEST_BODY_MB environment variable.

Video input requires an external FFmpeg installation to be mounted when the container starts. FFmpeg is not included in the image. Use a recent FFmpeg 8.x build.

Add the following options to the docker run command:

-v /path/to/external-ffmpeg:/opt/external-ffmpeg:ro \
-e PATH=/opt/external-ffmpeg/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
-e LD_LIBRARY_PATH=/opt/external-ffmpeg/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib

The mounted directory must provide bin/ffmpeg, bin/ffprobe, and the required FFmpeg/libav shared libraries under lib/.

Include a software decoder in the FFmpeg bundle for every codec you serve. For AV1, use libdav1d or libaom; FFmpeg’s built-in av1 decoder is hardware-only. If a stream cannot be decoded, NIM returns HTTP 400 and names the unsupported codec in the error response.

When setting PATH and LD_LIBRARY_PATH, preserve the image’s existing CUDA and NVIDIA paths. Do not keep only the standard system paths, and do not rely on a trailing colon in LD_LIBRARY_PATH=/opt/external-ffmpeg/lib: to inherit the original value. Otherwise, CUDA components such as FlashInfer and DeepGemm can fail to find required tools or libraries during service startup.

Model Identifier#

Every request must send the served model identifier in the model field. The launch command above sets it explicitly with NIM_SERVED_MODEL_NAME, so the identifier matches the model ID: meta-models/Muse-Spark-Open-816B-A42B-NVFP4.

Interact with the API#

The NIM exposes OpenAI-compatible inference endpoints:

Chat Completions: /v1/chat/completions
Text Completions: /v1/completions
Responses: /v1/responses

Tip

Chat Completions, Text Completions, and Responses support streaming.

Send a Chat Completion Request#

After the server is running, you can send a request to the chat completion endpoint:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
    "messages": [
      {
        "role": "user",
        "content": "Hello! How are you?"
      }
    ],
    "max_tokens": 128,
    "reasoning_effort": "none"
  }'

Illustrative response structure (not a captured model response):

This request disables reasoning with reasoning_effort: "none", so the response contains the answer directly and omits reasoning_content. Actual response text and metadata vary.

{
  "id": "chatcmpl-example",
  "object": "chat.completion",
  "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! I'm ready to help. What would you like to work on?"
      },
      "finish_reason": "stop"
    }
  ],
  "system_fingerprint": "sglang-0.5.21"
}

Configure Reasoning#

reasoning_effort supports four reasoning levels: low, medium, high, and xhigh. When you omit reasoning_effort, the default is xhigh.

To disable reasoning, set "reasoning_effort": "none" as in the preceding example, or pass "chat_template_kwargs": {"enable_thinking": false}. The model then answers directly, generates no reasoning tokens, and omits reasoning_content from the response.

When reasoning is enabled, non-streaming Chat Completions responses include reasoning_content. Reasoning tokens count toward max_tokens, so allow enough tokens for both reasoning and the final answer. A small max_tokens value at the default xhigh effort can produce only reasoning and no answer text. finish_reason shows which case it is: stop for a normal answer or length if the token limit was reached.

With temperature set to 0 and reasoning_effort set to xhigh, some prompts can repeat the same reasoning until max_tokens is reached and return empty content. This is known behavior of greedy decoding with this model checkpoint, rather than evidence of a NIM or cache defect. The model card recommends temperature set to 1.0, top_p set to 1.0, and top_k disabled. For tests that require temperature set to 0, use reasoning_effort set to medium or lower. These settings reduce the observed looping but do not guarantee a final answer for every request.

For the Anthropic-compatible /v1/messages endpoint, omitting the thinking block disables reasoning. When reasoning is enabled, output_config.effort accepts the same four levels: low, medium, high, and xhigh.

Hold a Multi-Turn Conversation#

The Chat Completions endpoint is stateless. To carry context across turns, resend the full conversation in the messages array, alternating user and assistant turns after an optional system message:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "My favorite color is blue."},
      {"role": "assistant", "content": "Noted — your favorite color is blue."},
      {"role": "user", "content": "What did I just tell you?"}
    ],
    "max_tokens": 4096,
    "temperature": 1.0,
    "reasoning_effort": "low"
  }'

Note

Append only the assistant content from prior turns to the history. The reasoning parser is enabled by default; do not feed the reasoning_content field back into messages.

Verify Health Endpoints#

Verify that the NIM container is running by checking its health endpoints. By default, these endpoints are served on port 8000. If you set NIM_HEALTH_PORT, use that port instead.

Live Endpoint#

Check whether the server is running:

curl -v http://localhost:8000/v1/health/live

Example response:

GET /v1/health/live HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.81.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Content-Length: 61
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate

{
  "object": "health.response",
  "message": "live",
  "status": "live"
}

Ready Endpoint#

Check whether the model is fully loaded and ready for inference:

curl -v http://localhost:8000/v1/health/ready

Example response:

GET /v1/health/ready HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.81.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Content-Length: 63
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate

{
  "object": "health.response",
  "message": "ready",
  "status": "ready"
}

Streaming#

To receive responses incrementally as they are generated, enable streaming by adding "stream": true to your request payload. Streaming is supported across the /v1/chat/completions, /v1/completions, and /v1/responses endpoints.

When streaming is enabled, the API returns a sequence of Server-Sent Events (SSE).

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-models/Muse-Spark-Open-816B-A42B-NVFP4",
    "messages": [
      {
        "role": "user",
        "content": "Write a short poem about a robot."
      }
    ],
    "max_tokens": 4096,
    "reasoning_effort": "low",
    "stream": true
  }'

For Chat Completions and Text Completions, the response streams back in chunks. Each chunk contains a data JSON object. These streams terminate with a data: [DONE] message:

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"meta-models/Muse-Spark-Open-816B-A42B-NVFP4","choices":[{"index":0,"delta":{"role":"assistant","content":"","reasoning_content":null},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"meta-models/Muse-Spark-Open-816B-A42B-NVFP4","choices":[{"index":0,"delta":{"reasoning_content":"Hmm, the user asked"},"logprobs":null,"finish_reason":null}]}

data: {"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"meta-models/Muse-Spark-Open-816B-A42B-NVFP4","choices":[{"index":0,"delta":{"content":"Steel"},"logprobs":null,"finish_reason":null}]}

...

data: [DONE]

For the Responses API, stream events use typed SSE events such as response.output_text.delta and terminate with response.completed.