Is this page helpful?

Quickstart#

How to Choose the Right Model Profile#

NIM VLM automatically selects the most optimal model profile based on the detected hardware (for example, number of GPUs or GPU architecture). If you need to manually override this selection, you can set the NIM_MODEL_PROFILE environment variable. For more information, see Profile Selection.

Run NIM#

Before running a NIM VLM container, make sure you have met all prerequisites and completed installation and configuration, including setting your API keys, logging in to Docker, pulling the container image, and configuring LOCAL_NIM_CACHE.

Tip

Mounting a local cache directory lets you avoid re-downloading the model on subsequent restarts. See Local cache for details.

Run the container using your NGC API key to authenticate and download the model.

docker run --gpus=all \
  -e NGC_API_KEY=$NGC_API_KEY \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -p 8000:8000 \
  nvcr.io/nim/mistralai/mistral-small-4-119b-2603:2.0.6-variant

Interact With the API#

There are two main inference endpoints:

  • Chat Completions: /v1/chat/completions

  • Text Completions: /v1/completions

Tip

Both endpoints support streaming.

Find the Model Name#

Replace <model-name> in the examples below with the model name served by your NIM container. To find it, query the models endpoint:

curl -s http://localhost:8000/v1/models

The id field in the response is the model name to use in your requests. For model-specific NIMs, this matches the model identifier (for example, mistralai/mistral-small-4-119b-2603). For model-free NIMs, the name is derived from the container image.

Note

The model name is user-configurable by setting the NIM_SERVED_MODEL_NAME environment variable. For more information, see Environment Variables.

Send a Chat Completion Request#

Once the server is running, you can send a request to the chat completion endpoint:

curl -X 'POST' \
'http://localhost:8000/v1/chat/completions' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "model": "mistralai/mistral-small-4-119b-2603",
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What is in this image?"
                    },
                    {
                        "type": "image_url",
                        "image_url":
                            {
                                "url": "https://assets.ngc.nvidia.com/products/api-catalog/phi-3-5-vision/example1b.jpg"
                            }
                    }
                ]
            }
        ],
        "max_tokens": 1024
    }'

The expected response:

{
   "id":"chatcmpl-b26abc7313f83ba9",
   "object":"chat.completion",
   "created":1781645253,
   "model":"mistralai/mistral-small-4-119b-2603",
   "choices":[
      {
         "index":0,
         "message":{
            "role":"assistant",
            "content":"The image captures a serene, rural landscape, featuring a wooden boardwalk that stretches away from
               the viewer, inviting a journey through the scene. The boardwalk is simple, with a wooden railing on one side,
               guiding the path. It is surrounded by lush, tall green grass and vegetation, suggesting a marsh or wetland area,
               which adds a sense of wildness to the scene. The vegetation is dense and appears undisturbed, enhancing the natural beauty.
               The sky above is expansive and mostly clear, with a few wispy clouds scattered across the blue canvas, indicating a calm,
               pleasant weather. The horizon is visible in the background, with a line of trees and shrubs, and perhaps some hills,
               marking the end of the view. The overall color palette is vibrant, with the green of the grass and vegetation contrasting
               against the blue of the sky, creating a visually pleasing and tranquil scene. This image likely evokes a sense of peace
               and offers a connection to nature, making it an ideal setting for a quiet walk or contemplation.",
            "refusal":null,
            "annotations":null,
            "audio":null,
            "function_call":null,
            "tool_calls":[

            ],
            "reasoning":null
         },
         "logprobs":null,
         "finish_reason":"stop",
         "stop_reason":null,
         "token_ids":null
      }
   ],
   "service_tier":null,
   "system_fingerprint":null,
   "usage":{
      "prompt_tokens":591,
      "total_tokens":806,
      "completion_tokens":215,
      "prompt_tokens_details":null
   },
   "prompt_logprobs":null,
   "prompt_token_ids":null,
   "kv_transfer_params":null
}

Verify Health Endpoints#

You can verify that the NIM container is running and ready to accept requests by checking its health endpoints. By default, these endpoints are served on port 8000. If you set NIM_HEALTH_PORT, use that port instead.

Live Endpoint#

Perform a liveness check to see if the server is running:

curl -v http://localhost:8000/v1/health/live

Example response:

GET /v1/health/live HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.81.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Content-Length: 61
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate

{
  "object": "health.response",
  "message": "live",
  "status": "live"
}

Ready Endpoint#

Perform a readiness check to see if the model is fully loaded and ready for inference:

curl http://localhost:8000/v1/health/ready

Example response:

GET /v1/health/ready HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.81.0
Accept: */*

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Content-Type: application/json
Content-Length: 63
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate

{
  "object": "health.response",
  "message": "ready",
  "status": "ready"
}