List Models

List all available models through the NIM Proxy microservice by running a GET API call.

Types of Models Auto-detected by NIM Proxy

• Fine-tuned models uploaded by NeMo Customizer.

• NIM deployed with the NeMo Deployment Management microservice.

• NIM deployed using Helm with the following label added to the spec object of the NIMService custom resource:

  spec:
     labels:
        app.nvidia.com/nim-type: inference
  

Prerequisites

Before you can list models, make sure that you have:

• Access to the NIM Proxy microservice through the base URL where the service is deployed, either as part of the platform or standalone. Store the base URL in an environment variable NIM_PROXY_BASE_URL.

To List Models

Choose one of the following options of listing models.

cURL

Make a GET request to the /v1/models endpoint. For more details on the request body, see the NIM Proxy API reference.

curl -X GET \
  "${NIM_PROXY_BASE_URL}/v1/models" \
  -H 'accept: application/json' | jq

Python SDK

Use the following Python SDK code to list models.

from nemo_microservices import NeMoMicroservices

client = NeMoMicroservices(
    base_url=os.environ["NEMO_BASE_URL"],
    inference_base_url=os.environ["NIM_PROXY_BASE_URL"]
)

response = client.inference.models.list()

Example Response

{
  "data": [
    {
      "id": "meta/llama-3.1-8b-instruct",
      "object": "model",
      "created": 1677652288,
      "owned_by": "meta",
      "permission": [],
      "root": "meta/llama-3.1-8b-instruct",
      "parent": null
    },
    {
      "id": "meta/llama-3.1-70b-instruct",
      "object": "model",
      "created": 1677652288,
      "owned_by": "meta",
      "permission": [],
      "root": "meta/llama-3.1-70b-instruct",
      "parent": null
    }
  ],
  "object": "list"
}