Is this page helpful?

Run:ai#

Deploy NIM LLM and VLM on Run:ai. Use this page to create an NVIDIA NIM inference workload, optionally attach LoRA adapters, and confirm that the service is ready.

Prerequisites#

Before you start, you must complete the following prerequisites:

  1. Obtain Run:ai access (SaaS or self-hosted) with GPU capacity for inference workloads.

  2. Open a Run:ai project where you can create inference workloads.

  3. Obtain an NGC API key for pulling NIM container images and downloading model artifacts.

Note

For Run:ai platform setup and operations guidance, refer to the Welcome to NVIDIA Run:ai Documentation. For baseline Run:ai workflow details, refer to Deploy Run:ai Inference Workloads with NVIDIA NIM.

Deploy NIM LLM and VLM on Run:ai#

To deploy NIM LLM and VLM on Run:ai, complete the following steps:

  1. Create a new inference workload in the Run:ai UI.

  2. Select NVIDIA NIM as the inference type.

  3. Set the workload name and credentials.

  4. Set the required GPU count for your model.

    Tip

    Start with one GPU and scale after you confirm successful model loading and readiness.

  5. Optional: Use advanced settings to specify a custom NIM image.

  6. Optional: Create a PVC-backed (or equivalent) data source for LoRA adapters. In the Run:ai UI, open Workload Manager > Assets > Data & Storage > Data Sources, and then create the data source. When Run:ai uses a Kubernetes cluster, it creates a PVC for the data source. Populate that volume with LoRA adapter files using the same directory structure described in Enable LoRA with Helm:

    /loras/
  adapter_name/
    adapter_config.json
    adapter_model.safetensors   # or adapter_model.bin

  7. Optional: Set NIM_PEFT_SOURCE to /loras in runtime environment variables, attach the Run:ai data source, and mount it at /loras.

    Note

    If the mounted data source does not contain adapter files, NIM starts normally, but no LoRA adapters are available at runtime.

  8. Create the inference workload.

  9. Verify workload readiness in the Run:ai UI and call the readiness endpoint from a client that can reach the workload endpoint. A healthy deployment shows the inference workload in a ready state and returns an HTTP 200 response from /v1/health/ready.

Next Steps#

After the Run:ai inference workload is ready, continue with Enable LoRA with Helm for LoRA adapter layout details and KServe for a Kubernetes InferenceService deployment.