Is this page helpful?

Deploy LLMs as NIM Microservices#

You can deploy LLMs as NIM microservices through the NeMo Deployment Management API. This tutorial shows how to deploy the meta/llama-3.1-8b-instruct model. You’ll use the deployed NIM microservice for fine-tuning, evaluation, and inference tasks by using the capabilities of the NeMo microservices platform in the subsequent tutorials.

Prerequisites#

Before you begin, complete the following prerequisites:

Demo Cluster Setup
(Optional) Install NeMo Microservices Python SDK
Two NVIDIA GPUs, B200 180GB, A100 80 GB, or H100 80 GB, and no other workloads running on them.

Deploy a NIM Microservice#

Start deploying the meta/llama-3.1-8b-instruct NIM microservice. You’ll use this NIM microservice for evaluation and inference tasks in the subsequent tutorials.

Note

NIM image versions 1.10 and later use "NIM_GUIDED_DECODING_BACKEND": "outlines" instead of fast_outlines.

To deploy the NIM, run the NeMo Deployment Management API as follows:

Python SDK

from nemo_microservices import NeMoMicroservices

client = NeMoMicroservices(
   base_url="http://nemo.test",
   inference_base_url="http://nim.test",
)

deployment = client.deployment.model_deployments.create(
    name="llama-3.1-8b-instruct",
    namespace="meta",
    config={
        "model": "meta/llama-3.1-8b-instruct",
        "nim_deployment": {
            "image_name": "nvcr.io/nim/meta/llama-3.1-8b-instruct",
            "image_tag": "1.8",
            "pvc_size": "25Gi",
            "gpu": 1,
            "additional_envs": {
                "NIM_GUIDED_DECODING_BACKEND": "outlines"
            }
        }
    }
)
print(deployment)

cURL

curl --location "http://nemo.test/v1/deployment/model-deployments" \
   -H 'accept: application/json' \
   -H 'Content-Type: application/json' \
   -d '{
      "name": "llama-3.1-8b-instruct",
      "namespace": "meta",
      "config": {
         "model": "meta/llama-3.1-8b-instruct",
         "nim_deployment": {
            "image_name": "nvcr.io/nim/meta/llama-3.1-8b-instruct",
            "image_tag": "1.8",
            "pvc_size":   "25Gi",
            "gpu":       1,
            "additional_envs": {
               "NIM_GUIDED_DECODING_BACKEND": "outlines"
            }
         }
      }
   }'

This returns an indicator that the deployment is pending as shown in the following output example. It might take about 10 minutes for the NIM to fully deploy.

Check the NIM status until it shows ready. Use the following command to verify the status. This requires the deployment information from the previous step so should be run in the same session.

Python SDK

# Using the deployment object from the previous step
deployment_status = client.deployment.model_deployments.retrieve(
   namespace=deployment.namespace,
   deployment_name=deployment.name
)
print(deployment_status)

cURL

curl --location "http://nemo.test/v1/deployment/model-deployments/meta/llama-3.1-8b-instruct" | jq

When deployment is complete, you can interact with it as you would with a standard NIM through the NIM Proxy microservice endpoint. For example, you can use the completion endpoint as follows.

Python SDK

Create a NeMoMicroservices client instance using the base URL of the NIM Proxy microservice and perform the task as follows.

import os
from nemo_microservices import NeMoMicroservices

client = NeMoMicroservices(base_url=os.environ["NEMO_BASE_URL"], inference_base_url=os.environ["NIM_PROXY_BASE_URL"])

response = client.chat.completions.create(
    model="meta/llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "what can you do?"}],
    temperature=0.7,
    max_tokens=100,
    stream=False,
)
for chunk in response:
    print(chunk)

cURL

Make a POST request to the /v1/chat/completions endpoint.

For more details on the request body, refer to the NIM for LLMs API reference and find the API named the same as v1/chat/completions. The NIM Proxy API endpoint routes your requests to the NIM for LLMs microservice.

curl -X POST \
  "${NIM_PROXY_BASE_URL}/v1/chat/completions" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "messages": [
        {
            "role":"user",
            "content":"Hello! How are you?"
        }
    ],
    "max_tokens": 32
  }' | jq