Completions#

Generate text completions using a model through the NIM Proxy microservice through a POST API call.

Prerequisites#

Before you can generate completions, make sure that you have:

  • Access to the NIM Proxy microservice through the base URL where the service is deployed. Store the base URL in an environment variable NIM_PROXY_BASE_URL.

  • A valid model name. To retrieve the list of models deployed as NIM microservices in your environment, use the ${NIM_PROXY_BASE_URL}/v1/models API. For more information, see List Models.

To Generate Completions#

Choose one of the following options of generating completions.

Create a NeMoMicroservices client instance using the base URL of the NIM Proxy microservice and perform the task as follows.

from nemo_microservices import NeMoMicroservices

client = NeMoMicroservices(
    base_url=os.environ["NEMO_BASE_URL"],
    inference_base_url=os.environ["NIM_PROXY_BASE_URL"]
)

response = client.completions.create(
    model="meta/llama-3.1-8b-instruct",
    prompt="Tell me about Cape Hatteras National Seashore in 50 words or less.",
    temperature=0.7,
    max_tokens=100,
    stream=False
)
print(response)

Make a POST request to the /v1/completions endpoint. For more details on the request body, refer to the NIM for LLMs API reference and find the API named the same as v1/completions. The NIM Proxy API endpoint routes your requests to the NIM for LLMs microservice.

curl -X POST \
  "${NIM_PROXY_BASE_URL}/v1/completions" \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "prompt": "Once upon a time",
    "max_tokens": 64
  }' | jq
Example Response
{
  "id": "cmpl-123",
  "object": "text_completion",
  "created": 1677652288,
  "model": "llama2-7b",
  "choices": [
    {
      "text": "In a sunlit studio, a robot named Pixel carefully dipped its metallic fingers into a palette of vibrant colors. Its optical sensors studied the blank canvas with intense focus, analyzing the interplay of light and shadow. With precise movements, it began to paint, each stroke a calculated expression of its growing understanding of art. The robot's journey from mechanical precision to artistic intuition was captured in the evolving masterpiece before it.",
      "index": 0,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 89,
    "total_tokens": 99
  }
}