Create ModelDeploymentConfig

POST

https://host.com/apis/models/v2/workspaces/:workspace/deployment-configs

POST

/apis/models/v2/workspaces/:workspace/deployment-configs

$ curl -X POST https://host.com/apis/models/v2/workspaces/workspace/deployment-configs \
>      -H "Content-Type: application/json" \
>      -d '{
>   "name": "nim-config-v1",
>   "engine": "nim",
>   "model_spec": {},
>   "executor_config": {
>     "gpu": 1
>   }
> }'

1 {
2   "name": "llama-3.1-8b",
3   "workspace": "research-team",
4   "created_at": "2024-01-15T09:30:00Z",
5   "updated_at": "2024-01-15T09:30:00Z",
6   "entity_version": 1,
7   "engine": "nim",
8   "model_spec": {
9     "model_type": "llm",
10     "model_namespace": "meta-ai",
11     "model_name": "llama-3.1-8b",
12     "model_revision": "v3.1.0",
13     "chat_template": "default_chat_template.jinja2",
14     "tool_call_config": {
15       "tool_call_parser": "openai",
16       "tool_call_plugin": "research-team/tool_plugins/custom_parser",
17       "auto_tool_choice": true
18     },
19     "lora_enabled": false
20   },
21   "executor_config": {
22     "gpu": 1,
23     "disk_size": "100Gi",
24     "image_name": "nemo/nimservice",
25     "image_tag": "v1.2.3",
26     "health_check_path": "/v1/health/ready",
27     "additional_envs": {},
28     "additional_args": [
29       "--log-level=info"
30     ],
31     "k8s_nim_operator_config": {
32       "resources": {},
33       "tolerations": [
34         {}
35       ],
36       "node_selector": {},
37       "startup_probe_grace_seconds": 600
38     },
39     "override_config": {}
40   },
41   "id": "d4f8a9b2-3c7e-4f1a-9b2d-8e5f7a1c2b3d",
42   "project": "projects/nemo-llm/deployments",
43   "description": "Deployment config for LLaMA 3.1 8B model with NIM engine",
44   "model_entity_id": "meta-ai/llama-3.1-8b"
45 }

Create a new ModelDeploymentConfig (version 1).

Path parameters

workspacestringRequired

Request

This endpoint expects an object.

namestringRequiredformat: "^[\w\-.]+$"<=255 characters

Name of the deployment configuration. Allowed characters: letters (a-z, A-Z), digits (0-9), underscores, hyphens, and dots.

engineenumRequired

Inference engine selecting the compiler path (nim/vllm/generic)

model_specobjectRequired

What model to serve and how — independent of the executor it runs on

executor_configobjectRequired

Compute + container settings for the executor the deployment runs on

projectstringOptionalformat: "^[\w\-./]+$"<=255 characters

The URN of the project associated with this deployment configuration

descriptionstringOptional<=1000 characters

Optional description of the deployment configuration

model_entity_idstringOptional<=255 characters

Optional reference to the base model entity ID for this deployment

Response

Create a new model deployment configuration

namestringformat: "^[\w\-.]+$"<=255 characters

Name of the entity. Name/workspace combo must be unique across all entities. Allowed characters: letters (a-z, A-Z), digits (0-9), underscores, hyphens, and dots.

workspacestringformat: "^[\w\-.]+$"<=255 characters

The workspace of the entity. Allowed characters: letters (a-z, A-Z), digits (0-9), underscores, hyphens, and dots.

created_atdatetime

The timestamp of model entity creation

updated_atdatetime

The timestamp of the last model entity update

entity_versioninteger

Version of this deployment config. Automatically managed.

engineenum

Inference engine selecting the compiler path (nim/vllm/generic)

model_specobject

What model to serve and how — independent of the executor it runs on

executor_configobject

Compute + container settings for the executor the deployment runs on

idstring

Unique identifier for the deployment config

projectstringformat: "^[\w\-./]+$"<=255 characters

The URN of the project associated with this entity.

descriptionstring<=1000 characters

Optional description of the deployment configuration

model_entity_idstring<=255 characters

Optional reference to the base model entity ID for this deployment

Errors

422

Unprocessable Entity Error