Create LLM Endpoints

In this guide, we'll show you how to create a dedicated endpoint from LLMs with inference engine vLLM and SGLang to serve models from Hugging Face.

Create LLM Endpoints with vLLM

vLLM is a fast and easy-to-use library for LLM inference and serving. To create a dedicated endpoint with vLLM, follow these steps:

1. Go to the Endpoints page and click Create Endpoint.
2. Select Create LLM Endpoint.
3. Select vLLM as the inference engine.
4. For Endpoint name, enter vllm-endpoint or any name you prefer.
5. For Model, click Load from Hugging Face and search by keyword. For example, meta-llama/Llama-3.1-8B-Instruct. If the model is gated, provide a Hugging Face token. Create a token in your Hugging Face account and save it as a secret in your workspace.
6. For Resource, choose an appropriate resource based on the model size.
7. For Image configuration, leave defaults as is. To add arguments, expand the command-line arguments section and add your own. vLLM arguments are listed here.
8. Leave other configurations at their defaults, or refer to endpoint configurations for details.

Once created, the endpoint appears on the Endpoints page. View logs for each replica by clicking the Logs button in the Replica section.

Test the endpoint in the playground by clicking the endpoint you created.

To access the endpoint via API, click the API tab on the endpoint detail page to find the API key and endpoint URL.

Create LLM Endpoints with SGLang

SGLang is a fast serving framework for large language models and vision language models. It enables faster, more controllable interactions by co-designing the backend runtime and frontend language. To create a dedicated endpoint with SGLang, follow these steps:

1. Go to the Endpoints page and click Create Endpoint.
2. Select Create LLM Endpoint.
3. Select SGLang as the inference engine.
4. For Endpoint name, enter sglang-endpoint or any name you prefer.
5. For Model, click Load from Hugging Face and search by keyword. For example, meta-llama/Llama-3.1-8B-Instruct. If the model is gated, provide a Hugging Face token. Create a token in your Hugging Face account and save it as a secret in your workspace.
6. For Resource, choose an appropriate resource based on the model size.
7. For Image configuration, leave defaults as is. To add arguments, expand the command-line arguments section and add your own. SGLang arguments are listed here.
8. Leave other configurations at their defaults, or refer to endpoint configurations for details.

Once created, the endpoint appears on the Endpoints page. View logs for each replica by clicking the Logs button in the Replica section.

Test the endpoint in the playground by clicking the endpoint you created.

To access the endpoint via API, click the API tab on the endpoint detail page to find the API key and endpoint URL.