nat.llm.huggingface_inference_llm

Classes

HuggingFaceInferenceLLMConfig

HuggingFace Inference API LLM provider for remote model inference.

Functions

huggingface_inference_provider(config, _builder)

Register HuggingFace Inference API as an LLM provider.

Module Contents

class HuggingFaceInferenceLLMConfig(/, **data: Any)

Bases: nat.data_models.llm.LLMBaseConfig, nat.data_models.retry_mixin.RetryMixin, nat.data_models.optimizable.OptimizableMixin, nat.data_models.thinking_mixin.ThinkingMixin

HuggingFace Inference API LLM provider for remote model inference.

Supports: - Serverless Inference API (default) - Dedicated Inference Endpoints (via endpoint_url) - Self-hosted TGI servers (via endpoint_url)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_name: str = None

api_key: nat.data_models.common.OptionalSecretStr = None

endpoint_url: str | None = None

max_new_tokens: int | None = None

temperature: float | None = None

top_p: float | None = None

top_k: int | None = None

repetition_penalty: float | None = None

seed: int | None = None

timeout: float = None

async huggingface_inference_provider(

config: HuggingFaceInferenceLLMConfig,

_builder: nat.builder.builder.Builder,

)

Register HuggingFace Inference API as an LLM provider.