nat.llm.huggingface_inference_llm

Classes

HuggingFaceInferenceLLMConfig

HuggingFace Inference API LLM provider for remote model inference.

Functions

huggingface_inference_provider(config, _builder)

Register HuggingFace Inference API as an LLM provider.

Module Contents

class HuggingFaceInferenceLLMConfig

Bases: nat.data_models.llm.LLMBaseConfig, nat.data_models.retry_mixin.RetryMixin, nat.data_models.optimizable.OptimizableMixin, nat.data_models.thinking_mixin.ThinkingMixin

HuggingFace Inference API LLM provider for remote model inference.

Supports: - Serverless Inference API (default) - Dedicated Inference Endpoints (via endpoint_url) - Self-hosted TGI servers (via endpoint_url)

model_config

model_name: str = None

api_key: nat.data_models.common.OptionalSecretStr = None

endpoint_url: str | None = None

max_new_tokens: int | None

temperature: float | None

top_p: float | None

top_k: int | None = None

repetition_penalty: float | None = None

seed: int | None = None

timeout: float = None

async huggingface_inference_provider(

config: HuggingFaceInferenceLLMConfig,

_builder: nat.builder.builder.Builder,

)

Register HuggingFace Inference API as an LLM provider.