nemo_export.tensorrt_llm_hf

TensorRT-LLM HuggingFace export functionality has been removed.

This module now only contains placeholder functions that raise NotImplementedError. TensorRT-LLM export support has been deprecated and removed from this codebase.

Module Contents

Classes

TensorRTLLMHF

Placeholder class for TensorRT-LLM HuggingFace export functionality.

Data

LOGGER

API

nemo_export.tensorrt_llm_hf.LOGGER = 'getLogger(...)'

class nemo_export.tensorrt_llm_hf.TensorRTLLMHF(

model_dir: str,

lora_ckpt_list: List[str] = None,

load_model: bool = True,

use_python_runtime: bool = True,

enable_chunked_context: bool = None,

max_tokens_in_paged_kv_cache: int = None,

multi_block_mode: bool = False,

)

Bases: nemo_export.tensorrt_llm.TensorRTLLM

Placeholder class for TensorRT-LLM HuggingFace export functionality.

Note: TensorRT-LLM export support has been removed from this codebase. All methods will raise NotImplementedError.

Initialization

Initialize TensorRTLLMHF exporter.

Raises:

NotImplementedError – This functionality has been removed.

abstractmethod export_hf_model(

hf_model_path: str,

max_batch_size: int = 8,

tensor_parallelism_size: int = 1,

max_input_len: int = 256,

max_output_len: Optional[int] = None,

max_num_tokens: Optional[int] = None,

opt_num_tokens: Optional[int] = None,

dtype: Optional[str] = None,

max_seq_len: Optional[int] = 512,

gemm_plugin: str = 'auto',

remove_input_padding: bool = True,

use_paged_context_fmha: bool = True,

paged_kv_cache: bool = True,

multiple_profiles: bool = False,

reduce_fusion: bool = False,

model_type: Optional[str] = None,

delete_existing_files: bool = True,

)

Export HuggingFace model to TensorRT-LLM.

Raises:

NotImplementedError – This functionality has been removed.

abstractmethod get_hf_model_type(hf_model_path: str) → str

Get HuggingFace model type.

Raises:

NotImplementedError – This functionality has been removed.

abstractmethod get_hf_model_dtype(hf_model_path: str) → str

Get HuggingFace model dtype.

Raises:

NotImplementedError – This functionality has been removed.

abstract property get_supported_hf_model_mapping

Get supported HuggingFace model mapping.

Raises:

NotImplementedError – This functionality has been removed.