nemo_automodel.components.models.inkling.processing

Native processor construction and multimodal token replacement for Inkling.

Module Contents

Classes

Name	Description
`InklingProcessor`	Combine the native Inkling image/audio processors with a tokenizer.
`InklingProcessorKwargs`	Processor kwargs with Inkling’s reference audio-loading default.

Functions

Name	Description
`build_inkling_processor`	Load Inkling’s native processor without Transformers model registration.

Data

_INKLING_END_OF_SAMPLING_TOKEN

__all__

API

class nemo_automodel.components.models.inkling.processing.InklingProcessor(
    feature_extractor: nemo_automodel.components.models.inkling.feature_extraction.InklingFeatureExtractor,
    image_processor: nemo_automodel.components.models.inkling.image_processing.InklingImageProcessor,
    tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase,
    chat_template: str | None = None,
    image_token: str = '<|unused_200054|>',
    audio_token: str = '<|unused_200053|>',
    image_bos_token: str = '<|content_image|>',
    audio_bos_token: str = '<|content_audio_input|>',
    num_dmel_bins: int = 16,
    dmel_min_value: float = -7.0,
    dmel_max_value: float = 2.0,
    kwargs: typing.Any = {}
)

Bases: ProcessorMixin

Combine the native Inkling image/audio processors with a tokenizer.

audio_bos_token_id

audio_token

= getattr(tokenizer, 'audio_token', audio_token)

audio_token_id

bin_centers

image_bos_token_id

image_token

= getattr(tokenizer, 'image_token', image_token)

image_token_id

model_input_names

list[str]

Return the deduplicated model input field names.

unused_input_names

list[str]

Return processor-only fields omitted from model inputs.

nemo_automodel.components.models.inkling.processing.InklingProcessor.__call__(
    images: typing.Any | None = None,
    text: str | list[str] | None = None,
    videos: typing.Any | None = None,
    audio: typing.Any | None = None,
    kwargs: typing.Any = {}
) -> transformers.feature_extraction_utils.BatchFeature

Prepare text, image patches, and dMel tokens without version-specific HF hooks.

Transformers releases before Inkling was upstreamed dispatch audio directly to the feature extractor and do not expand multimodal placeholders. Owning this small dispatcher keeps the checkpoint usable with AutoModel’s pinned Transformers version.

nemo_automodel.components.models.inkling.processing.InklingProcessor._extract_dmel_bins(
    input_features: torch.Tensor
) -> torch.Tensor

Quantize continuous log-mel values into dMel token IDs.

Parameters:

input_features

torch.Tensor

Tensor of shape [batch, frames, mel_bins].

Returns: torch.Tensor

Int tensor of shape [batch, frames, mel_bins].

nemo_automodel.components.models.inkling.processing.InklingProcessor._normalize_audio_batch(
    audio: typing.Any
) -> list[typing.Any]

staticmethod

Normalize one waveform or a batch into a list of clips.

nemo_automodel.components.models.inkling.processing.InklingProcessor._process_audio(
    audio: typing.Any,
    kwargs: typing.Any = {}
) -> tuple[dict[str, torch.Tensor], list[str]]

Extract, quantize, and count a batch of audio clips.

nemo_automodel.components.models.inkling.processing.InklingProcessor._replace_multimodal_tokens(
    text: list[str],
    image_replacements: list[str],
    audio_replacements: list[str]
) -> list[str]

Expand media placeholders once per corresponding input, in batch order.

nemo_automodel.components.models.inkling.processing.InklingProcessor.replace_audio_token(
    audio_inputs: dict[str, torch.Tensor],
    audio_idx: int
) -> str

Return one soft placeholder per valid audio frame.

nemo_automodel.components.models.inkling.processing.InklingProcessor.replace_image_token(
    image_inputs: dict[str, torch.Tensor],
    image_idx: int
) -> str

Return one soft placeholder per encoded image patch.

class nemo_automodel.components.models.inkling.processing.InklingProcessorKwargs()

Bases: ProcessingKwargs

Processor kwargs with Inkling’s reference audio-loading default.

_defaults

nemo_automodel.components.models.inkling.processing.build_inkling_processor(
    pretrained_model_name_or_path: str,
    kwargs: typing.Any = {}
) -> nemo_automodel.components.models.inkling.processing.InklingProcessor

Load Inkling’s native processor without Transformers model registration.

Parameters:

pretrained_model_name_or_path

str

Hugging Face model ID or local snapshot.

**kwargs

AnyDefaults to {}

Download/cache arguments accepted by Transformers from_pretrained methods.

Returns: InklingProcessor

A configured native Inkling processor.

nemo_automodel.components.models.inkling.processing._INKLING_END_OF_SAMPLING_TOKEN = '<|content_model_end_sampling|>'

nemo_automodel.components.models.inkling.processing.__all__ = ['InklingProcessor', 'build_inkling_processor']