nemo_automodel.components.models.muse_spark.processing

Native Transformers processor construction for Muse Spark Open.

Module Contents

Classes

Name	Description
`MuseSparkProcessor`	Combine the Muse Spark Open tokenizer with native multimodal preprocessing.
`MuseSparkProcessorKwargs`	Standard text/image/video/audio processing arguments.

Functions

Name	Description
`_flatten_media_groups`	Flatten per-sample nested media while retaining sample counts.
`_load_chat_template`	Load the checkpoint’s nonstandard Muse chat-template filename.
`_validate_grouped_placeholders`	Check that every sample has exactly one placeholder per media value.
`build_muse_spark_processor`	Build the native processor directly from a Muse Spark Open checkpoint.

Data

_AUDIO_SAMPLES_PER_TOKEN

_AUDIO_SAMPLE_RATE

API

class nemo_automodel.components.models.muse_spark.processing.MuseSparkProcessor(
    image_processor: nemo_automodel.components.models.muse_spark.image_processing.MuseSparkImageProcessor,
    tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase,
    chat_template: str | None = None,
    image_token: str = '<image>',
    patch_token: str = '<|patch|>',
    image_start_token: str = '<|image_start|>',
    image_end_token: str = '<|image_end|>',
    video_token: str = '<video>',
    video_patch_token: str = '<|video|>',
    video_start_token: str = '<|vid_start|>',
    video_end_token: str = '<|vid_end|>',
    video_separator_token: str = '<|vid_frame_separator|>',
    audio_token: str = '<audio>',
    audio_patch_token: str = '<|audio|>',
    audio_start_token: str = '<|soft_audio_start|>',
    audio_end_token: str = '<|soft_audio_end|>',
    timestamp_format: str = 'Time: {s:.1f}s'
)

Bases: ProcessorMixin

Combine the Muse Spark Open tokenizer with native multimodal preprocessing.

audio_token_id

= tokenizer.convert_tokens_to_ids(audio_patch_token)

image_token_id

= tokenizer.convert_tokens_to_ids(patch_token)

model_input_names

list[str]

Return model inputs produced by this processor.

video_token_id

= tokenizer.convert_tokens_to_ids(video_patch_token)

nemo_automodel.components.models.muse_spark.processing.MuseSparkProcessor.__call__(
    images: typing.Any | None = None,
    text: str | list[str] | None = None,
    videos: typing.Any | None = None,
    audio: typing.Any | None = None,
    kwargs: typing.Any = {}
) -> transformers.feature_extraction_utils.BatchFeature

Tokenize prompts after expanding image/video/audio placeholders.

Expansion goes through ProcessorMixin.get_text_with_replacements so the processor can return text_replacement_offsets; apply_chat_template needs them to keep {% generation %} spans aligned with expanded media.

class nemo_automodel.components.models.muse_spark.processing.MuseSparkProcessorKwargs()

Bases: ProcessingKwargs

Standard text/image/video/audio processing arguments.

_defaults

= {'text_kwargs': {'padding': False}}

nemo_automodel.components.models.muse_spark.processing._flatten_media_groups(
    media: typing.Any,
    batch_size: int
) -> tuple[list[typing.Any], list[int]]

Flatten per-sample nested media while retaining sample counts.

nemo_automodel.components.models.muse_spark.processing._load_chat_template(
    path_or_id: str,
    kwargs: typing.Any = {}
) -> str | None

Load the checkpoint’s nonstandard Muse chat-template filename.

nemo_automodel.components.models.muse_spark.processing._validate_grouped_placeholders(
    texts: list[str],
    placeholder: str,
    counts: list[int],
    num_replacements: int
) -> None

Check that every sample has exactly one placeholder per media value.

nemo_automodel.components.models.muse_spark.processing.build_muse_spark_processor(
    pretrained_model_name_or_path: str,
    kwargs: typing.Any = {}
) -> nemo_automodel.components.models.muse_spark.processing.MuseSparkProcessor

Build the native processor directly from a Muse Spark Open checkpoint.

nemo_automodel.components.models.muse_spark.processing._AUDIO_SAMPLES_PER_TOKEN = 1920

nemo_automodel.components.models.muse_spark.processing._AUDIO_SAMPLE_RATE = 24000