Is this page helpful?

Magpie-TTS Longform Inference#

This document describes how longform (multi-sentence) text-to-speech inference works in Magpie-TTS.

Overview#

Magpie-TTS supports generating speech for long text inputs by processing them in smaller, sentence-level chunks while maintaining prosodic continuity across the entire utterance. This approach overcomes the context window limitations of the underlying transformer architecture.

When Longform is Used#

Longform inference is automatically triggered based on word count thresholds (approximately 20 seconds of audio):

Language Word Thresholds#

Language

Word Threshold

English

45 words

Spanish

73 words

French

69 words

German

50 words

Italian

53 words

Vietnamese

50 words

Japanese

50 words

Hindi

50 words

Note

Longform is best supported for English. Mandarin currently falls back to standard inference.

Algorithm#

The longform inference algorithm processes long text through the following steps:

Step 1: Sentence Splitting#

The input text is split into individual sentences using punctuation markers (., ?, !, ...). The splitting is intelligent and handles abbreviations like “Dr.”, “Mr.”, “a.m.” by checking if the period is followed by a space.

Example:

Input:  "Dr. Smith arrived early. How are you today?"
Output: ["Dr. Smith arrived early.", "How are you today?"]

Step 2: State Initialization#

A ChunkState object is created to track information across sentence chunks:

  • History text tokens: Text from previous chunks for context

  • History encoder context: Encoder outputs that provide continuity

  • Attention tracking: Monitors which positions have been attended to

Step 3: Iterative Chunk Processing#

For each sentence chunk, the following sub-steps are performed:

  1. Context Preparation: Prepend history text and encoder context from previous chunks to maintain prosodic continuity.

  2. Attention Prior Application: Apply a learned attention prior that guides the model to attend to the correct text positions, preventing repetition or skipping.

  3. Autoregressive Generation: Generate audio codes token-by-token using the transformer decoder with temperature sampling.

  4. State Update: Update the chunk state with:

    • New history text (last N tokens)

    • New encoder context

    • Updated attention tracking

  5. Code Collection: Store the generated audio codes for this chunk.

Step 4: Code Concatenation#

After all chunks are processed, concatenate the audio codes from each chunk along the time dimension into a single sequence.

Step 5: Audio Decoding#

Pass the concatenated codes through the neural audio codec decoder to produce the final waveform.

Key Components#

  1. Sentence Splitting (split_by_sentence): Intelligently splits text on sentence boundaries while handling abbreviations (e.g., “Dr.”, “Mr.”).

  2. Chunk State (ChunkState): Maintains context across chunks:

    • history_text: Text tokens from previous chunks

    • history_context_tensor: Encoder outputs for continuity

    • last_attended_timesteps: Attention tracking for smooth transitions

  3. Attention Prior: Guides the model’s attention to maintain proper alignment and prevent repetition/skipping.

Usage#

Method 2: Using CLI (magpietts_inference.py)#

For batch inference from manifests:

# Auto-detect longform based on text length (default)
python examples/tts/magpietts_inference.py \
    --nemo_files /path/to/magpietts.nemo \
    --datasets_json_path /path/to/evalset_config.json \
    --out_dir /path/to/output \
    --codecmodel_path /path/to/codec.nemo \
    --longform_mode auto

# Force longform inference for all inputs
python examples/tts/magpietts_inference.py \
    --nemo_files /path/to/magpietts.nemo \
    --datasets_json_path /path/to/evalset_config.json \
    --out_dir /path/to/output \
    --codecmodel_path /path/to/codec.nemo \
    --longform_mode always \
    --longform_max_decoder_steps 50000

Longform CLI Options:

Option

Default

Description

--longform_mode

auto

auto: detect from text, always: force longform, never: disable

Configuration Dataclasses#

ModelInferenceParameters#

Model inference parameters, including long-form and chunked-inference tuning values. These can be set in the model’s inference_parameters configuration:

@dataclass
class ModelInferenceParameters:
    """Model specific parameters that are sent to inference functions.

    This dataclass should contain all parameters that are model specific and should not change on a per run basis.

    Attributes:
        max_decoder_steps (int): Maximum number of decoder steps. Autoregressive for loop will terminate here.
        temperature (float): Sampling temperature.
        topk (int): Number of top-probability tokens to consider in sampling.
        cfg_scale (float): Scale factor for classifier-free guidance. Only used if use_cfg=True.
        apply_attention_prior (bool): Whether to apply attention prior.
        attention_prior_epsilon (float): Base probability for non-targeted positions.
        attention_prior_lookahead_window (int): Size of the forward-looking window to search for the next attended
            timestep. Determines how far ahead from the last attended timestep to look.
        estimate_alignment_from_layers (Optional[List[int]]): Layers to use for alignment estimation.
        apply_prior_to_layers (Optional[List[int]]): Layers to apply prior to.
        start_prior_after_n_audio_steps (int): Which step to start enabling the attention prior.
        use_LT_kv_cache (bool): Whether to use KV cache for the autoregressive local transformer.
        ignore_finished_sentence_tracking (bool): Whether to ignore finished sentence tracking.
        eos_detection_method (str): EOS detection method. See the EOSDetectionMethod class.
        min_generated_frames (int): Setting this greater than 0 prevents rare cases of first-frame termination. Any
            number greater between 1 and 4 should work, but 4 lines up with the codec's minimum frame requirement.
        attention_sink_threshold (int): Times a position may be attended before standard inference advances past it.
        history_len_heuristic (int): Maximum history tokens retained across text chunks.
        prior_weights_init (Tuple[float, ...]): Attention prior weights used when initializing a new chunk.
        prior_weights (Tuple[float, ...]): Attention prior weights used during chunked generation.
        finished_limit_with_eot (int): Near-end steps before allowing EOS in the final chunk.
        finished_limit_without_eot (int): Near-end steps before allowing EOS in a non-final chunk.
        finished_limit_first_chunk (int): Near-end steps before allowing EOS in the first chunk.
        forceful_chunk_end_threshold (int): Near-end steps before forcibly ending a non-final chunk.
        argmax_temperature (float): Temperature used for the argmax EOS-detection sample.
        short_sentence_threshold (int): Texts at or below this length use a uniform chunked attention prior.
        chunked_attention_sink_threshold (int): Times a position may be attended before the chunked prior penalizes it.
        near_end_threshold (int): Positions from the text end that are treated as near the end.
    """

    max_decoder_steps: int = 500
    temperature: float = 0.7
    topk: int = 80
    cfg_scale: float = 2.5
    apply_attention_prior: bool = True
    attention_prior_epsilon: float = 0.1
    attention_prior_lookahead_window: int = 5
    estimate_alignment_from_layers: Optional[List[int]] = None
    apply_prior_to_layers: Optional[List[int]] = None
    start_prior_after_n_audio_steps: int = 0
    use_LT_kv_cache: bool = True
    ignore_finished_sentence_tracking: bool = True
    eos_detection_method: str = "argmax_or_multinomial_any"
    min_generated_frames: int = 4
    attention_sink_threshold: int = 8
    history_len_heuristic: int = 20
    prior_weights_init: Tuple[float, ...] = (0.5, 1.0, 0.8, 0.2, 0.2)
    prior_weights: Tuple[float, ...] = (0.2, 1.0, 0.6, 0.4, 0.2, 0.2)
    finished_limit_with_eot: int = 5
    finished_limit_without_eot: int = 1
    finished_limit_first_chunk: int = 20
    forceful_chunk_end_threshold: int = 3
    argmax_temperature: float = 0.01
    short_sentence_threshold: int = 35
    chunked_attention_sink_threshold: int = 10
    near_end_threshold: int = 3

    @classmethod
    def from_dict(cls, data: dict) -> 'ModelInferenceParameters':
        # Get the names of fields defined in the dataclass
        field_names = {field.name for field in fields(cls)}
        # Filter the input dictionary to include only valid fields
        filtered_data = {k: v for k, v in data.items() if k in field_names}
        # Instantiate the dataclass with the filtered data

        # Double check for renamed fields: prior_epsilon and lookahead_window_size
        # These fields are currently used in nvidia/magpie_tts_multilingual_357m with commit hash: 291da79
        if 'prior_epsilon' in data:
            filtered_data['attention_prior_epsilon'] = data['prior_epsilon']
        if 'lookahead_window_size' in data:
            filtered_data['attention_prior_lookahead_window'] = data['lookahead_window_size']
        for field_name in ('prior_weights_init', 'prior_weights'):
            if field_name in filtered_data:
                filtered_data[field_name] = tuple(filtered_data[field_name])
        return cls(**filtered_data)

ChunkState#

Mutable state passed between chunk iterations:

@dataclass
class ChunkState:
    """Mutable state persisting across chunks during chunked generation.

    Created by the inference runner via model.create_chunk_state(),
    passed to generate_speech(), and updated in-place across chunk iterations.

    Attributes:
        batch_size: Number of items in the batch.
        history_text: Text tokens from previous chunks. Shape: (B, T).
        history_text_lens: Lengths of history text per batch item. Shape: (B,).
        history_context_tensor: Encoder output from previous chunks. Shape: (B, T, E).
        end_indices: Maps batch indices to overall timestep where they ended.
        overall_idx: Global timestep counter across all chunks.
        left_offset: Sliding window offset per batch item for attention tracking.
        previous_attn_len: Attention lengths from previous chunk per batch item.
        last_attended_timesteps: Tracking of attended positions across decoding.
    """

    batch_size: int
    history_text: Optional[torch.Tensor] = None
    history_text_lens: Optional[torch.Tensor] = None
    history_context_tensor: Optional[torch.Tensor] = None
    end_indices: Dict[int, int] = field(default_factory=dict)
    overall_idx: int = 0
    left_offset: List[int] = field(default_factory=list)
    previous_attn_len: List[int] = field(default_factory=list)
    last_attended_timesteps: List[List[int]] = field(default_factory=list)

    def __post_init__(self):
        """Initialize batch-sized lists if not provided."""
        if not self.left_offset:
            self.left_offset = [0] * self.batch_size
        if not self.last_attended_timesteps:
            self.last_attended_timesteps = [[1] * self.batch_size]

Best Practices#

  1. Use ``apply_TN=True`` for raw text to ensure proper normalization before synthesis.

  2. Increase ``max_decoder_steps`` for very long texts (default 50000 is usually sufficient).

  3. Use ``longform_mode=”auto”`` (default) to let the system decide based on text length.

  4. For non-English languages, be aware that longform performance may vary. English is best supported.

Limitations#

  • Mandarin (zh): Currently falls back to standard inference due to character-based tokenization complexities.

  • Prosodic boundaries: While the algorithm maintains continuity, natural paragraph breaks may not always be perfectly preserved in non-English languages.

See Also#