Packed Sequences#

Packed sequences are a fine-tuning technique that reduces padding waste by concatenating multiple examples into one pack while preserving sequence boundaries for attention. In Megatron Bridge, this is primarily a supervised fine-tuning and PEFT optimization rather than a general pretraining feature.

This page is the stable overview for what packed sequences are, when to use them, and which constraints are durable. For operational setup, code anchors, and verification commands, see skills/nemo-mbridge-perf-sequence-packing/SKILL.md.

What It Is#

Fine-tuning datasets often contain examples with highly variable lengths. When those examples are batched conventionally, many tokens in each batch are just padding. Packed sequences reduce that waste by building longer packs from multiple examples and carrying boundary metadata into the attention path.

In Bridge today, there are four distinct packing paths plus long-context enablement through context parallelism:

Path	Use case	Key config
Offline packed SFT	Text-only finetuning	`enable_offline_packing=True` plus `offline_packing_specs`
Runtime in-batch packing	GPT-SFT JSONL, Direct Hugging Face, and supported VLM finetuning	`enable_in_batch_packing=True`
Energon online packing	Qwen-VL data using the model-owned Energon collator	`packing_buffer_size=<candidate samples per worker>`
Global-batch online packing	Variable-length SFT, packed every step across data-parallel ranks	`enable_global_batch_packing=True`
Long-context (CP)	Pretrain / finetune at 16K-128K+	`context_parallel_size > 1`

These are related but they are not the same knob. The four packing paths solve padding waste and differ in when, and over which samples, they form packs; long-context training primarily addresses activation memory and communication tradeoffs at larger sequence lengths.

The shared implementation lives under megatron.bridge.data.packing: offline GPT SFT materialization, packed Parquet runtime datasets, bin-packing algorithms, and collate-time THD packing each have separate modules. Energon online packing uses the task encoder’s native select_samples_to_pack and pack_selected_samples API while reusing the same canonical THD batch builder. Global-batch online packing keeps only its unpacked-sample contract there, because Megatron-Core’s scheduler forms its packs inside the training loop. Ordinary non-packed padding remains in megatron.bridge.data.collators. Use scripts/training/prepare_gpt_sft_packed_data.py when packed GPT SFT artifacts should be prepared before launching training.

For GPTSFTDatasetConfig, in-batch packing works with both local mmap JSONL schemas: prompt/completion (GPTSFTDataset) and chat (GPTSFTChatDataset). Tokenization remains lazy: workers mmap the JSONL and read, parse, and tokenize only the rows selected for the current logical microbatch. Collation then concatenates those rows into one physical THD batch row; it does not materialize an offline dataset or load the full source into RAM. Use a microbatch-yielding single or cyclic dataloader; GPT-SFT in-batch packing does not support the global-batch batch dataloader.

Local GPT-SFT data can also use per_split_data_source_manifest_path with MLM-style alternating ratios and JSONL paths. Offline packing resolves the weighted raw row stream first and writes one packed Parquet cache for the blended split. The input JSONL files remain memory-mapped; blending does not concatenate them into another raw file or load them all into host memory. A one-path split keeps the established single-file behavior.

When to Use It#

Packed sequences are a good fit when all of the following are true:

you are doing SFT, PEFT, or supported VLM finetuning using one of the three packing paths above
your examples have variable lengths and padding waste is significant
you can tolerate the micro-batch constraints of packed training

Packed sequences are usually not the right answer when:

you are doing standard Megatron-style pretraining, which already concatenates documents during sampling
you want long-context training in general, where context parallelism is often the main technique
your model family or recipe explicitly opts out of packed-sequence support

Choosing the Offline Pack Length#

For text-only LLM SFT and PEFT, use 8192 as the first offline-pack target when the model context limit, memory, and recipe support allow it. Compare candidate lengths at the same token slots per optimizer step:

token_slots_per_step = packed_sequence_size * global_batch_size

Thus 2K/GBS32, 4K/GBS16, and 8K/GBS8 each retain 65,536 token slots per step. A longer pack can contain more source examples in the physical MBS1 row and reduce gradient accumulation and launch overhead. It also consumes more activation memory and can encounter fixed-width kernel constraints, so measure the candidates rather than treating 8K as unconditional.

Offline packing requires MBS1. The selected GBS must be divisible by and no smaller than data parallel size; an 8K/GBS8 run therefore requires DP no larger than 8. Keep model, dataset, and packed sequence lengths equal, write changed packing configurations to a fresh output root, and verify the resolved post-setup configuration. Changing pack length can alter truncation and pack membership even when token slots stay constant, so rerun loss and stability checks before replacing existing verification evidence.

Derive the internal sequence alignment from the resolved topology for both SFT and PEFT:

cp_multiple = 2 * CP if CP > 1 else 1
sp_multiple = CP * TP if sequence parallelism is enabled and TP > 1 else 1
pad_seq_to_mult = lcm(cp_multiple, sp_multiple)

For example, TP1/CP1 with SP disabled uses 1, while TP4/CP1 with SP enabled uses 4. The difference comes from execution topology, not from whether the trainable set is full SFT or PEFT. Pin the derived value explicitly and rebuild the packed output after changing topology because the alignment changes pack membership.

Keep this internal alignment separate from fixed final pack width. pad_to_max_length=true is needed when a dispatcher or kernel requires a static width, such as a HybridEP combine kernel with a fixed token chunk, or when using CUDA graphs. CUDA graphs additionally require pad_cu_seqlens=true and packing metadata. Ordinary eager offline packing does not universally require fixed-width padding.

Choosing Runtime In-Batch Packing#

Use GPT-SFT in-batch packing when retaining the original JSONL is preferable to generating packed Parquet artifacts. Enable it directly on the dataset config and use a logical micro-batch larger than one:

dataset.enable_in_batch_packing=true
dataset.dataloader_type=single
train.micro_batch_size=4

The collator preserves each sample’s prompt/completion or chat loss mask and emits current MCore packed metadata (cu_seqlens_q, cu_seqlens_kv, and the corresponding padded boundaries when CP/SP alignment is required). The model sees one physical THD row. enable_in_batch_packing and enable_offline_packing are mutually exclusive. The batch dataloader is not supported for GPT-SFT in-batch packing; use single or cyclic.

Stable Constraints#

The durable constraints for packed sequences in Bridge are:

offline packed SFT requires configured micro_batch_size == 1
GPT-SFT/Direct-HF/VLM in-batch packing requires configured micro_batch_size > 1; collation flattens those input rows into one physical THD batch row
GPT-SFT in-batch packing requires dataloader_type="single" or "cyclic"; the global-batch "batch" dataloader is not supported
Energon online packing currently supports the eager Qwen-VL collator path, requires physical micro_batch_size == 1, the generic vlm_step, per-token loss, and ddp.average_in_collective=False
standard eager alltoall expert parallelism has functional coverage for Qwen3.6-35B-A3B at TP1/PP1/EP8 with EP communication overlap disabled; this is not a performance claim; other EP dispatchers are accepted with fixed-width native packs but do not yet have equivalent runtime evidence; THD boundaries produce a padding mask that excludes fixed-width gaps from MoE auxiliary-loss, z-loss, and expert-bias statistics
current MCore may still dispatch those padded positions; expert-capacity/token- dropping configurations do not yet have native-packing runtime coverage
packing_buffer_size counts candidate samples independently in every Energon worker; it is not a byte cache or a packed-sequence length
Energon native packing and collator-owned enable_in_batch_packing=True are mutually exclusive
Energon native packing does not currently support CUDA graphs, Qwen3-VL DistTrain, or pipeline parallelism; MTP is supported via MCore’s packed sequence boundary-aware token rolling; requested MoE expert-parallel communication overlap is disabled with a warning so training uses the non-overlapped path
when context parallelism is used, sequence length must satisfy the standard CP divisibility constraints
GPT-SFT and Direct-HF sequence length must also satisfy the LCM of the training and evaluation CP constraints and CP * TP when sequence parallelism is enabled
for fine-tuning with CP enabled, per-token loss behavior and reduction settings matter
combined recipes using offline, in-batch, or Energon native packing automatically enable safe uneven-input padding for eager HybridEP; unpacked BSHD recipes preserve their configured setting
the THD safety path pads only to the group-wide aligned maximum before dispatch and trims the padding after combine
CUDA-graph-friendly packed metadata requires additional padding constraints

Model-family support is not universal. Some families and recipe paths explicitly opt out of packed sequences or related packing modes.

HybridEP CUDA-graph configs preserve their explicit uneven-input setting because the safety path performs a host scalar synchronization that is not capture-safe. They must provide equal per-rank dispatch shapes. Disable CUDA graphs when packed THD token counts can differ so the recipe can enable safe padding. Direct model- provider callers must explicitly enable the setting when they supply uneven THD inputs because no combined recipe is available to infer their layout.

Global-Batch Online Packing#

Global-batch online packing forms packs once per training step, after a data-parallel all-gather of sample lengths, using Megatron-Core’s sequence-packing scheduler. Its candidate pool is the whole global batch across data-parallel ranks. It is the only packing path whose pool crosses rank boundaries, so it can even out packed work across ranks instead of packing each rank’s samples on their own.

Path	Switch	Packs formed	Candidate pool
Offline packed SFT	`enable_offline_packing` plus `offline_packing_specs`	ahead of time, on disk	whole dataset
Runtime in-batch packing	`enable_in_batch_packing`	at collate time	one microbatch on one rank
Energon online packing	`packing_buffer_size`	in the dataloader buffer	one worker’s buffer on one rank
Global-batch online packing	`enable_global_batch_packing`	every step, after a DP all-gather	the global batch across DP ranks

How It Works#

Every step, Megatron-Core pulls the step’s samples on each data-parallel rank, all-gathers their lengths, and uses the dp_balanced scheduler to form packs of at most model.max_seqlen_per_dp_cp_rank x context_parallel_size tokens. Each pack becomes one THD microbatch with concatenated tokens, cu_seqlens boundaries, per-sequence position ids, and variable-length attention, and it runs on the model’s context-parallel group. The number of microbatches therefore changes from step to step, and Bridge passes the scheduled count to the pipeline schedule.

Bridge wires the scheduler in through the same points the other packing paths use:

dataset.enable_global_batch_packing drives model.sequence_packing_scheduler, which defaults to dp_balanced.
The mode counts as THD input, so MoE token dispatch gets the same safe-padding treatment as the other packing paths.
It shares the runtime THD constraints listed below with Energon online packing.
The context-parallel alignment multiple comes from the same derivation the other packing paths use.
It marks the model for variable-length pipeline shapes.

train_step and evaluate wrap the raw data iterator once per step with Megatron-Core’s wrap_data_iterator and run the scheduled number of microbatches. get_batch delegates to Megatron-Core’s packed batch fetch, which returns a finished PackedSeqParams. The scheduler’s exact per-step token statistics feed the FLOPs accounting, and the logged MoE and MTP auxiliary losses are averaged over the microbatches that actually ran. The code lives in megatron.bridge.data.packing.global_batch for the sample contract and megatron.bridge.training.global_batch_packing for the training-loop glue.

Datasets That Yield Unpacked Samples#

The scheduler consumes unpacked per-sample dictionaries, one sample per micro_batch_size = 1 microbatch, delivered as a list rather than stacked into a batch. Each sample carries tokens, labels, and position_ids as int64 [L], loss_mask as float32 [L], and original_seq_len and padded_seq_len as int32 [1]. Every padded_seq_len is a multiple of the context-parallel alignment, which is 2 x context_parallel_size, times the tensor-parallel size under sequence parallelism.

Global-batch packing is available for SFT through GPTSFTDatasetConfig(enable_global_batch_packing=True). Its per-sample collate shifts labels, builds the loss mask, and pads each sequence to the alignment multiple. Use a single or cyclic dataloader. Dataset configs declare the capability through yields_unpacked_samples, which validation checks.

Configuration#

from megatron.bridge.recipes.utils.dataset_utils import default_coderforge_config

cfg.dataset = default_coderforge_config(seq_length=16384)
cfg.dataset.enable_global_batch_packing = True
cfg.dataset.dataloader_type = "single"
cfg.model.context_parallel_size = 2
cfg.model.max_seqlen_per_dp_cp_rank = 8192  # tokens per rank in one packed microbatch
cfg.model.calculate_per_token_loss = True
cfg.ddp.average_in_collective = False
cfg.train.micro_batch_size = 1

ConfigContainer.validate enforces the runtime THD constraints shared with Energon online packing: micro batch size 1, per-token loss with average_in_collective=False, no CUDA graphs, and no pipeline parallelism. It also requires:

a single or cyclic dataloader;
an explicit model.max_seqlen_per_dp_cp_rank whose product with context_parallel_size covers the longest sample;
no virtual pipeline parallelism, Mamba hybrid models, or Megatron FSDP;
no other packing path on the same dataset;
a dataset config that declares yields_unpacked_samples.

Validation also checks that the pinned Megatron-Core registers the requested scheduler, and reports what is missing before training starts.

When It Helps#

Balancing across ranks matters when there is more than one data-parallel rank. Per-rank packing can hand one rank several long samples while another gets short ones, and the lighter ranks then wait at the gradient all-reduce. The scheduler sees every rank’s samples, so it can spread that work. With a single data-parallel rank there is nothing to balance, and the mode reduces to per-step packing of the local samples.

Evaluation uses the same scheduler, so each validation pass pulls a fixed number of samples inside collective calls. The loader therefore checks up front that the validation set holds enough samples for every evaluation in the run.

Relationship to Long-Sequence Training#

Packed sequences and long-sequence training are often mentioned together because both affect sequence layout and memory behavior, but they solve different problems:

packed sequences mainly reduce padding waste in fine-tuning datasets
long-sequence training mainly addresses activation memory and communication tradeoffs at larger sequence lengths

For long-sequence training guidance, see:

docs/performance-guide.md
docs/training/hierarchical-context-parallel.md

Practical Caveats#

The most stable caveats to remember are:

Packed-sequence support is recipe- and model-family-specific.
Fine-tuning sequence packing should not be assumed to work with every other training feature.
Setting a distinct evaluation CP only reserves compatible data shapes; activating it requires decentralized process groups and caller-managed eval groups. The eval-CP example demonstrates topology plumbing, not a complete real-data recipe; validation sharding and batch math must use the eval DP.
Packed sequences improve efficiency primarily by reducing padding waste, not by replacing long-context parallelism or memory-planning techniques.
An Energon checkpoint restores the loader’s buffered samples and selected pack groups, but exact resumption still requires the same dataset, processor, topology, sequence length, and packing-buffer configuration.
progress.txt Tokens and the time/tokens runtime metric currently report configured token capacity (consumed_train_samples * model.seq_length), not exact packed-token utilization. Finite partial Energon packs can therefore make those values larger than the physical token slots executed, and neither metric excludes alignment padding to represent useful source tokens.