GPT OSS 120B#
GPT OSS is a Mixture-of-Experts (MoE) language model family featuring two variants: GPT OSS 20B and GPT OSS 120B. These models are designed with advanced attention mechanisms and MoE architectures optimized for long-context understanding.
The GPT OSS models feature decoder-only architectures with routed expert layers, supporting context lengths up to 128K tokens through YaRN position embeddings. Both variants use grouped-query attention and specialized attention mechanisms including sliding window attention with learnable softmax.
GPT OSS models are supported via the Bridge system with specialized configurations for MoE optimizations and long-context training.
Verified configurations#
Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
Run a configuration#
Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
Import · CPU
â—‹ UnverifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model openai/gpt-oss-120b --hf-revision b5c939de8f754692c1647ca79fbf85e8c1e70f8a --megatron-path work/model-verification/gpt-oss-120b/cpu-megatron --torch-dtype bfloat16 --trust-remote-codeExpected result
Future verification must import the pinned MXFP4 Hugging Face checkpoint, dequantize it into a reloadable BF16 Megatron checkpoint at iter_0000000, and audit keys, shapes, dtypes, and values against the recorded source revision.
Import · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 4 --gpus-per-node 8 --hf-model openai/gpt-oss-120b --hf-revision b5c939de8f754692c1647ca79fbf85e8c1e70f8a --megatron-path work/model-verification/gpt-oss-120b/imported-megatron --torch-dtype bfloat16 --tp 1 --pp 4 --ep 8 --etp 1 --trust-remote-codeExpected result
Imports the pinned MXFP4 checkpoint on 32 H100 GPUs and creates a reloadable TP1/PP4/EP8/ETP1 BF16 Megatron checkpoint at iter_0000000 with 32 DistCP shards, metadata, train-state, tokenizer, and latest-marker artifacts.
Export · CPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model unsloth/gpt-oss-120b-BF16 --hf-revision e7523373bc44b42296b43202e265a1eebf2ee16f --megatron-path work/model-verification/gpt-oss-120b/imported-megatron/iter_0000000 --hf-path work/model-verification/gpt-oss-120b/cpu-hf-export --torch-dtype bfloat16 --trust-remote-codeExpected result
The single-process CPU export exits successfully and writes the complete BF16 Hugging Face checkpoint. Exhaustive comparison against the pinned unsloth/gpt-oss-120b-BF16 reference covers all 615 tensors and 116,829,156,672 values with exact keys, shapes, dtypes, and bitwise-equal values. Transformers strictly reloads all 615 state tensors natively as GptOssForCausalLM with attention_bias=true and no loading discrepancies, and the tokenizer also reloads successfully.
Export · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 4 --gpus-per-node 8 --hf-model unsloth/gpt-oss-120b-BF16 --megatron-path work/model-verification/gpt-oss-120b/imported-megatron/iter_0000000 --hf-path work/model-verification/gpt-oss-120b/hf-export --torch-dtype bfloat16 --export-weight-dtype bfloat16 --distributed-save --tp 1 --pp 4 --ep 8 --etp 1 --trust-remote-codeExpected result
Exports the distributed Megatron checkpoint to a reloadable BF16 Hugging Face checkpoint with attention_bias=true. The exported checkpoint has 73 safetensors shards and 615 BF16 tensors totaling 116,829,156,672 parameters, with no missing, unexpected, shape, dtype, or value mismatches against unsloth/gpt-oss-120b-BF16.
Pretrain · H100
✓ VerifiedRecorded metrics
- Initial loss
- 10.91136
- Final loss
- 5.828481
- Step time · last 10 avg
- 16,664.930 ms
- Model throughput · last 10 avg
- 58.420 TFLOP/s/GPU
- Token throughput · last 10 avg
- 1,966.285 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 8 --gpus-per-node 8 --recipe gpt_oss_120b_pretrain_config --mode pretrain --dataset megatron-indexed --seq_length 4096 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 40 'dataset.blend=[["work/data/rp2/head_01"],null]' dataset.path_to_cache=work/cache/gpt-oss-120b/rp2 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2/tokenizer.model dataset.random_seed=1234 rng.seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 model.moe_router_force_load_balancing=false ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true checkpoint.load=null checkpoint.finetune=false checkpoint.save_optim=true checkpoint.save_rng=true --save_dir work/model-verification/gpt-oss-120b/pretrain-reference-checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
On 64x H100, the command completes exactly 100 bounded pretraining optimizer steps using the recipe-owned GBS/MBS, natural routing, and BF16 configuration. All 100 optimizer-step rows are present exactly once with finite loss, zero skipped iterations, and zero NaN iterations. Loss decreases from 10.91136 to 5.828481, all five metrics are recorded, the post-setup run_config.yaml persists, and complete 64-shard iter_0000050 and iter_0000100 checkpoints are saved with metadata, train state, tokenizer artifacts, and latest-checkpoint markers suitable for direct resume.
SFT · H100
✓ VerifiedRecorded metrics
- Initial loss
- 6.751403
- Final loss
- 0.7468084
- Step time · last 10 avg
- 8,063.150 ms
- Model throughput · last 10 avg
- 33.800 TFLOP/s/GPU
- Token throughput · last 10 avg
- 1,015.980 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 4 --gpus-per-node 8 --recipe gpt_oss_120b_sft_32gpu_h100_bf16_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/gpt-oss-120b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="b5c939de8f754692c1647ca79fbf85e8c1e70f8a"' dataset.hf_output_root=work/data/tulu3/gpt-oss-120b-sft-2k-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true +dataset.offline_packing_specs.pad_seq_to_mult=1 model.tensor_model_parallel_size=2 model.sequence_parallel=true model.recompute_granularity=selective 'model.recompute_modules=[moe_act]' scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/gpt-oss-120b/sft-2k-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
Completed 100 full-SFT steps from the imported checkpoint on 32x H100 with immutable Tulu3 selection, assistant-token masking, offline packing, TP2/SP, and selective MoE-activation recompute. All 100 optimizer-step rows are present with finite losses, zero skipped iterations, and zero NaN iterations; loss decreases from 6.751403 to 0.7468084. All five metrics are recorded, the checkpoint-local post-setup run_config.yaml persists, and the final full-model checkpoint at iter_0000100 is complete with 32 DistCP shards, metadata, train state, tokenizer artifacts, and latest-checkpoint marker 100.
Long Context · H100
✓ VerifiedRecorded metrics
- Initial loss
- 6.666666
- Final loss
- 1.23803
- Step time · last 10 avg
- 24,388.780 ms
- Model throughput · last 10 avg
- 5.070 TFLOP/s/GPU
- Token throughput · last 10 avg
- 111.964 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 6 --gpus-per-node 8 --recipe gpt_oss_120b_sft_48gpu_h100_bf16_32k_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/gpt-oss-120b/imported-megatron/iter_0000000 --max_steps 20 --seq_length 32768 --lr 1e-6 --min_lr 0 --warmup_iters 2 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="b5c939de8f754692c1647ca79fbf85e8c1e70f8a"' dataset.hf_output_root=work/data/tulu3/gpt-oss-120b-sft-long-context-32k-cp4-pp6-nofuse-gbs4-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true +dataset.offline_packing_specs.pad_seq_to_mult=8 scheduler.lr_decay_iters=20 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null checkpoint.save=null logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
Completed 20 long-context full-SFT optimizer steps from the imported checkpoint on 48x H100 with 32K sequence length, immutable Tulu3 selection, assistant-token masking, offline packing, TP2/SP, CP4 with a2a context-parallel communication, PP6, EP4, disabled cross-entropy and bias-activation fusions, and selective MoE-activation recompute. All 20 optimizer-step rows are present exactly once with finite losses, zero skipped iterations, and zero NaN iterations. Loss decreases from 6.666666 to 1.238030, all five metrics are recorded, and no checkpoint is saved because this bounded long-context verification run disables checkpoint output.
LoRA · H100
✓ VerifiedRecorded metrics
- Initial loss
- 6.667154
- Final loss
- 0.9883968
- Step time · last 10 avg
- 11,509.100 ms
- Model throughput · last 10 avg
- 89.850 TFLOP/s/GPU
- Token throughput · last 10 avg
- 2,847.138 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 1 --gpus-per-node 8 --recipe gpt_oss_120b_peft_8gpu_h100_bf16_config --mode lora --dataset tulu3 --pretrained_checkpoint work/model-verification/gpt-oss-120b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 1e-4 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="b5c939de8f754692c1647ca79fbf85e8c1e70f8a"' dataset.hf_output_root=work/data/tulu3/gpt-oss-120b-peft-2k-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true +dataset.offline_packing_specs.pad_seq_to_mult=1 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/gpt-oss-120b/peft-2k-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
Completed 100 LoRA steps from the imported checkpoint with the base frozen, model-native target modules, immutable Tulu3 selection, assistant-token masking, and offline packing. All 100 optimizer-step rows are present with finite losses, zero skipped iterations, and zero NaN iterations; the checkpoint-local post-setup run_config.yaml persists and latest_checkpointed_iteration.txt points to the reloadable final adapter checkpoint at iter_0000100.
Benchmark · GB200
✓ VerifiedRecorded metrics
- Initial loss
- 12.79104
- Final loss
- 0.0324612
- Step time · last 10 avg
- 4,995.630 ms
- Model throughput · last 10 avg
- 535.230 TFLOP/s/GPU
- Token throughput · last 10 avg
- 16,398.332 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 16 --gpus-per-node 4 --recipe gpt_oss_120b_pretrain_64gpu_gb200_bf16_config --mode pretrain --max_steps 50 logger.log_throughput=trueExpected result
On exactly 64 GB200s, the canonical BF16 mock-data performance recipe completes 50 optimizer steps at TP1/PP1/CP1/EP64/ETP1, GBS/MBS 1280/4, forced router balancing, selective layernorm and MoE activation recompute, and expert-parallel communication overlap. All 50 optimizer-step rows are present exactly once with finite loss, zero skipped iterations, and zero NaN iterations. Loss decreases from 12.79104 to 0.03246120. The final ten logged step times average 4995.630 ms, measured model throughput averages 535.230 TFLOP/s/GPU, and derived token-slot throughput averages 16398.332 tokens/s/GPU over the same final-ten window. The resolved configuration persists. Mock data and forced routing make the loss a finiteness check rather than convergence evidence.
Benchmark · GB300
✓ VerifiedRecorded metrics
- Initial loss
- 12.78424
- Final loss
- 0.0985439
- Step time · last 10 avg
- 2,481.140 ms
- Model throughput · last 10 avg
- 1,077.350 TFLOP/s/GPU
- Token throughput · last 10 avg
- 33,017.081 tokens/s/GPU
Exact command
./scripts/training/train.sh --wait --nodes 16 --gpus-per-node 4 --recipe gpt_oss_120b_pretrain_64gpu_gb300_fp8mx_config --mode pretrain --max_steps 50 --seq_length 4096 logger.save_config_filepath=work/model-verification/gpt-oss-120b/gb300-performance/ConfigContainer.yamlExpected result
On exactly 64 GB300s, the canonical MXFP8 mock-data recipe completes exactly 50 optimizer steps at TP1/PP1/CP1/EP16/ETP1, GBS/MBS 1280/4, and sequence length 4096. All 50 keyed rows have finite loss with zero skipped or NaN iterations. Loss moves from 12.78424 to 0.0985439; the final ten steps average 2481.140 ms, 1077.350 TFLOP/s/GPU, and 33017.081 tokens/s/GPU. The resolved configuration persists.