GPT OSS 20B#
GPT OSS is a Mixture-of-Experts (MoE) language model family featuring two variants: GPT OSS 20B and GPT OSS 120B. These models are designed with advanced attention mechanisms and MoE architectures optimized for long-context understanding.
The GPT OSS models feature decoder-only architectures with routed expert layers, supporting context lengths up to 128K tokens through YaRN position embeddings. Both variants use grouped-query attention and specialized attention mechanisms including sliding window attention with learnable softmax.
GPT OSS models are supported via the Bridge system with specialized configurations for MoE optimizations and long-context training.
Verified configurations#
Choose an exact recorded configuration to see its command and expected result. These selectors are generated from the authoritative verification cards and never synthesize combinations.
Run a configuration#
Choose a workflow, precision, and exact recorded combination. The command and expected result update below.
Import · CPU
â—‹ UnverifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device cpu --nodes 1 --hf-model openai/gpt-oss-20b --hf-revision 6cee5e81ee83917806bbde320786a8fb61efebee --megatron-path work/model-verification/gpt-oss-20b/cpu-megatron --torch-dtype bfloat16 --trust-remote-codeExpected result
Future verification must import the pinned MXFP4 Hugging Face checkpoint, dequantize it into a reloadable BF16 Megatron checkpoint at iter_0000000, and audit keys, shapes, dtypes, and values against the recorded source revision.
Import · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh import --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model openai/gpt-oss-20b --hf-revision 6cee5e81ee83917806bbde320786a8fb61efebee --megatron-path work/model-verification/gpt-oss-20b/imported-megatron --torch-dtype bfloat16 --tp 2 --pp 2 --ep 2 --etp 1 --trust-remote-codeExpected result
The command exits successfully, imports the pinned MXFP4 Hugging Face checkpoint on 8 GPUs, and creates iter_0000000 as a distributed TP2/PP2/EP2/ETP1 Megatron checkpoint with eight DistCP shards, metadata, train state, tokenizer artifacts, and latest-checkpoint markers.
Export · CPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device cpu --nodes 1 --hf-model unsloth/gpt-oss-20b-BF16 --hf-revision cc89b3e7fd423253264883a80a4fa5abc619649f --megatron-path work/model-verification/gpt-oss-20b/imported-megatron/iter_0000000 --hf-path work/model-verification/gpt-oss-20b/cpu-hf-export --torch-dtype bfloat16 --trust-remote-codeExpected result
The synchronous CPU workflow exports the verified distributed Megatron checkpoint to the pinned public unquantized BF16 Hugging Face layout. Its 411 tensors, shapes, dtypes, and all 20,914,757,184 values match the BF16 reference bitwise at zero tolerance. Native Transformers loading succeeds as GptOssForCausalLM with no missing, unexpected, mismatched, or errored weights, and the tokenizer reloads from the export.
Export · GPU
✓ VerifiedExact command
./scripts/conversion/convert.sh export --executor slurm --device gpu --nodes 1 --gpus-per-node 8 --hf-model unsloth/gpt-oss-20b-BF16 --megatron-path work/model-verification/gpt-oss-20b/imported-megatron/iter_0000000 --hf-path work/model-verification/gpt-oss-20b/hf-export --torch-dtype bfloat16 --export-weight-dtype bfloat16 --distributed-save --tp 2 --pp 2 --ep 2 --etp 1 --trust-remote-codeExpected result
The command exits successfully, exports the distributed Megatron checkpoint to the public unquantized BF16 Hugging Face layout, and writes a reloadable GptOssForCausalLM checkpoint with attention_bias=true. The export has 411 BF16 tensors, no missing or unexpected keys relative to unsloth/gpt-oss-20b-BF16, and all 20,914,757,184 parameters match that BF16 reference bitwise.
Pretrain · H100
✓ VerifiedRecorded metrics
- Initial loss
- 10.90364
- Final loss
- 5.725391
- Step time · last 10 avg
- 22,400.880 ms
- Model throughput · last 10 avg
- 117.010 TFLOP/s/GPU
- Token throughput · last 10 avg
- 5,851.199 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 2 --gpus-per-node 8 --recipe gpt_oss_20b_pretrain_16gpu_h100_bf16_config --mode pretrain --dataset megatron-indexed --seq_length 4096 --max_steps 100 --lr 3e-4 --min_lr 3e-5 --warmup_iters 40 'dataset.blend=[["work/data/rp2/head_01"],null]' dataset.path_to_cache=work/cache/qwen3-30b-a3b/rp2 tokenizer.tokenizer_type=SentencePieceTokenizer tokenizer.tokenizer_model=work/data/rp2/tokenizer.model dataset.random_seed=1234 rng.seed=1234 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 ddp.check_for_nan_in_grad=true ddp.check_for_large_grads=true rerun_state_machine.check_for_nan_in_loss=true checkpoint.load=null checkpoint.finetune=false checkpoint.save_optim=true checkpoint.save_rng=true --save_dir work/model-verification/gpt-oss-20b/pretrain-reference-checkpoints --save_interval 50 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
On 16x H100, the command completes exactly 100 bounded pretraining optimizer steps. All 100 optimizer-step rows are present exactly once with finite loss, zero skipped iterations, and zero NaN iterations. Loss decreases from 10.90364 to 5.725391, all five metrics are recorded, the post-setup run_config.yaml persists, and complete 16-shard iter_0000050 and iter_0000100 checkpoints are saved with metadata, train state, tokenizer artifacts, and latest-checkpoint markers suitable for direct resume.
SFT · H100
✓ VerifiedRecorded metrics
- Initial loss
- 2.130852
- Final loss
- 0.758708
- Step time · last 10 avg
- 6,618.850 ms
- Model throughput · last 10 avg
- 109.050 TFLOP/s/GPU
- Token throughput · last 10 avg
- 4,950.709 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 1 --gpus-per-node 8 --recipe gpt_oss_20b_sft_8gpu_h100_bf16_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/gpt-oss-20b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 5e-6 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="6cee5e81ee83917806bbde320786a8fb61efebee"' dataset.hf_output_root=work/data/tulu3/gpt-oss-20b-sft-2k-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true +dataset.offline_packing_specs.pad_seq_to_mult=1 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/gpt-oss-20b/sft-2k-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
On 8x H100, the command completes exactly 100 full-SFT optimizer steps from the imported GPT-OSS 20B checkpoint with immutable Tulu3 data selection, assistant-token loss masking, and offline packing. All 100 optimizer-step rows are present exactly once with finite loss, zero skipped iterations, and zero NaN iterations. Loss decreases from 2.130852 to 0.758708, all five metrics are recorded, packing is 98.56% efficient, and a complete 8-shard iter_0000100 full-model checkpoint is saved with metadata, train state, tokenizer artifacts, and latest-checkpoint markers.
Long Context · H100
✓ VerifiedRecorded metrics
- Initial loss
- 2.235548
- Final loss
- 1.175326
- Step time · last 10 avg
- 306,108.170 ms
- Model throughput · last 10 avg
- 13.430 TFLOP/s/GPU
- Token throughput · last 10 avg
- 428.189 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 1 --gpus-per-node 8 --recipe gpt_oss_20b_sft_8gpu_h100_bf16_32k_config --mode sft --dataset tulu3 --pretrained_checkpoint work/model-verification/gpt-oss-20b/imported-megatron/iter_0000000 --max_steps 20 --seq_length 32768 --tensor_model_parallel_size 4 --context_parallel_size 2 --lr 1e-6 --min_lr 0 --warmup_iters 2 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="6cee5e81ee83917806bbde320786a8fb61efebee"' dataset.hf_output_root=work/data/tulu3/gpt-oss-20b-long-context-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true +dataset.offline_packing_specs.pad_seq_to_mult=4 model.sequence_parallel=true model.cp_comm_type=a2a model.cross_entropy_loss_fusion=false model.recompute_granularity=full model.recompute_method=uniform model.recompute_num_layers=1 scheduler.lr_decay_iters=20 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null checkpoint.save=null logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
Completed 20 optimizer steps using the 32K SFT recipe, immutable Tulu3 data selection, offline packing, TP4, CP2, sequence parallelism, non-fused cross entropy, and full activation recompute. The run loaded 120 packed rows, recorded 99.79% packing efficiency with 83.333 source sequences per pack on average, produced finite loss from 2.235548 to 1.175326, and reported zero skipped and zero NaN iterations.
LoRA · H100
✓ VerifiedRecorded metrics
- Initial loss
- 2.129699
- Final loss
- 1.075515
- Step time · last 10 avg
- 50,689.940 ms
- Model throughput · last 10 avg
- 113.240 TFLOP/s/GPU
- Token throughput · last 10 avg
- 5,171.519 tokens/s/GPU
Exact command
./scripts/training/train.sh --nodes 1 --gpus-per-node 1 --recipe gpt_oss_20b_peft_1gpu_h100_bf16_config --mode lora --dataset tulu3 --pretrained_checkpoint work/model-verification/gpt-oss-20b/imported-megatron/iter_0000000 --max_steps 100 --seq_length 2048 --lr 1e-4 --min_lr 0 --warmup_iters 10 'dataset.hf_dataset.split="train[:10000]"' 'dataset.hf_dataset.load_kwargs={revision:"b14afda60f1bbebe55d5d2fa1e4df5042f97f8be"}' '++tokenizer.hf_tokenizer_kwargs.revision="6cee5e81ee83917806bbde320786a8fb61efebee"' dataset.hf_output_root=work/data/tulu3/gpt-oss-20b-peft-2k-b14afda60f1b dataset.hf_rewrite=true dataset.seed=1234 rng.seed=5678 dataset.do_validation=false dataset.hf_validation_proportion=null dataset.enable_offline_packing=true +dataset.offline_packing_specs.pad_seq_to_mult=1 scheduler.lr_decay_iters=100 validation.eval_iters=0 validation.eval_interval=0 checkpoint.load=null --save_dir work/model-verification/gpt-oss-20b/peft-2k-checkpoints --save_interval 100 logger.log_interval=1 logger.log_throughput=true logger.tensorboard_dir=nullExpected result
On 1x H100, the command completes exactly 100 LoRA PEFT optimizer steps from the imported GPT-OSS 20B checkpoint with the base model frozen, LoRA dim 32, alpha 32, dropout 0.0, target modules linear_qkv, linear_proj, linear_fc1, and linear_fc2, immutable Tulu3 data selection, assistant-token loss masking, and offline packing. All 100 optimizer-step rows are present exactly once with finite loss, zero skipped iterations, and zero NaN iterations. Loss decreases from 2.129699 to 1.075515, all five metrics are recorded, packing is 98.56% efficient, and a complete single-shard iter_0000100 adapter checkpoint is saved with metadata, train state, run_config.yaml, and latest-checkpoint markers.