MiniCPM-o 4.5 understanding#
This integration combines MiniCPM-o 4.5’s Hugging Face vision/audio encoders and projectors with Megatron’s Qwen3-compatible language decoder. It supports text, image-text, audio-text, and combined image/audio understanding inputs.
The published checkpoint is openbmb/MiniCPM-o-4_5. Its understanding stage has
approximately 9B parameters. TTS weights are excluded, and exported configurations
set init_tts=false. This is not a speech-generation or streaming model.
Reused components#
MiniCPMOBridge inherits Qwen3’s configuration and weight mappings; its builder
delegates decoder construction to GPTModelBuilder. QKV and gated-MLP conversion
use the shared mapping classes. Replicated perception gradients use the same
hook_hf_module_setattr_for_tp_grad_sync utility as Qwen-VL and Qwen-Omni.
The perception modules remain MiniCPM-o’s native vision encoder, resampler, Whisper audio encoder, and projectors. MiniCPM-o uses ordinary Qwen3 RoPE and per-sample media spans, so it does not use Qwen-VL’s MRoPE/DeepStack decoder or replace its encoders with Qwen encoders. The audio adapter only translates the native encoder’s legacy return contract and delegates attention math to HF.
Model construction#
The checkpoint’s custom code also requires the minicpmo Python module from
minicpmo-utils, even with TTS disabled. In the NeMo container, install the
understanding-stage utility package without replacing the container’s media
libraries with its older dependency pins:
uv pip install --no-deps "minicpmo-utils==1.0.7"
The H100 and GB200 functional launchers install this dependency in the same Python environment as the test. Hugging Face config/code/tokenizer cache files and this Python package are separate requirements; no pretrained weights or TTS assets are needed by the toy checkpoint test.
Use the builder API, with explicit permission to execute the checkpoint’s custom code:
import os
import torch
from megatron.bridge import AutoBridge
torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
torch.distributed.init_process_group("nccl")
bridge = AutoBridge.from_hf_pretrained(
"openbmb/MiniCPM-o-4_5", trust_remote_code=True
)
config = bridge.get_model_config()
config.tensor_model_parallel_size = 2
config.pipeline_model_parallel_size = 1
config.context_parallel_size = 1
config.sequence_parallel = False
config.bf16 = True
config.params_dtype = torch.bfloat16
models = bridge.get_model(config, load_weights=True, wrap_with_ddp=False)
Run distributed construction with one process per GPU. The model config exposes
independent freeze_language_model, freeze_vision_model, freeze_audio_model,
freeze_vision_projection, and freeze_audio_projection flags. By default,
perception modules are frozen and the language decoder is trainable. Tensor-parallel
ranks synchronize gradients of replicated perception parameters when they are
trainable.
The forward method accepts dense token IDs, positions, a boolean causal attention
mask, and media produced with the checkpoint processor. Media boundaries are
per sample; they must not be flattened across a batch. Images use pixel_values,
tgt_sizes, and image_bound; audio uses audio_features, audio_feature_lens,
and audio_bounds.
Checkpoint conversion#
See the conversion example for
shared CLI import/export commands. Keep the original checkpoint’s remote code,
tokenizer, and processor available when exporting. The exported checkpoint
contains the understanding stage only. Strict export excludes only the declared
tts. source module, including when TTS and understanding tensors share a source
shard. Missing language, vision, audio, or projector weights still fail export.
Current scope#
MiniCPM-o 4.5 only; 2.6 is rejected rather than treated as Qwen3.
Dense, non-streaming training forwards with TP; PP=CP=1 and SP disabled.
No packed sequences, MTP, quantized weights, or incremental Megatron inference cache. Generation through a separate rollout backend is a separate integration.
No SFT/LoRA recipe or speech-generation support is implied by the model entry.
Exact checkpoint round-trips and forward correlation are different checks. BF16 implementations can choose different top tokens when their logits are close; weight equality alone does not establish forward or RL correctness.