Hy-MT2-30B-A3B

Hy-MT2-30B-A3B is Tencent’s translation Mixture-of-Experts language model with 30B total parameters and 3B activated per token. It features 48 transformer layers (layer 0 dense, layers 1-47 MoE), 128 routed experts plus 1 shared expert with top-8 sigmoid routing, Grouped Query Attention (32 Q / 4 KV heads), per-head QK RMSNorm, RoPE, and an in-forward fp32 upcast on the language-model head (enable_lm_head_fp32). It supports a 256K context window.

Set up NeMo AutoModel with the latest container or follow the installation instructions.

Fine-Tune Hy-MT2-30B-A3B

From the repository root, run:

uv run automodel --nproc-per-node=8 examples/llm_finetune/hy_mt2/hy_mt2_30b_a3b_sft.yaml

Refer to the NeMo AutoModel Installation Guide and LLM Fine-Tuning Guide.

Choose a Workflow

Model Reference

Model Architecture

Available Models

Related Resources

• LLM Fine-Tuning Guide
• Large MoE Fine-Tuning Guide