Model CoverageLarge Language Models

Hy-MT2 (Hunyuan-MT2)

View as Markdown

Hy-MT2-30B-A3B is Tencent’s translation Mixture-of-Experts language model with 30B total parameters and 3B activated per token. It features 48 transformer layers (layer 0 dense, layers 1–47 MoE), 128 routed experts plus 1 shared expert with top-8 sigmoid routing, Grouped Query Attention (32 Q / 4 KV heads), per-head QK RMSNorm, RoPE, and an in-forward fp32 upcast on the language-model head (enable_lm_head_fp32). It supports a 256K context window.

TaskText Generation (MoE, translation)
ArchitectureHyMT2ForCausalLM
Parameters30B total / 3B activated
HF Orgtencent

Available Models

  • Hy-MT2-30B-A3B: 30B total, top-8 routed experts (out of 128) activated per token, plus 1 shared expert

Architectures

  • HyMT2ForCausalLM

Example HF Models

ModelHF ID
Hy-MT2-30B-A3Btencent/Hy-MT2-30B-A3B

Example Recipes

RecipeDescription
hy_mt2_30b_a3b_sft.yamlSFT — Hy-MT2-30B-A3B with FSDP2 + EP8 + fp32 LM head

Try with NeMo AutoModel

1. Install (NeMo AutoModel):

$pip install nemo-automodel

2. Clone the repo to get the example recipes:

$git clone https://github.com/NVIDIA-NeMo/Automodel.git
$cd Automodel

3. Run the recipe from inside the repo:

$automodel --nproc-per-node=8 examples/llm_finetune/hy_mt2/hy_mt2_30b_a3b_sft.yaml

Refer to the NeMo AutoModel Installation Guide and LLM Fine-Tuning Guide.

Fine-Tune the Model

Refer to the LLM Fine-Tuning Guide and the Large MoE Fine-Tuning Guide.

Hugging Face Model Cards