Hy3-preview

Hy3-preview is a 295B Mixture-of-Experts language model from Tencent. It features 80 transformer layers (layer 0 dense, layers 1-79 MoE), 192 routed experts plus 1 shared expert with top-8 sigmoid routing, Grouped Query Attention (64 Q / 8 KV heads), per-head QK RMSNorm, RoPE, and an e_score_correction_bias gate buffer for expert-load correction. It supports a 256K context window.

Set up NeMo AutoModel with the latest container or follow the installation instructions.

Fine-Tune Hy3-preview

From the repository root, run:

uv run automodel --nproc-per-node=8 examples/llm_finetune/hy_v3/hy3_preview_deepep.yaml

See the NeMo AutoModel Installation Guide and LLM Fine-Tuning Guide.

Choose a Workflow

Model Reference

Model Architecture

Available Models

Related Resources

• LLM Fine-Tuning Guide
• Large MoE Fine-Tuning Guide