core.inference.moe

Submodules

• core.inference.moe.activations
• core.inference.moe.fused_moe
• core.inference.moe.permute
• core.inference.moe.vllm_fused_moe
• core.inference.moe.metadata

Package Contents

Classes

InferenceGroupedGemmBackend

Backend for grouped GEMM operations during inference.

API

class core.inference.moe.InferenceGroupedGemmBackend(*args, **kwds)

Bases: enum.Enum

Backend for grouped GEMM operations during inference.

The string value matches the inference_grouped_gemm_backend config field so TransformerConfig.post_init can convert via InferenceGroupedGemmBackend(str).

Initialization

FLASHINFER

‘flashinfer’

TORCH

‘torch’

VLLM

‘vllm’