core.ops.mtp_kl_loss#

Temperature-scaled teacher-to-student KL over tensor-parallel vocabulary shards.

Module Contents#

Classes#

Functions#

_compile_kernel

_local_sum_exp

_local_kl

_kl_grad

vocab_parallel_log_normalizer

Return FP32 logsumexp(logits / temperature) across the full vocabulary.

vocab_parallel_kl_div

Return per-token scaled KL(teacher || student), with no teacher gradients.

API#

core.ops.mtp_kl_loss._compile_kernel(fn)#
core.ops.mtp_kl_loss._local_sum_exp(
logits: torch.Tensor,
maximum: torch.Tensor,
scale: float,
) → torch.Tensor#
core.ops.mtp_kl_loss._local_kl(
student: torch.Tensor,
teacher: torch.Tensor,
student_norm: torch.Tensor,
teacher_norm: torch.Tensor,
scale: float,
) → torch.Tensor#
core.ops.mtp_kl_loss._kl_grad(
student: torch.Tensor,
teacher: torch.Tensor,
student_norm: torch.Tensor,
teacher_norm: torch.Tensor,
scale: float,
weight: torch.Tensor,
) → torch.Tensor#
core.ops.mtp_kl_loss.vocab_parallel_log_normalizer(
logits: torch.Tensor,
temperature: float,
tp_group: torch.distributed.ProcessGroup | None = None,
) → torch.Tensor#

Return FP32 logsumexp(logits / temperature) across the full vocabulary.

Pass no group for gathered logits. Call under no-grad: this is metadata for the manually differentiated KL, not a differentiable distributed reduction.

class core.ops.mtp_kl_loss._VocabParallelKL#

Bases: torch.autograd.Function

static forward(
ctx,
student,
teacher,
teacher_norm,
temperature,
tp_group,
loss_scale,
)#

Compute scaled teacher-to-student KL loss across vocabulary shards.

static backward(ctx, grad_output)#

Return the scaled KL gradient for student logits only.

core.ops.mtp_kl_loss.vocab_parallel_kl_div(
student_logits: torch.Tensor,
teacher_logits: torch.Tensor,
temperature: float = 1.0,
tp_group: torch.distributed.ProcessGroup | None = None,
teacher_log_normalizer: torch.Tensor | None = None,
loss_scale: float | None = None,
) → torch.Tensor#

Return per-token scaled KL(teacher || student), with no teacher gradients.

loss_scale replaces the default temperature-squared multiplier. It must be finite and nonnegative; None preserves the default T² scaling.

Logits have shape [sequence, batch, vocabulary shard]; both inputs must use the same shard. The FP32 result has shape [sequence, batch]. Backward recomputes probabilities, retaining no additional vocabulary-width activations.