core.ops.mtp_kl_loss#
Temperature-scaled teacher-to-student KL over tensor-parallel vocabulary shards.
Module Contents#
Classes#
Functions#
Return FP32 logsumexp(logits / temperature) across the full vocabulary. |
|
Return per-token scaled KL(teacher || student), with no teacher gradients. |
API#
- core.ops.mtp_kl_loss._compile_kernel(fn)#
- core.ops.mtp_kl_loss._local_sum_exp(
- logits: torch.Tensor,
- maximum: torch.Tensor,
- scale: float,
- core.ops.mtp_kl_loss._local_kl(
- student: torch.Tensor,
- teacher: torch.Tensor,
- student_norm: torch.Tensor,
- teacher_norm: torch.Tensor,
- scale: float,
- core.ops.mtp_kl_loss._kl_grad(
- student: torch.Tensor,
- teacher: torch.Tensor,
- student_norm: torch.Tensor,
- teacher_norm: torch.Tensor,
- scale: float,
- weight: torch.Tensor,
- core.ops.mtp_kl_loss.vocab_parallel_log_normalizer(
- logits: torch.Tensor,
- temperature: float,
- tp_group: torch.distributed.ProcessGroup | None = None,
Return FP32 logsumexp(logits / temperature) across the full vocabulary.
Pass no group for gathered logits. Call under no-grad: this is metadata for the manually differentiated KL, not a differentiable distributed reduction.
- class core.ops.mtp_kl_loss._VocabParallelKL#
Bases:
torch.autograd.Function- static forward(
- ctx,
- student,
- teacher,
- teacher_norm,
- temperature,
- tp_group,
- loss_scale,
Compute scaled teacher-to-student KL loss across vocabulary shards.
- static backward(ctx, grad_output)#
Return the scaled KL gradient for student logits only.
- core.ops.mtp_kl_loss.vocab_parallel_kl_div(
- student_logits: torch.Tensor,
- teacher_logits: torch.Tensor,
- temperature: float = 1.0,
- tp_group: torch.distributed.ProcessGroup | None = None,
- teacher_log_normalizer: torch.Tensor | None = None,
- loss_scale: float | None = None,
Return per-token scaled KL(teacher || student), with no teacher gradients.
loss_scalereplaces the default temperature-squared multiplier. It must be finite and nonnegative;Nonepreserves the default T² scaling.Logits have shape [sequence, batch, vocabulary shard]; both inputs must use the same shard. The FP32 result has shape [sequence, batch]. Backward recomputes probabilities, retaining no additional vocabulary-width activations.