core.activations

Module Contents

Functions

squared_relu	Squared ReLU activation
tanh_soft_clamp	Tanh Soft Clamp to precondition activation inputs.
situ	Sigmoid-Tanh Unit from Kimi K3: s * tanh(x / s) * sigmoid(x)
situ_glu	SiTU-GLU: situ(x_gate, gate_scale) * (x_linear + linear_offset).
quick_gelu	Quick GELU activation
fast_gelu	Fast GELU activation

API

core.activations.squared_relu(x: torch.Tensor) → torch.Tensor

Squared ReLU activation

core.activations.tanh_soft_clamp(x: torch.Tensor, scale: float) → torch.Tensor

Tanh Soft Clamp to precondition activation inputs.

core.activations.situ(x: torch.Tensor, scale: float) → torch.Tensor

Sigmoid-Tanh Unit from Kimi K3: s * tanh(x / s) * sigmoid(x)

core.activations.situ_glu(

x: torch.Tensor,

gate_scale: float,

linear_scale: Optional[float] = None,

linear_offset: float = 0.0,

) → torch.Tensor

SiTU-GLU: situ(x_gate, gate_scale) * (x_linear + linear_offset).

core.activations.quick_gelu(x: torch.Tensor) → torch.Tensor

Quick GELU activation

core.activations.fast_gelu(x: torch.Tensor) → torch.Tensor

Fast GELU activation