bridge.models.inkling.modeling_inkling2.ops#
Numerical operators of the Inkling2 language decoder.
The operators follow the reference implementationβs arithmetic: FP32
accumulation, and rounding back to the activation dtype only where the
reference does. Sequence tensors are sequence-first, [S, B, ...]. A
positions tensor of shape [S, B] holds each tokenβs offset inside its
document, so packed documents never exchange information through the short
convolutions.
With compiled=True an operator runs through torch.compile as one full
graph, a few fused kernels instead of a chain of FP32 passes. Dynamo builds a
graph for every new call signature (shapes, strides, source count, grad mode),
so compiled operators suit training at fixed sequence shapes.
Module Contents#
Classes#
Supplies the tokens before a context-parallel rankβs chunks to a causal short convolution. |
Functions#
Return |
|
Mix residual sources with Block Attention Residuals. |
|
Add sublayer outputs to the partial block in FP32 and round once, optionally compiled. |
|
Return |
|
Apply Inkling2βs static residual short convolution. |
|
Apply the full-attention K/V short convolution with per-token dynamic taps. |
|
Scale each vector down to |
|
Return the full-attention length scaling |
|
Softcapped SwiGLU over concatenated |
|
Return each tokenβs offset inside its document as |
|
Return a document index per token of a packed sequence (all zeros without |
Data#
API#
- bridge.models.inkling.modeling_inkling2.ops.ATTN_RES_EPS#
1e-06
- bridge.models.inkling.modeling_inkling2.ops.L2_CLIP_EPS#
1e-06
- bridge.models.inkling.modeling_inkling2.ops._RECOMPILE_LIMIT#
512
- bridge.models.inkling.modeling_inkling2.ops._ACCUMULATED_RECOMPILE_LIMIT#
8192
- bridge.models.inkling.modeling_inkling2.ops._COMPILED: dict[tuple[collections.abc.Callable, bool | None], collections.abc.Callable]#
None
- bridge.models.inkling.modeling_inkling2.ops._compiled(
- fn: collections.abc.Callable,
- *,
- dynamic: bool | None = False,
Return
fncompiled as one full graph, built once per process.
- class bridge.models.inkling.modeling_inkling2.ops.ShortConvHalo#
Bases:
typing.ProtocolSupplies the tokens before a context-parallel rankβs chunks to a causal short convolution.
- extend(x: torch.Tensor) torch.Tensor#
Return the local rows
[S_local, ...]followed by the halo rows of every rank.
- shift(
- extended: torch.Tensor,
- x: torch.Tensor,
- lag: int,
Return
x[t - lag]for every local token, zero before the start of its document.
- bridge.models.inkling.modeling_inkling2.ops.attn_res_mix(
- sources: collections.abc.Sequence[torch.Tensor],
- queries: torch.Tensor,
- *,
- normalize_values: bool,
- partial_addend: torch.Tensor | None = None,
- compiled: bool = False,
Mix residual sources with Block Attention Residuals.
Each pseudo-query attends over the sources independently at every token: keys are the RMS-normalized sources, the query is scaled by
hidden_size**-0.5, and the values are the sources themselves or, withnormalize_values, their RMS-normalized form.- Parameters:
sources β Completed block representations followed by the current partial block, each
[..., H]with identical shapes.queries β Pseudo-queries,
[H]or[Q, H].normalize_values β Whether to mix RMS-normalized values.
partial_addend β Optional sublayer output added to the last source in FP32 before mixing.
compiled β Whether to run the mixture through
torch.compile.
- Returns:
FP32 mixtures of shape
[Q, ..., H].
- bridge.models.inkling.modeling_inkling2.ops._attn_res_mix(
- sources: list[torch.Tensor],
- queries: torch.Tensor,
- *,
- normalize_values: bool,
- partial_addend: torch.Tensor | None,
- bridge.models.inkling.modeling_inkling2.ops.residual_add(
- partial: torch.Tensor,
- *addends: torch.Tensor,
- compiled: bool = False,
Add sublayer outputs to the partial block in FP32 and round once, optionally compiled.
- bridge.models.inkling.modeling_inkling2.ops._residual_add(
- partial: torch.Tensor,
- *addends: torch.Tensor,
- bridge.models.inkling.modeling_inkling2.ops._lagged(
- x: torch.Tensor,
- lag: int,
- positions: torch.Tensor,
Return
x[t - lag], zero where that token lies before the document start.
- bridge.models.inkling.modeling_inkling2.ops._shifted(
- current: torch.Tensor,
- lag: int,
- positions: torch.Tensor,
- halo: bridge.models.inkling.modeling_inkling2.ops.ShortConvHalo | None,
- extended: torch.Tensor | None,
- bridge.models.inkling.modeling_inkling2.ops.static_short_conv(
- x: torch.Tensor,
- weight: torch.Tensor,
- positions: torch.Tensor,
- *,
- halo: bridge.models.inkling.modeling_inkling2.ops.ShortConvHalo | None = None,
- compiled: bool = False,
Apply Inkling2βs static residual short convolution.
y[t, c] = (x[t, c] + sum_lag w[c, W - 1 - lag] * x[t - lag, c]) / sqrt(W + 1).- Parameters:
x β Input of shape
[S, B, C].weight β Per-channel taps of shape
[C, W](the HF[C, 1, W]layout squeezed).positions β Offsets inside each document,
[S, B].halo β Under context parallelism, supplies the tokens before the local chunks.
compiled β Whether to run the convolution through
torch.compile; with a halo, which exchanges rows between ranks, it runs eagerly.
- Returns:
Output with the dtype of
x.
- bridge.models.inkling.modeling_inkling2.ops._static_short_conv(
- x: torch.Tensor,
- weight: torch.Tensor,
- positions: torch.Tensor,
- *,
- halo: bridge.models.inkling.modeling_inkling2.ops.ShortConvHalo | None,
- bridge.models.inkling.modeling_inkling2.ops.dynamic_short_conv(
- x: torch.Tensor,
- static_weight: torch.Tensor,
- dynamic_weight: torch.Tensor,
- positions: torch.Tensor,
- *,
- halo: bridge.models.inkling.modeling_inkling2.ops.ShortConvHalo | None = None,
- compiled: bool = False,
Apply the full-attention K/V short convolution with per-token dynamic taps.
y[t, h, c] = (x[t, h, c] + sum_lag (s[lag, h, c] + d[t, h, lag]) * x[t - lag, h, c]) / sqrt(2W + 1).- Parameters:
x β Input of shape
[S, B, H * D].static_weight β Static taps of shape
[W, H * D].dynamic_weight β Dynamic taps of shape
[S, B, H, W], shared within each head.positions β Offsets inside each document,
[S, B].halo β Under context parallelism, supplies the tokens before the local chunks.
compiled β Whether to run the convolution through
torch.compile; with a halo, which exchanges rows between ranks, it runs eagerly.
- Returns:
Output with the dtype of
x.
- bridge.models.inkling.modeling_inkling2.ops._dynamic_short_conv(
- x: torch.Tensor,
- static_weight: torch.Tensor,
- dynamic_weight: torch.Tensor,
- positions: torch.Tensor,
- *,
- halo: bridge.models.inkling.modeling_inkling2.ops.ShortConvHalo | None,
- bridge.models.inkling.modeling_inkling2.ops.l2_clip(
- x: torch.Tensor,
- threshold: float,
- *,
- scale: torch.Tensor | None = None,
- output_scale: torch.Tensor | None = None,
- compiled: bool = False,
Scale each vector down to
thresholdwhen its L2 norm exceeds it.- Parameters:
x β Input of shape
[..., heads, D].threshold β Maximum norm.
scale β Optional learned per-head, per-channel scale
[heads, D]applied before clipping.output_scale β Optional FP32 per-token factor broadcast over the last two dimensions, applied after rounding the clipped vector.
compiled β Whether to run the clip through
torch.compile.
- Returns:
Output with the dtype of
x.
- bridge.models.inkling.modeling_inkling2.ops._l2_clip(
- x: torch.Tensor,
- threshold: float,
- *,
- scale: torch.Tensor | None,
- output_scale: torch.Tensor | None,
- bridge.models.inkling.modeling_inkling2.ops.length_scaling_tau(
- positions: torch.Tensor,
- n_floor: float,
- alpha: float,
Return the full-attention length scaling
1 + alpha * log(max((p + 1) / n_floor, 1)).
- bridge.models.inkling.modeling_inkling2.ops.situ_and_mul(
- inputs: torch.Tensor,
- gate_softcap: float,
- up_softcap: float,
- *,
- compiled: bool = False,
Softcapped SwiGLU over concatenated
[gate, up]halves.The sigmoid uses the raw gate, while the multiplier and the up projection are tanh soft-capped:
cap_g(g) * sigmoid(g) * cap_u(u). A zero cap disables it. Withcompiled, the token count, which changes with every batchβs routing, becomes a dynamic dimension after its first change.
- bridge.models.inkling.modeling_inkling2.ops._situ_and_mul(
- inputs: torch.Tensor,
- gate_softcap: float,
- up_softcap: float,
- bridge.models.inkling.modeling_inkling2.ops.document_positions(
- seq_len: int,
- batch: int,
- device: torch.device,
- cu_seqlens: torch.Tensor | None = None,
Return each tokenβs offset inside its document as
[S, B]int64.Without
cu_seqlensevery batch row is one document. With packed sequences (B == 1),cu_seqlensdelimits the documents; tokens past its last boundary form their own document.
- bridge.models.inkling.modeling_inkling2.ops.document_ids(
- seq_len: int,
- device: torch.device,
- cu_seqlens: torch.Tensor | None = None,
Return a document index per token of a packed sequence (all zeros without
cu_seqlens).