bridge.models.inkling.modeling_inkling2.ops#

Numerical operators of the Inkling2 language decoder.

The operators follow the reference implementation’s arithmetic: FP32 accumulation, and rounding back to the activation dtype only where the reference does. Sequence tensors are sequence-first, [S, B, ...]. A positions tensor of shape [S, B] holds each token’s offset inside its document, so packed documents never exchange information through the short convolutions.

With compiled=True an operator runs through torch.compile as one full graph, a few fused kernels instead of a chain of FP32 passes. Dynamo builds a graph for every new call signature (shapes, strides, source count, grad mode), so compiled operators suit training at fixed sequence shapes.

Module Contents#

Classes#

ShortConvHalo

Supplies the tokens before a context-parallel rank’s chunks to a causal short convolution.

Functions#

_compiled

Return fn compiled as one full graph, built once per process.

attn_res_mix

Mix residual sources with Block Attention Residuals.

_attn_res_mix

residual_add

Add sublayer outputs to the partial block in FP32 and round once, optionally compiled.

_residual_add

_lagged

Return x[t - lag], zero where that token lies before the document start.

_shifted

static_short_conv

Apply Inkling2’s static residual short convolution.

_static_short_conv

dynamic_short_conv

Apply the full-attention K/V short convolution with per-token dynamic taps.

_dynamic_short_conv

l2_clip

Scale each vector down to threshold when its L2 norm exceeds it.

_l2_clip

length_scaling_tau

Return the full-attention length scaling 1 + alpha * log(max((p + 1) / n_floor, 1)).

situ_and_mul

Softcapped SwiGLU over concatenated [gate, up] halves.

_situ_and_mul

document_positions

Return each token’s offset inside its document as [S, B] int64.

document_ids

Return a document index per token of a packed sequence (all zeros without cu_seqlens).

Data#

API#

bridge.models.inkling.modeling_inkling2.ops.ATTN_RES_EPS#

1e-06

bridge.models.inkling.modeling_inkling2.ops.L2_CLIP_EPS#

1e-06

bridge.models.inkling.modeling_inkling2.ops._RECOMPILE_LIMIT#

512

bridge.models.inkling.modeling_inkling2.ops._ACCUMULATED_RECOMPILE_LIMIT#

8192

bridge.models.inkling.modeling_inkling2.ops._COMPILED: dict[tuple[collections.abc.Callable, bool | None], collections.abc.Callable]#

None

bridge.models.inkling.modeling_inkling2.ops._compiled(
fn: collections.abc.Callable,
*,
dynamic: bool | None = False,
) → collections.abc.Callable#

Return fn compiled as one full graph, built once per process.

class bridge.models.inkling.modeling_inkling2.ops.ShortConvHalo#

Bases: typing.Protocol

Supplies the tokens before a context-parallel rank’s chunks to a causal short convolution.

extend(x: torch.Tensor) → torch.Tensor#

Return the local rows [S_local, ...] followed by the halo rows of every rank.

shift(
extended: torch.Tensor,
x: torch.Tensor,
lag: int,
) → torch.Tensor#

Return x[t - lag] for every local token, zero before the start of its document.

bridge.models.inkling.modeling_inkling2.ops.attn_res_mix(
sources: collections.abc.Sequence[torch.Tensor],
queries: torch.Tensor,
*,
normalize_values: bool,
partial_addend: torch.Tensor | None = None,
compiled: bool = False,
) → torch.Tensor#

Mix residual sources with Block Attention Residuals.

Each pseudo-query attends over the sources independently at every token: keys are the RMS-normalized sources, the query is scaled by hidden_size**-0.5, and the values are the sources themselves or, with normalize_values, their RMS-normalized form.

Parameters:
  • sources – Completed block representations followed by the current partial block, each [..., H] with identical shapes.

  • queries – Pseudo-queries, [H] or [Q, H].

  • normalize_values – Whether to mix RMS-normalized values.

  • partial_addend – Optional sublayer output added to the last source in FP32 before mixing.

  • compiled – Whether to run the mixture through torch.compile.

Returns:

FP32 mixtures of shape [Q, ..., H].

bridge.models.inkling.modeling_inkling2.ops._attn_res_mix(
sources: list[torch.Tensor],
queries: torch.Tensor,
*,
normalize_values: bool,
partial_addend: torch.Tensor | None,
) → torch.Tensor#
bridge.models.inkling.modeling_inkling2.ops.residual_add(
partial: torch.Tensor,
*addends: torch.Tensor,
compiled: bool = False,
) → torch.Tensor#

Add sublayer outputs to the partial block in FP32 and round once, optionally compiled.

bridge.models.inkling.modeling_inkling2.ops._residual_add(
partial: torch.Tensor,
*addends: torch.Tensor,
) → torch.Tensor#
bridge.models.inkling.modeling_inkling2.ops._lagged(
x: torch.Tensor,
lag: int,
positions: torch.Tensor,
) → torch.Tensor#

Return x[t - lag], zero where that token lies before the document start.

bridge.models.inkling.modeling_inkling2.ops._shifted(
current: torch.Tensor,
lag: int,
positions: torch.Tensor,
halo: bridge.models.inkling.modeling_inkling2.ops.ShortConvHalo | None,
extended: torch.Tensor | None,
) → torch.Tensor#
bridge.models.inkling.modeling_inkling2.ops.static_short_conv(
x: torch.Tensor,
weight: torch.Tensor,
positions: torch.Tensor,
*,
halo: bridge.models.inkling.modeling_inkling2.ops.ShortConvHalo | None = None,
compiled: bool = False,
) → torch.Tensor#

Apply Inkling2’s static residual short convolution.

y[t, c] = (x[t, c] + sum_lag w[c, W - 1 - lag] * x[t - lag, c]) / sqrt(W + 1).

Parameters:
  • x – Input of shape [S, B, C].

  • weight – Per-channel taps of shape [C, W] (the HF [C, 1, W] layout squeezed).

  • positions – Offsets inside each document, [S, B].

  • halo – Under context parallelism, supplies the tokens before the local chunks.

  • compiled – Whether to run the convolution through torch.compile; with a halo, which exchanges rows between ranks, it runs eagerly.

Returns:

Output with the dtype of x.

bridge.models.inkling.modeling_inkling2.ops._static_short_conv(
x: torch.Tensor,
weight: torch.Tensor,
positions: torch.Tensor,
*,
halo: bridge.models.inkling.modeling_inkling2.ops.ShortConvHalo | None,
) → torch.Tensor#
bridge.models.inkling.modeling_inkling2.ops.dynamic_short_conv(
x: torch.Tensor,
static_weight: torch.Tensor,
dynamic_weight: torch.Tensor,
positions: torch.Tensor,
*,
halo: bridge.models.inkling.modeling_inkling2.ops.ShortConvHalo | None = None,
compiled: bool = False,
) → torch.Tensor#

Apply the full-attention K/V short convolution with per-token dynamic taps.

y[t, h, c] = (x[t, h, c] + sum_lag (s[lag, h, c] + d[t, h, lag]) * x[t - lag, h, c]) / sqrt(2W + 1).

Parameters:
  • x – Input of shape [S, B, H * D].

  • static_weight – Static taps of shape [W, H * D].

  • dynamic_weight – Dynamic taps of shape [S, B, H, W], shared within each head.

  • positions – Offsets inside each document, [S, B].

  • halo – Under context parallelism, supplies the tokens before the local chunks.

  • compiled – Whether to run the convolution through torch.compile; with a halo, which exchanges rows between ranks, it runs eagerly.

Returns:

Output with the dtype of x.

bridge.models.inkling.modeling_inkling2.ops._dynamic_short_conv(
x: torch.Tensor,
static_weight: torch.Tensor,
dynamic_weight: torch.Tensor,
positions: torch.Tensor,
*,
halo: bridge.models.inkling.modeling_inkling2.ops.ShortConvHalo | None,
) → torch.Tensor#
bridge.models.inkling.modeling_inkling2.ops.l2_clip(
x: torch.Tensor,
threshold: float,
*,
scale: torch.Tensor | None = None,
output_scale: torch.Tensor | None = None,
compiled: bool = False,
) → torch.Tensor#

Scale each vector down to threshold when its L2 norm exceeds it.

Parameters:
  • x – Input of shape [..., heads, D].

  • threshold – Maximum norm.

  • scale – Optional learned per-head, per-channel scale [heads, D] applied before clipping.

  • output_scale – Optional FP32 per-token factor broadcast over the last two dimensions, applied after rounding the clipped vector.

  • compiled – Whether to run the clip through torch.compile.

Returns:

Output with the dtype of x.

bridge.models.inkling.modeling_inkling2.ops._l2_clip(
x: torch.Tensor,
threshold: float,
*,
scale: torch.Tensor | None,
output_scale: torch.Tensor | None,
) → torch.Tensor#
bridge.models.inkling.modeling_inkling2.ops.length_scaling_tau(
positions: torch.Tensor,
n_floor: float,
alpha: float,
) → torch.Tensor#

Return the full-attention length scaling 1 + alpha * log(max((p + 1) / n_floor, 1)).

bridge.models.inkling.modeling_inkling2.ops.situ_and_mul(
inputs: torch.Tensor,
gate_softcap: float,
up_softcap: float,
*,
compiled: bool = False,
) → torch.Tensor#

Softcapped SwiGLU over concatenated [gate, up] halves.

The sigmoid uses the raw gate, while the multiplier and the up projection are tanh soft-capped: cap_g(g) * sigmoid(g) * cap_u(u). A zero cap disables it. With compiled, the token count, which changes with every batch’s routing, becomes a dynamic dimension after its first change.

bridge.models.inkling.modeling_inkling2.ops._situ_and_mul(
inputs: torch.Tensor,
gate_softcap: float,
up_softcap: float,
) → torch.Tensor#
bridge.models.inkling.modeling_inkling2.ops.document_positions(
seq_len: int,
batch: int,
device: torch.device,
cu_seqlens: torch.Tensor | None = None,
) → torch.Tensor#

Return each token’s offset inside its document as [S, B] int64.

Without cu_seqlens every batch row is one document. With packed sequences (B == 1), cu_seqlens delimits the documents; tokens past its last boundary form their own document.

bridge.models.inkling.modeling_inkling2.ops.document_ids(
seq_len: int,
device: torch.device,
cu_seqlens: torch.Tensor | None = None,
) → torch.Tensor#

Return a document index per token of a packed sequence (all zeros without cu_seqlens).