Is this page helpful?

core.transformer.wide_residual_config#

Module Contents#

Classes#

WideResidualConfig

Configuration for streamwise wide residuals around ordinary-width branches.

API#

class core.transformer.wide_residual_config.WideResidualConfig#

Configuration for streamwise wide residuals around ordinary-width branches.

The model carries num_streams contiguous residual streams, each with TransformerConfig.hidden_size features. Attention, MLP, and MoE branches continue to operate at the ordinary hidden size.

num_streams: int#

None

Number of ordinary-width streams carried by the model; must be greater than one.

streamwise_sigmoid_init_scale: float#

0.01

Symmetric initialization spread for streamwise write logits.

learned_retention: bool#

False

Apply one bounded learned carry factor to each residual stream.

retention_init: float#

0.999

Initial retention factor; values near one preserve the initial function.

retention_max_forget: float#

0.1

Maximum forget rate in 1 - max_forget * sigmoid(-logit).

__post_init__() → None#