Is this page helpful?

core.models.vision.encoder_registry#

Central registry of per-encoder defaults.

One EncoderSpec per vision_model_type carries everything callers need to instantiate the encoder without hard-coding numbers in multiple places:

  • patch_dim, default image size, class_token_len, native-spatial-merge flag

  • implementation model type, default converted checkpoint directory

  • pixel_mean / pixel_std (ImageNet-style normalisation)

  • the full TransformerConfig arch (num_layers, hidden_size, ffn_hidden_size, activation, normalisation, bias flags, RoPE flags, …)

Consumers:

  • examples/multimodal/config.py::get_vision_model_config (via apply_to_config)

  • examples/multimodal/v3/energon_multimodal_provider.py (pixel statistics)

  • examples/multimodal/multimodal_args.py::resolve_multimodal_encoder_args

Adding a new encoder = one entry here.

Module Contents#

Classes#

EncoderSpec

Per-encoder defaults: image geometry, pixel stats, transformer arch.

Functions#

_gelu_tanh

_resolve_activation

_radio_h_spec

get_spec

Return the spec for vision_model_type, or raise KeyError with a list of known types.

Data#

_CLIP_MEAN

_CLIP_STD

_IN_MEAN

_IN_STD

_HALF_MEAN

_HALF_STD

ActivationFunc

REGISTRY

API#

core.models.vision.encoder_registry._CLIP_MEAN#

(0.48145466, 0.4578275, 0.40821073)

core.models.vision.encoder_registry._CLIP_STD#

(0.26862954, 0.26130258, 0.27577711)

core.models.vision.encoder_registry._IN_MEAN#

(0.485, 0.456, 0.406)

core.models.vision.encoder_registry._IN_STD#

(0.229, 0.224, 0.225)

core.models.vision.encoder_registry._HALF_MEAN#

(0.5, 0.5, 0.5)

core.models.vision.encoder_registry._HALF_STD#

(0.5, 0.5, 0.5)

core.models.vision.encoder_registry.ActivationFunc#

None

core.models.vision.encoder_registry._gelu_tanh(x)#
core.models.vision.encoder_registry._resolve_activation(
activation_func: core.models.vision.encoder_registry.ActivationFunc,
) → Callable#
class core.models.vision.encoder_registry.EncoderSpec#

Per-encoder defaults: image geometry, pixel stats, transformer arch.

name: str#

None

patch_dim: int#

None

default_img_h: int#

None

default_img_w: int#

None

class_token_len: int#

0

has_native_spatial_merge: bool#

False

model_type: Optional[str]#

None

checkpoint_dir: Optional[str]#

None

dynamic_resolution: bool#

False

pixel_shuffle: bool#

False

conv_merging: bool#

False

use_tiling: bool#

False

max_num_tiles: int#

1

use_thumbnail: bool#

False

dynamic_resolution_max_patches: int#

0

dynamic_resolution_max_side: Optional[int]#

None

radio_force_eval_mode: bool#

False

radio_hf_resolution: bool#

False

pixel_mean: Tuple[float, float, float]#

None

pixel_std: Tuple[float, float, float]#

None

num_layers: Optional[int]#

None

hidden_size: Optional[int]#

None

num_attention_heads: Optional[int]#

None

num_query_groups: Optional[int]#

None

ffn_hidden_size: Optional[int]#

None

kv_channels: Optional[int]#

None

gated_linear_unit: bool#

False

activation_func: core.models.vision.encoder_registry.ActivationFunc#

‘gelu’

add_bias_linear: bool#

True

add_qkv_bias: bool#

True

normalization: str#

‘LayerNorm’

layernorm_epsilon: Optional[float]#

None

qk_layernorm: Optional[bool]#

None

rotary_interleaved: bool#

False

tp_round_up_heads: bool#

False

apply_to_config(config, apply_query_key_layer_scaling: bool = False)#

Write this spec onto an existing TransformerConfig and return it.

core.models.vision.encoder_registry._radio_h_spec(
name: str,
checkpoint_dir: str,
*,
dynamic_resolution: bool,
pixel_shuffle: bool,
use_tiling: bool,
max_num_tiles: int = 1,
use_thumbnail: bool = False,
radio_hf_resolution: bool = False,
) → core.models.vision.encoder_registry.EncoderSpec#
core.models.vision.encoder_registry.REGISTRY: Dict[str, core.models.vision.encoder_registry.EncoderSpec]#

None

core.models.vision.encoder_registry.get_spec(
vision_model_type: str,
) → core.models.vision.encoder_registry.EncoderSpec#

Return the spec for vision_model_type, or raise KeyError with a list of known types.