Is this page helpful?

core.inference.text_generation_server.dynamic_text_gen_server.vlm_dynamic_inference#

Helpers for VLM dynamic-batching inference.

Exposes the small surface that the dynamic text generation server needs to support multimodal checkpoints:

  • func:

    add_vlm_inference_args — argparse group for VLM-specific args

  • func:

    _detect_vlm_from_checkpoint — peek at saved training args and decide GPT-vs-VLM, with CLI > checkpoint > parser-default precedence

  • func:

    _print_resolved_args — diagnostic dump of the args namespace after the late checkpoint resolution above

  • func:

    get_model — build and load either a GPT or LLaVA model

The image-preprocessing helpers live in :mod:.image_preprocessing and are re-exported here for backwards compatibility with older standalone callers.

Module Contents#

Functions#

add_vlm_inference_args

Add VLM-specific inference arguments on top of the standard inference args.

_jsonable_arg_value

_arg_value_changed

_print_resolved_args

Print args after late checkpoint resolution.

_detect_vlm_from_checkpoint

Peek at the checkpoint’s saved training args to detect VLM vs GPT.

get_model

Build and load the model; dispatches to the right model_provider.

Data#

_MISSING

API#

core.inference.text_generation_server.dynamic_text_gen_server.vlm_dynamic_inference.add_vlm_inference_args(parser)#

Add VLM-specific inference arguments on top of the standard inference args.

core.inference.text_generation_server.dynamic_text_gen_server.vlm_dynamic_inference._MISSING#

‘object(…)’

core.inference.text_generation_server.dynamic_text_gen_server.vlm_dynamic_inference._jsonable_arg_value(value)#
core.inference.text_generation_server.dynamic_text_gen_server.vlm_dynamic_inference._arg_value_changed(before, after)#
core.inference.text_generation_server.dynamic_text_gen_server.vlm_dynamic_inference._print_resolved_args(title, args)#

Print args after late checkpoint resolution.

Megatron’s standard args table is emitted during parse/initialize, before this server copies VLM-only fields out of the checkpoint. Print a second table at the point where the values are the ones model construction will actually consume, followed by a per-attr provenance dump showing where each VLM-relevant value came from (CLI, checkpoint, parser default).

core.inference.text_generation_server.dynamic_text_gen_server.vlm_dynamic_inference._detect_vlm_from_checkpoint(args, user_passed_attrs=None)#

Peek at the checkpoint’s saved training args to detect VLM vs GPT.

Returns True if the checkpoint was trained as a VLM (has language_model_type), False otherwise. As a side-effect, copies VLM-specific args from the checkpoint into the current args namespace so the multimodal model_provider can access them, and records resolution provenance on args._vlm_arg_resolution for the diagnostic dump.

Precedence for each attr is CLI > checkpoint > parser default. Callers pass user_passed_attrs to indicate which attribute names the user actually typed on the command line; those values are left alone.

core.inference.text_generation_server.dynamic_text_gen_server.vlm_dynamic_inference.get_model(
is_vlm: bool,
) → megatron.core.transformer.module.MegatronModule#

Build and load the model; dispatches to the right model_provider.