`core.tokenizers.text.parsers.qwen3_coder_tool_parser`#

Module Contents#

Classes#

`_Qwen3CoderToolParser`
`Qwen3CoderToolParser`	Parser for Qwen3 Coder style tool calls.
`Qwen3CoderToolCombinedParser`	Qwen3 Coder tool parser where <tool_call> also ends an open reasoning block.

Functions#

`_is_json_finite`	Whether a parsed JSON value is free of inf/nan.
`_extract_types_from_schema`	Collect every JSON Schema type a property may take.
`_coerce_to_schema_type`	Best-effort coercion of a raw string to a JSON Schema type.

Data#

`logger`
`_TYPE_ALIASES`
`_TYPE_PRIORITY`
`ToolCall`
`FunctionCall`
`ChatCompletionToolsParam`
`ChatCompletionRequest`
`ExtractedToolCallInformation`

API#

core.tokenizers.text.parsers.qwen3_coder_tool_parser.logger#: ‘getLogger(…)’

core.tokenizers.text.parsers.qwen3_coder_tool_parser._TYPE_ALIASES: dict[str, str]#: None

core.tokenizers.text.parsers.qwen3_coder_tool_parser._TYPE_PRIORITY#: (‘null’, ‘integer’, ‘number’, ‘boolean’, ‘object’, ‘array’, ‘string’)

core.tokenizers.text.parsers.qwen3_coder_tool_parser._is_json_finite(obj: Any) → bool#

Whether a parsed JSON value is free of inf/nan.

json.dumps renders those as Infinity/NaN, which is not valid JSON.

core.tokenizers.text.parsers.qwen3_coder_tool_parser._extract_types_from_schema(schema: Any) → list[str]#

Collect every JSON Schema type a property may take.

Port of extract_types_from_schema: handles type as a string or list, infers types from enum members, recurses through anyOf/oneOf/allOf, and falls back to [“string”] when nothing can be determined.

core.tokenizers.text.parsers.qwen3_coder_tool_parser._coerce_to_schema_type( value: str, schema_type: str | list[str], ) → Any#

Best-effort coercion of a raw string to a JSON Schema type.

Port of coerce_to_schema_type. Tries each declared type in priority order and returns the first that converts, then falls back to a plain JSON parse, then to the raw string. Note this means a “null” literal only becomes JSON null when the property actually admits null – for a string-typed property it stays the four characters the model wrote.

core.tokenizers.text.parsers.qwen3_coder_tool_parser.ToolCall#: None

core.tokenizers.text.parsers.qwen3_coder_tool_parser.FunctionCall#: None

core.tokenizers.text.parsers.qwen3_coder_tool_parser.ChatCompletionToolsParam#: None

core.tokenizers.text.parsers.qwen3_coder_tool_parser.ChatCompletionRequest#: None

core.tokenizers.text.parsers.qwen3_coder_tool_parser.ExtractedToolCallInformation#: None

class core.tokenizers.text.parsers.qwen3_coder_tool_parser._Qwen3CoderToolParser#

tool_call_start_token: str#: ‘<tool_call>’

tool_call_end_token: str#: ‘</tool_call>’

tool_call_prefix: str#: ‘<function=’

tool_call_complete_regex#: ‘compile(…)’

_tool_call_start_re#: ‘compile(…)’

_tool_call_end_re#: ‘compile(…)’

_parameter_start_re#: ‘compile(…)’

_parameter_end_re#: ‘compile(…)’

tool_call_function_regex#: ‘compile(…)’

tool_call_parameter_regex#: ‘compile(…)’

_generate_tool_call_id() → str#: Generate a unique tool call ID.

_get_arguments_config( func_name: str, tools: list[core.tokenizers.text.parsers.qwen3_coder_tool_parser.ChatCompletionToolsParam] | None, ) → dict#: Extract argument configuration for a function.

_convert_param_value( param_value: str, param_name: str, param_config: dict, func_name: str, ) → Any#: Convert a parameter value using its declared schema type.

_parse_xml_function_call( function_call_str: str, tools: list[core.tokenizers.text.parsers.qwen3_coder_tool_parser.ChatCompletionToolsParam] | None, finished: bool = True, ) → core.tokenizers.text.parsers.qwen3_coder_tool_parser.ToolCall | None#

_find_tool_call_end(text: str, pos: int) → int | None#

Find the </tool_call> that closes the call body starting at pos.

A </tool_call>-looking substring inside a parameter’s value must not end the call early – vLLM’s engine is a state machine that knows it is inside a parameter. Doing this with a single regex (trying, at every position, both “this looks like a parameter block” and “this is one plain character” as alternatives) is what caused catastrophic backtracking: with no </tool_call> to match, the engine tried exponentially many ways to partition the text between those two alternatives before giving up. This walks forward with bounded re searches instead – pos only ever advances and each search is bounded to a small window (the next parameter, if any), so the total work across the whole scan stays linear in the length of text even when many parameters in a row never close.

Per tool_call_parameter_regex’s grammar, a parameter is only recognised when closed by </parameter> or followed by another <parameter=; an unclosed parameter with nothing after it is simply dropped. Two cases follow from that:

A parameter that DOES eventually close absorbs everything in its value, including any </tool_call>-looking text in it, no matter how far away that close is.
A parameter that is instead abandoned for another <parameter= (never closed) does NOT retroactively absorb a </tool_call> that appeared in its dropped span – that occurrence is the real terminator, since the parameter that would have “explained it away” never actually got recognised.

Returns:: The index of </tool_call>, or None if the call is not (yet) closed.

_get_function_calls(model_output: str) → list[str]#

extract_tool_calls( model_output: str, tools: list[core.tokenizers.text.parsers.qwen3_coder_tool_parser.ChatCompletionToolsParam] | None, finished: bool = True, ) → core.tokenizers.text.parsers.qwen3_coder_tool_parser.ExtractedToolCallInformation#: Extracts the tool calls from the text using <tool_call>…</tool_call> tags.

class core.tokenizers.text.parsers.qwen3_coder_tool_parser.Qwen3CoderToolParser#

Bases: megatron.core.tokenizers.text.parsers.base_parser.BaseParser

Parser for Qwen3 Coder style tool calls.

streaming_markers#: (‘<tool_call>’, ‘<function=’)

static parse(text: str, **kwargs) → tuple[str, dict[str, list[dict]]]#

Extracts the tool calls from the text using <tool_call>…</tool_call> tags. Uses the _Qwen3CoderToolParser class (copied from vLLM) to extract the tool calls.

Parameters:: text (str) – The text to parse.
Returns:: A tuple containing the unprocessed text and a dictionary with the extracted tool calls.
Return type:: tuple[str, dict[str, str]]

class core.tokenizers.text.parsers.qwen3_coder_tool_parser.Qwen3CoderToolCombinedParser#

Bases: core.tokenizers.text.parsers.qwen3_coder_tool_parser.Qwen3CoderToolParser

Qwen3 Coder tool parser where <tool_call> also ends an open reasoning block.

qwen3-coder-tool leaves a tool call emitted before </think> inside the reasoning text, like vLLM’s standalone reasoning parsers paired with the qwen3_coder tool parser. This variant treats <tool_call> as an implicit end of reasoning and parses it, like vLLM’s combined qwen3 parser.

implicit_reasoning_end_markers#: (‘<tool_call>’,)

core.tokenizers.text.parsers.qwen3_coder_tool_parser#