`nemo_rl.data.xtoken.native_chat`#

Typed native ChatML rendering and assistant supervision for distillation.

The semantic parser supports the Qwen/Nano ChatML layouts. It deliberately rejects changed or missing source regions instead of guessing a loss mask.

Module Contents#

Classes#

RenderedChatDocument

One unpadded native rendering with source-bound loss/alignment metadata.

Functions#

`_render_chat_text`	Render training context without dropping required template controls.
`_prepare_native_thinking_messages`	Copy source messages and expose leading reasoning separately from prose.
`_final_content_span`	Locate the final answer while keeping the full history in both renders.
`_eot_index`
`_tokenize_chat_text`	Tokenize the full conversation and supervise only its final assistant turn.
`_reject_embedded_chatml_markers`	Reject ambiguous literal turn delimiters in the supported native parser.
`_render_and_tokenize_chat`	Render the full chat and supervise its final assistant message only.
`_render_native_tool_call`	Render one complete native call for exact binding to its original turn.
`_native_scalar_spans`	Locate direct bool/null values without parsing markup inside arguments.
`_native_answer_parts_for_document`	Bind structured calls to independent KD coordinates, preserving CE.

Data#

`_THINK_PATTERN`
`_IM_START_ASSISTANT`
`_IM_END`

API#

nemo_rl.data.xtoken.native_chat._THINK_PATTERN#: ‘compile(…)’

nemo_rl.data.xtoken.native_chat._IM_START_ASSISTANT#: ‘<|im_start|>assistant\n’

nemo_rl.data.xtoken.native_chat._IM_END#: ‘<|im_end|>’

class nemo_rl.data.xtoken.native_chat.RenderedChatDocument#

One unpadded native rendering with source-bound loss/alignment metadata.

rendered_text: str#: None

input_ids: list[int]#: None

offsets: list[tuple[int, int]]#: None

assistant_mask: list[int]#: None

assistant_spans: list[tuple[int, int]]#: None

eot_indices: list[int]#: None

source_turn_indices: list[int]#: None

alignment_regions: list[nemo_rl.algorithms.x_token.token_aligner.NativeAlignmentRegions]#: None

answer_parts: list[list[nemo_rl.algorithms.x_token.token_aligner.NativeAlignmentPart] | None]#: None

nemo_rl.data.xtoken.native_chat._render_chat_text( tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, messages: list[dict[str, Any]], *, preserve_thinking: bool, tools: list[dict[str, Any]] | None = None, ) → str#: Render training context without dropping required template controls.

nemo_rl.data.xtoken.native_chat._prepare_native_thinking_messages( messages: list[dict[str, Any]], ) → list[dict[str, Any]]#: Copy source messages and expose leading reasoning separately from prose.

nemo_rl.data.xtoken.native_chat._final_content_span( tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, messages: list[dict[str, Any]], text: str, *, preserve_thinking: bool, tools: list[dict[str, Any]] | None, ) → tuple[int, int, str] | None#: Locate the final answer while keeping the full history in both renders.

nemo_rl.data.xtoken.native_chat._eot_index( offsets: list[tuple[int, int]], text: str, position: int, ) → int#

nemo_rl.data.xtoken.native_chat._tokenize_chat_text( tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, text: str, messages: list[dict[str, Any]], message_loss_mask: list[int], max_len: int, *, include_thinking_in_loss: bool, native_semantic_messages: list[dict[str, Any]] | None, tools: list[dict[str, Any]] | None = None, ) → nemo_rl.data.xtoken.native_chat.RenderedChatDocument#

Tokenize the full conversation and supervise only its final assistant turn.

Reject documents that exceed the context limit instead of truncating them.

nemo_rl.data.xtoken.native_chat._reject_embedded_chatml_markers(value: Any) → None#: Reject ambiguous literal turn delimiters in the supported native parser.

nemo_rl.data.xtoken.native_chat._render_and_tokenize_chat( tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, messages: list[dict[str, Any]], max_len: int, *, tools: list[dict[str, Any]] | None = None, message_loss_mask: list[int], include_thinking_in_loss: bool, native_thinking_alignment: bool, ) → nemo_rl.data.xtoken.native_chat.RenderedChatDocument#

Render the full chat and supervise its final assistant message only.

The processor validates the incoming message mask. Earlier selections are context only here; the final message must be an assistant selected for loss.

nemo_rl.data.xtoken.native_chat._render_native_tool_call( tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, tool_call: Dict[str, Any], ) → str#

Render one complete native call for exact binding to its original turn.

No schema or prose is supplied, so literal tool markup inside argument values cannot be mistaken for the outer call’s first/last delimiters. The auxiliary render never replaces the model’s actual input.

nemo_rl.data.xtoken.native_chat._native_scalar_spans( tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, tool_call: Dict[str, Any], rendered_call: str, ) → List[Tuple[str, Tuple[int, int]]]#: Locate direct bool/null values without parsing markup inside arguments.

nemo_rl.data.xtoken.native_chat._native_answer_parts_for_document( tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, messages: list[dict[str, Any]], selected: list[int], document: nemo_rl.data.xtoken.native_chat.RenderedChatDocument, ) → List[List[nemo_rl.algorithms.x_token.token_aligner.NativeAlignmentPart] | None]#: Bind structured calls to independent KD coordinates, preserving CE.

nemo_rl.data.xtoken.native_chat#