nemo_rl.data.deepseek_v4_tokenizer

Module Contents

Functions

should_use_deepseek_v4_chat_template	Return whether the config explicitly requests the DeepSeek V4 renderer.
get_deepseek_v4_tokenizer	Wrap a fast tokenizer with vLLM 0.25.1’s DeepSeek V4 encoder.

Data

DEEPSEEK_V4_CHAT_TEMPLATE

API

nemo_rl.data.deepseek_v4_tokenizer.DEEPSEEK_V4_CHAT_TEMPLATE

‘deepseek_v4’

nemo_rl.data.deepseek_v4_tokenizer.should_use_deepseek_v4_chat_template(

tokenizer_config: collections.abc.Mapping[str, Any],

) → bool

Return whether the config explicitly requests the DeepSeek V4 renderer.

nemo_rl.data.deepseek_v4_tokenizer.get_deepseek_v4_tokenizer(

tokenizer: Any,

chat_template_kwargs: dict[str, Any] | None = None,

) → Any

Wrap a fast tokenizer with vLLM 0.25.1’s DeepSeek V4 encoder.