nemo_rl.utils.fastokens#

Config-gated integration with the fastokens Rust-backed BPE tokenizer.

Enabling policy.tokenizer.use_fastokens monkey-patches HuggingFace transformers tokenizers with fastokens’ accelerated encode/decode implementation (~10x faster BPE encoding). The patch is idempotent — calling it multiple times in the same process is a no-op after the first successful application.

The config field controls this NeMo-RL-side patch when no environment override is set. NRL_USE_FASTOKENS is the top-level NeMo-RL override and is mirrored to VLLM_USE_FASTOKENS so vLLM workers follow the same setting. A standalone VLLM_USE_FASTOKENS setting is left for vLLM and does not control this NeMo-RL-side patch.

See: https://github.com/Atero-ai/fast-tokens

Module Contents#

Functions#

normalize_fastokens_env

Mirror NeMo-RL’s fastokens override to vLLM’s fastokens flag.

maybe_patch_fastokens

Apply the fastokens monkey-patch when enabled.

Data#

logger

_patched

_NRL_ENV_VAR

_VLLM_ENV_VAR

API#

nemo_rl.utils.fastokens.logger#

‘getLogger(…)’

nemo_rl.utils.fastokens._patched#

False

nemo_rl.utils.fastokens._NRL_ENV_VAR#

‘NRL_USE_FASTOKENS’

nemo_rl.utils.fastokens._VLLM_ENV_VAR#

‘VLLM_USE_FASTOKENS’

nemo_rl.utils.fastokens.normalize_fastokens_env() → None[source]#

Mirror NeMo-RL’s fastokens override to vLLM’s fastokens flag.

nemo_rl.utils.fastokens.maybe_patch_fastokens(enabled: bool) → None[source]#

Apply the fastokens monkey-patch when enabled.

Parameters:

enabled – The resolved policy.tokenizer.use_fastokens config value. The NRL_USE_FASTOKENS env var, when set, overrides this: "1" forces on, anything else forces off.