nemo_rl.utils.fastokens#

Config-gated integration with the fastokens Rust-backed BPE tokenizer.

Enabling policy.tokenizer.use_fastokens monkey-patches HuggingFace transformers tokenizers with fastokens’ accelerated encode/decode implementation (~10x faster BPE encoding). The patch is idempotent — calling it multiple times in the same process is a no-op after the first successful application.

The config field is the source of truth. The NRL_USE_FASTOKENS environment variable, when set, overrides the config as an escape hatch for toggling without editing YAML: NRL_USE_FASTOKENS=1 forces on, any other value forces off.

See: https://github.com/Atero-ai/fast-tokens

Module Contents#

Functions#

maybe_patch_fastokens

Apply the fastokens monkey-patch when enabled.

Data#

logger

_patched

API#

nemo_rl.utils.fastokens.logger#

‘getLogger(…)’

nemo_rl.utils.fastokens._patched#

False

nemo_rl.utils.fastokens.maybe_patch_fastokens(enabled: bool) → None[source]#

Apply the fastokens monkey-patch when enabled.

Parameters:

enabled – The resolved policy.tokenizer.use_fastokens config value. The NRL_USE_FASTOKENS env var, when set, overrides this: "1" forces on, anything else forces off.