> For clean Markdown content of this page, append .md to this URL. For the complete documentation index, see https://docs.nvidia.com/dynamo/llms.txt. For full content including API reference and SDK examples, see https://docs.nvidia.com/dynamo/llms-full.txt.
# Fastokens Tokenizer
`fastokens` is an optional tokenizer backend for BPE `tokenizer.json` models. It uses the Rust encoder from the [`fastokens` GitHub repository](https://github.com/crusoecloud/fastokens) for text-to-token-ID conversion while Dynamo continues to use HuggingFace `tokenizers` for decoding and streaming output.
## Why Use Fastokens in Dynamo
The Dynamo frontend tokenizes every incoming prompt before sending the request to an inference backend. For short prompts, that cost is usually small. For agentic, retrieval-augmented generation (RAG), and long-context workloads, tokenization can become a meaningful part of Time To First Token (TTFT), especially when KV cache hit rates are high and the model path is already fast.
Use `fastokens` when tokenization is visible in your frontend latency profile and your model uses a supported BPE tokenizer. It improves how tokenization scales on modern CPUs through:
- Parallel pre-tokenization for long inputs.
- Parallel BPE encoding with per-thread and shared caches.
- Reused buffers and reduced allocation overhead.
- PCRE2 JIT regex support where the tokenizer pattern allows it.
Performance gains generally increase with prompt size, but the result depends on tokenizer structure, CPU, concurrency, cache hit rate, and how much of TTFT is spent before generation. The [Crusoe and NVIDIA fastokens writeup](https://www.crusoe.ai/resources/blog/reducing-ttft-by-cpumaxxing-tokenization) provides benchmarks across models, datasets, CPU architectures, and input lengths from 512 to 100K tokens.
## When to Enable It
Enable `fastokens` when:
- Prompts are long, commonly thousands to tens of thousands of tokens.
- Your workload is prefill-heavy, agentic, or RAG-heavy.
- TTFT remains high even when KV cache hit rates are strong.
- Frontend tokenizer latency shows up in metrics, traces, or profiling.
- Your model uses a BPE `tokenizer.json`.
Stay on the default backend if:
- Prompts are short and tokenization is not on the critical path.
- You are validating a new or unusual tokenizer and want maximum compatibility first.
- The frontend logs that `fastokens` failed to load and fell back to HuggingFace.
- Your model uses `.model` or `.tiktoken` tokenizer files, where this flag has no effect.
## Enable and Validate Fastokens
`fastokens` is a **frontend** setting. Enable it on the Frontend component with the `DYN_TOKENIZER` environment variable:
```yaml
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeployment
metadata:
name: my-deployment
spec:
components:
- name: Frontend
type: frontend
replicas: 1
podTemplate:
spec:
containers:
- name: main
image: ${RUNTIME_IMAGE}
env:
- name: DYN_TOKENIZER
value: fastokens
# ... worker component unchanged
```
You can also set the same option with the `--tokenizer fastokens` frontend flag. The flag takes precedence when both settings are present.
To return to the default HuggingFace tokenizer backend, set `DYN_TOKENIZER=default` or omit it. Changing the value requires the Frontend pod to be replaced, which `kubectl apply` does automatically.
No client changes are required. Request payloads, OpenAI-compatible API behavior, and streamed responses remain the same.
Check the frontend startup logs after enabling the setting.
When `fastokens` is active, look for:
```text
Using fastokens tokenizer backend
```
If the tokenizer is unsupported, Dynamo keeps serving with the default backend and logs:
```text
Failed to load fastokens, falling back to HuggingFace
```
The fallback keeps the deployment healthy, but the model does not receive the `fastokens` speedup.
Dynamo includes a frontend benchmark sweep that compares HuggingFace and `fastokens` across input sequence length, concurrency, and worker count.
```bash
cd benchmarks/frontend/scripts
python3 sweep_runner.py \
--tokenizers hf,fastokens \
--concurrency 32,64,128 \
--isl 512,2048,8192
```
Use local mocker runs to isolate frontend and tokenizer overhead. Use vLLM or SGLang runs to measure end-to-end TTFT impact with a real backend.
See the [frontend benchmarking guide](https://github.com/ai-dynamo/dynamo/tree/main/benchmarks/frontend/README.md) and the [scaling-test recipe](https://github.com/ai-dynamo/dynamo/blob/main/benchmarks/frontend/scripts/scaling-test.md) for a full walkthrough.
## How Dynamo Integrates Fastokens
Dynamo uses a hybrid integration:
- **Encoding**: `fastokens` converts prompt text to token IDs.
- **Decoding**: HuggingFace `tokenizers` converts generated token IDs back to text.
Both backends load from the same `tokenizer.json`, so supported tokenizers should produce the same token IDs as the default HuggingFace path. If `fastokens` cannot load the tokenizer file, Dynamo logs a warning and falls back to the default backend instead of dropping requests.
```mermaid
flowchart TD
Start["Frontend starts with
--tokenizer fastokens"] --> Kind{"Tokenizer file type"}
Kind -->|BPE tokenizer.json| Load{"fastokens loads?"}
Kind -->|.model / .tiktoken| Other["Use existing TikToken backend"]
Load -->|Yes| Fast["Encode with fastokens
Decode with HuggingFace"]
Load -->|No| Warn["Log warning"] --> Default["Use HuggingFace backend"]
Fast --> Serve["Serve requests"]
Default --> Serve
Other --> Serve
```
## Compatibility
| Tokenizer format | Behavior with `--tokenizer fastokens` |
|---|---|
| BPE `tokenizer.json` | Dynamo tries to encode with `fastokens` and decode with HuggingFace. |
| BPE `tokenizer.json` with unsupported components | Dynamo logs a warning and falls back to HuggingFace. |
| TikToken `.model` or `.tiktoken` | Unchanged. Dynamo uses the existing TikToken backend. |
`fastokens` targets BPE tokenizer pipelines. It is focused on inference and does not support every HuggingFace `tokenizers` feature; additional encoding outputs and some normalizers or pre-tokenizers are not available.
The `fastokens` repository maintains the current [tested models list](https://github.com/crusoecloud/fastokens#tested-models).
For any new model, validate on representative prompts before rolling out broadly. The safest check is to compare token IDs against the default backend and confirm the frontend logs show the fast path was selected.
## Troubleshooting
Make sure the setting is applied to the frontend process, not only to the backend worker. Set `DYN_TOKENIZER=fastokens` or `--tokenizer fastokens` on the **Frontend** component and re-apply the deployment so the frontend pod restarts. For benchmark DynamoGraphDeployment templates, use `DYN_TOKENIZER=fastokens`; the sweep runner maps `--tokenizers fastokens` to that value and restarts the frontend pod.
The model's tokenizer file uses a feature that `fastokens` does not support, or it is not a BPE `tokenizer.json` path. Dynamo has already fallen back to HuggingFace and should keep serving traffic. Check the tokenizer format, compare it against the [tested models list](https://github.com/crusoecloud/fastokens#tested-models), and use `--tokenizer default` to avoid the warning.
Use only `fastokens` or `default` for `DYN_TOKENIZER`. Values such as `fast`, `hf`, or `huggingface` are benchmark-runner aliases, not valid values for the frontend environment variable.
The `fastokens` setting has no effect for TikToken-format tokenizers. Dynamo uses the existing TikToken backend, so you should not expect the `Using fastokens tokenizer backend` log or a `fastokens` speedup.
First confirm the fast path is active in logs. If it is, tokenization may not be the bottleneck for this workload. Check prompt length, cache hit rate, backend prefill time, frontend CPU saturation, and the `dynamo_frontend_tokenizer_latency_ms` metric. Short prompts and decode-heavy traffic often show little end-to-end change.
Inspect each run artifact and frontend log to confirm the tokenizer backend changed. In Kubernetes mode, the DynamoGraphDeployment frontend pod must be replaced after `DYN_TOKENIZER` changes. In local mocker mode, start with larger input sequence length values such as 8192 or higher so tokenization is large enough to measure.
Do not roll out that model with `fastokens`. Reproduce the mismatch with a minimal prompt and file an issue with the model name, tokenizer file, prompt, and whether the model appears on the tested models list.
Decoding still uses HuggingFace, so this is usually not caused by the `fastokens` setting. Verify that the tokenizer files match the model weights and that the default backend produces the expected output.
## See Also
- [`fastokens`: A Solution to the Tokenization Bottleneck](https://www.crusoe.ai/resources/blog/reducing-ttft-by-cpumaxxing-tokenization)
- [`fastokens` on GitHub](https://github.com/crusoecloud/fastokens)
- [Tokenizer component reference](/dynamo/dev/knowledge-base/modular-components/frontend/tokenizer)
- [Frontend configuration reference](/dynamo/dev/components/frontend-configuration)
- [Frontend benchmarking](https://github.com/ai-dynamo/dynamo/tree/main/benchmarks/frontend/README.md)