Language Detector
The language detector rail is an input guardrail that blocks user messages written in languages outside a configured allowlist.
Language detection runs locally through fastText language identification, which is provided via the "nemoguardrails[multilingual]" optional extra.
When a message is blocked, the bot responds with a refusal message in the user’s own language whenever one is available, falling back either to a user-configured refusal message or to an English-language refusal.
Prerequisites
The language detector rail relies on the multilingual extra. To install, run pip install "nemoguardrails[multilingual]".
Configuration
The rail supports two mutually exclusive modes. allowed_languages is an allowlist: only messages written in the listed languages are let through. blocked_languages is a blocklist: messages in the listed languages are blocked and everything else is allowed. If neither allowed_languages or blocked_languages are set, the default configuration will only allow English.
Both settings accept the language codes supported by the fastText language identification model (for example en, es, de, fr, zh, ja, ko; see the full list of 176 codes).
Or, to block specific languages and allow the rest:
You cannot combine both allowed_languages and blocked_languages, you may only specify one.
If the language detector returns no language (for example, for empty input or when detection is unavailable or errors), use fail_open to control the rail behavior.
The default value is false which blocks the message; set it to true to allow the message.
You can customize the refusal messages used when a message is blocked. When the language detector identifies a disallowed language, it will look for a matching refusal message
from refusal_messages. Languages without a custom message use the built-in defaults (English, Spanish, Chinese, German, French, Hindi, Japanese, Arabic, and Thai); languages without a custom or default message will fall back to an English refusal.
By default, fast-langdetect only considers the first 80 characters in a message, which maximizes prediction accuracy.
You can manually tune this by setting num_characters_for_prediction:
However, fast-langdetect advises that values larger or smaller than 80 will reduce accuracy.
Usage
To use the language detector as an input rail, include the language detector check input flow in the input rails section of the config.yml file:
For Colang 2.x configurations using flow-based rails, call the flow from your input rails flow instead:
Limitations
- Language detection is heuristic. Very short or mixed-language messages may be misidentified.