Skip to navigation

nemoguardrails.library.utils.language_detection

View as MarkdownOpen in Claude

Shared language-detection utilities for rails.

Detection is powered by fast-langdetect (the fastText lid.176 model), which supports the 176 language codes listed in FASTTEXT_LANGUAGES: https://fasttext.cc/docs/en/language-identification.html

Module Contents

Functions

NameDescription
detect_languageDetect the language of text using fast-langdetect.
get_refusal_messageReturn the refusal message for lang.

Data

DEFAULT_REFUSAL_MESSAGES

SUPPORTED_LANGUAGES

_DETECTORS

log

API

nemoguardrails.library.utils.language_detection.detect_language(
text: str,
max_length: typing.Optional[int] = None
) -> typing.Optional[str]

Detect the language of text using fast-langdetect.

Returns the ISO code of the most likely language (one of FASTTEXT_LANGUAGES), or None when detection is unavailable (fast-langdetect not installed) or fails.

nemoguardrails.library.utils.language_detection.get_refusal_message(
lang: str,
custom_messages: typing.Optional[typing.Dict[str, str]] = None
) -> str

Return the refusal message for lang.

Preference order: a custom message for the language, the built-in default for the language, a custom English message, then the built-in English one.

nemoguardrails.library.utils.language_detection.DEFAULT_REFUSAL_MESSAGES: Dict[str, str] = {'en': "I'm sorry, I can't respond to that.", 'es': 'Lo siento, no puedo respond...
nemoguardrails.library.utils.language_detection.SUPPORTED_LANGUAGES: FrozenSet[str] = frozenset(DEFAULT_REFUSAL_MESSAGES)
nemoguardrails.library.utils.language_detection._DETECTORS: Dict[Optional[int], Any] = {}
nemoguardrails.library.utils.language_detection.log = logging.getLogger(__name__)