nemoguardrails.library.hf_classifier.chunking
nemoguardrails.library.hf_classifier.chunking
Automatic windows for local text classifiers without truncation.
Module Contents
Classes
Functions
API
Bases: ChunkPipeline, TextClassificationPipeline
Reuse HF tokenization, inference and scoring for oversized text inputs.
_inference_lock
Preserve native truncation, otherwise verify and bound overlapping windows.
Dataclass
Validated tokenization that can be passed to HF data-loader workers.
encoding
is_chunked
window_count
Reject missing or changed content before any window reaches inference.