Skip to navigation

nemoguardrails.library.hf_classifier.chunking

View as MarkdownOpen in Claude

Automatic windows for local text classifiers without truncation.

Module Contents

Classes

NameDescription
ChunkedTextClassificationPipelineReuse HF tokenization, inference and scoring for oversized text inputs.
_PreparedInputValidated tokenization that can be passed to HF data-loader workers.

Functions

NameDescription
_content_tokens-
_input_limit-
_verify_coverageReject missing or changed content before any window reaches inference.

API

class nemoguardrails.library.hf_classifier.chunking.ChunkedTextClassificationPipeline(
args = (),
kwargs = {}
)

Bases: ChunkPipeline, TextClassificationPipeline

Reuse HF tokenization, inference and scoring for oversized text inputs.

_inference_lock
= threading.Lock()
nemoguardrails.library.hf_classifier.chunking.ChunkedTextClassificationPipeline.__call__(
inputs,
kwargs = {}
)
nemoguardrails.library.hf_classifier.chunking.ChunkedTextClassificationPipeline.__getstate__()
nemoguardrails.library.hf_classifier.chunking.ChunkedTextClassificationPipeline.__setstate__(
state
)
nemoguardrails.library.hf_classifier.chunking.ChunkedTextClassificationPipeline._forward(
model_inputs
)
nemoguardrails.library.hf_classifier.chunking.ChunkedTextClassificationPipeline._prepare_input(
inputs,
max_windows = 128,
kwargs = {}

Preserve native truncation, otherwise verify and bound overlapping windows.

nemoguardrails.library.hf_classifier.chunking.ChunkedTextClassificationPipeline._sanitize_parameters(
kwargs = {}
)
nemoguardrails.library.hf_classifier.chunking.ChunkedTextClassificationPipeline.get_iterator(
inputs,
num_workers,
batch_size,
preprocess_params,
forward_params,
postprocess_params
)
nemoguardrails.library.hf_classifier.chunking.ChunkedTextClassificationPipeline.postprocess(
model_outputs,
function_to_apply = None,
top_k = 1,
_legacy = True
)
nemoguardrails.library.hf_classifier.chunking.ChunkedTextClassificationPipeline.preprocess(
inputs,
kwargs = {}
)
class nemoguardrails.library.hf_classifier.chunking._PreparedInput(
encoding: transformers.BatchEncoding,
is_chunked: bool = False
)
Dataclass

Validated tokenization that can be passed to HF data-loader workers.

encoding
BatchEncoding
is_chunked
bool = False
window_count
int
nemoguardrails.library.hf_classifier.chunking._content_tokens(
encoding: typing.Any
) -> list[int]
nemoguardrails.library.hf_classifier.chunking._input_limit(
pipe: typing.Any,
max_length: int | None = None
) -> int | None
nemoguardrails.library.hf_classifier.chunking._verify_coverage(
expected: list[int],
encoded: transformers.BatchEncoding,
stride: int,
direction: str
) -> None

Reject missing or changed content before any window reaches inference.