nemo_curator.utils.grouping

Utility Functions for grouping iterables.

This module provides a collection of utility functions designed to assist with common tasks related to manipulating and transforming iterables in Python.

These utilities are generic and work with any iterable types. They’re particularly useful for data processing tasks, batching operations, and other scenarios where dividing data into specific groupings is necessary.

Module Contents

Functions

Name	Description
`pairwise`	Return pairs of consecutive items from the input iterable.
`split_by_chunk_size`	Split an iterable into chunks of the specified size.
`split_into_n_chunks`	Split an iterable into a specified number of chunks.

Data

T

API

nemo_curator.utils.grouping.pairwise(
    iterable: collections.abc.Iterable[nemo_curator.utils.grouping.T]
) -> collections.abc.Iterable[tuple[nemo_curator.utils.grouping.T, nemo_curator.utils.grouping.T]]) -> collections.abc.Iterable[tuple[nemo_curator.utils.grouping.T, nemo_curator.utils.grouping.T]]

Return pairs of consecutive items from the input iterable.

Parameters:

iterable

Iterable[T]

The input iterable.

Returns: Iterable[tuple[T, T]]

Iterable[tuple[T, T]]: Pairs of consecutive items.

nemo_curator.utils.grouping.split_by_chunk_size(
    iterable: collections.abc.Iterable[nemo_curator.utils.grouping.T],
    chunk_size: int,
    custom_size_func: typing.Callable[[T], int] = lambda x: 1,
    drop_incomplete_chunk: bool = False
) -> collections.abc.Generator[list[nemo_curator.utils.grouping.T], None, None]

Split an iterable into chunks of the specified size.

Yields:

Generator[list[T], None, None]: Chunks of the input iterable.

Parameters:

iterable

Iterable[T]

The input iterable to be split.

chunk_size

int

Size of each chunk.

custom_size_func

typing.CallableDefaults to lambda x: 1

function

drop_incomplete_chunk

boolDefaults to False

If True, drops the last chunk if its size is less than the specified chunk size. Defaults to False.

nemo_curator.utils.grouping.split_into_n_chunks(
    iterable: collections.abc.Iterable[nemo_curator.utils.grouping.T],
    num_chunks: int
) -> collections.abc.Generator[list[nemo_curator.utils.grouping.T], None, None]

Split an iterable into a specified number of chunks.

Yields:

Generator[list[T], None, None]: Chunks of the input iterable.

Parameters:

iterable

Iterable[T]

The input iterable to be split.

num_chunks

int

The desired number of chunks.

nemo_curator.utils.grouping.T = typing.TypeVar('T')