nemo_curator.utils.column_utils

Module Contents

Functions

Name	Description
`pyarrow_string_to_pandas_dtype`	Map Arrow strings to a pandas dtype with functional string operations.
`pyarrow_to_pandas_dtype`	Map Arrow types to pandas dtypes with functional string operations.
`resolve_filename_column`	Resolve the filename column name based on the input parameter.

API

nemo_curator.utils.column_utils.pyarrow_string_to_pandas_dtype(
    arrow_type: pyarrow.DataType,
    na_value: object = pd.NA
) -> pandas.StringDtype | None

Map Arrow strings to a pandas dtype with functional string operations.

nemo_curator.utils.column_utils.pyarrow_to_pandas_dtype(
    arrow_type: pyarrow.DataType
) -> pandas.StringDtype | pandas.ArrowDtype

Map Arrow types to pandas dtypes with functional string operations.

nemo_curator.utils.column_utils.resolve_filename_column(
    add_filename_column: bool | str
) -> str | None

Resolve the filename column name based on the input parameter.

Parameters:

add_filename_column

bool | str

Can be:

True: Use default filename column name
False: No filename column
str: Use the provided string as column name

Returns: str | None

str | None: The filename column name or None if not needed