nemo_curator.utils.lance

Module Contents

Name	Description
`add_lance_metadata_columns`	-
`encode_lance_blob_columns`	Rebuild Lance Blob v2 arrays from materialized binary columns.
`materialize_lance_blob_columns`	Replace scanned Blob v2 descriptors with binary payloads.

nemo_curator.utils.lance.add_lance_metadata_columns(
    table: pyarrow.Table
) -> pyarrow.Table

nemo_curator.utils.lance.encode_lance_blob_columns(
    table: pyarrow.Table,
    schema: pyarrow.Schema
) -> pyarrow.Table

Rebuild Lance Blob v2 arrays from materialized binary columns.

nemo_curator.utils.lance.materialize_lance_blob_columns(
    dataset: lance.LanceDataset,
    table: pyarrow.Table
) -> pyarrow.Table

Replace scanned Blob v2 descriptors with binary payloads.

nemo_curator.utils.lance.LANCE_FRAGID_COLUMN = '__lance_fragid'

nemo_curator.utils.lance.LANCE_ROWADDR_COLUMN = '__lance_rowaddr'

nemo_curator.utils.lance.LANCE_ROWID_COLUMN = '__lance_rowid'