API ReferenceFull Library ReferenceNemo CuratorNemo CuratorUtils

nemo_curator.utils.lance

View as Markdown

Module Contents

Functions

NameDescription
add_lance_metadata_columns-
encode_lance_blob_columnsRebuild Lance Blob v2 arrays from materialized binary columns.
materialize_lance_blob_columnsReplace scanned Blob v2 descriptors with binary payloads.

Data

LANCE_FRAGID_COLUMN

LANCE_ROWADDR_COLUMN

LANCE_ROWID_COLUMN

API

nemo_curator.utils.lance.add_lance_metadata_columns(
    table: pyarrow.Table
) -> pyarrow.Table
 
nemo_curator.utils.lance.encode_lance_blob_columns(
    table: pyarrow.Table,
    schema: pyarrow.Schema
) -> pyarrow.Table

Rebuild Lance Blob v2 arrays from materialized binary columns.

nemo_curator.utils.lance.materialize_lance_blob_columns(
    dataset: lance.LanceDataset,
    table: pyarrow.Table
) -> pyarrow.Table

Replace scanned Blob v2 descriptors with binary payloads.

nemo_curator.utils.lance.LANCE_FRAGID_COLUMN = '__lance_fragid'
 
nemo_curator.utils.lance.LANCE_ROWADDR_COLUMN = '__lance_rowaddr'
 
nemo_curator.utils.lance.LANCE_ROWID_COLUMN = '__lance_rowid'