bridge.data.packing.paths

Path detection and resolution for packed Parquet artifacts.

Module Contents

Functions

is_packed_parquet_file	Check if a path refers to a packed Parquet file or pattern.
is_packed_parquet_spec	Check if a spec refers to a packed Parquet source (file, directory, or glob).
_is_parquet_file	Check if a path refers to any Parquet file.
_resolve_parquet_paths	Resolve a file path specification to a list of actual file paths.
resolve_packed_parquet_paths	Resolve a packed parquet spec to a list of shard file paths.

API

bridge.data.packing.paths.is_packed_parquet_file(path) → bool

Check if a path refers to a packed Parquet file or pattern.

Parameters:

path – A Path object or string path.

Returns:

True if the path ends with .idx.parquet or .idx.pq, or contains a glob pattern that would match such files.

bridge.data.packing.paths.is_packed_parquet_spec(spec: str | pathlib.Path) → bool

Check if a spec refers to a packed Parquet source (file, directory, or glob).

This predicate reflects what the dataset loader supports in packed mode:

• Single .parquet/.idx.parquet/.idx.pq files

• Glob patterns ending in .parquet/.idx.parquet/.idx.pq

• Directories containing parquet files

Parameters:

spec – A path specification (file, directory, or glob pattern).

Returns:

True if the spec could refer to packed Parquet data.

bridge.data.packing.paths._is_parquet_file(path: str) → bool

Check if a path refers to any Parquet file.

Parameters:

path – A string path.

Returns:

True if the path ends with .parquet or .pq (case-insensitive).

bridge.data.packing.paths._resolve_parquet_paths(file_path: str) → list[str]

Resolve a file path specification to a list of actual file paths.

Supports:

• Single file: “data.idx.parquet”, “shard_0.parquet”

• Glob pattern: “data*.idx.parquet”, “shard_*.parquet”

• Directory: “/path/to/data/” (globs for *.parquet and *.pq)

Parameters:

file_path – Path specification (file, glob pattern, or directory).

Returns:

Sorted list of resolved file paths.

Raises:

ValueError – If no matching files are found.

bridge.data.packing.paths.resolve_packed_parquet_paths(spec: str | pathlib.Path) → list[str]

Resolve a packed parquet spec to a list of shard file paths.

Public wrapper around the internal _resolve_parquet_paths function. Use this to validate and resolve packed parquet specs before dataset creation.

Supports:

• Single file: “data.idx.parquet”, “shard_0.parquet”

• Glob pattern: “data*.idx.parquet”, “shard_*.parquet”

• Directory: “/path/to/data/” (globs for *.parquet and *.pq)

Parameters:

spec – Path specification (file, glob pattern, or directory).

Returns:

Sorted list of resolved file paths.

Raises:

ValueError – If no matching files are found.