bridge.data.packing.paths#

Path detection and resolution for packed Parquet artifacts.

Module Contents#

Functions#

is_packed_parquet_file

Check if a path refers to a packed Parquet file or pattern.

is_packed_parquet_spec

Check if a spec refers to a packed Parquet source (file, directory, or glob).

_is_parquet_file

Check if a path refers to any Parquet file.

_resolve_parquet_paths

Resolve a file path specification to a list of actual file paths.

resolve_packed_parquet_paths

Resolve a packed parquet spec to a list of shard file paths.

_refresh_directory_metadata

Force a listdir on the nearest existing ancestor to bust stale NFS directory-cache entries.

resolve_packed_parquet_paths_with_retry

Resolve packed parquet spec with NFS-aware retries and directory-metadata refresh.

Data#

logger

API#

bridge.data.packing.paths.logger#

‘getLogger(…)’

bridge.data.packing.paths.is_packed_parquet_file(path) → bool#

Check if a path refers to a packed Parquet file or pattern.

Parameters:

path – A Path object or string path.

Returns:

True if the path ends with .idx.parquet or .idx.pq, or contains a glob pattern that would match such files.

bridge.data.packing.paths.is_packed_parquet_spec(spec: str | pathlib.Path) → bool#

Check if a spec refers to a packed Parquet source (file, directory, or glob).

This predicate reflects what the dataset loader supports in packed mode:

  • Single .parquet/.idx.parquet/.idx.pq files

  • Glob patterns ending in .parquet/.idx.parquet/.idx.pq

  • Directories containing parquet files

Parameters:

spec – A path specification (file, directory, or glob pattern).

Returns:

True if the spec could refer to packed Parquet data.

bridge.data.packing.paths._is_parquet_file(path: str) → bool#

Check if a path refers to any Parquet file.

Parameters:

path – A string path.

Returns:

True if the path ends with .parquet or .pq (case-insensitive).

bridge.data.packing.paths._resolve_parquet_paths(file_path: str) → list[str]#

Resolve a file path specification to a list of actual file paths.

Supports:

  • Single file: “data.idx.parquet”, “shard_0.parquet”

  • Glob pattern: “data*.idx.parquet”, “shard_*.parquet”

  • Directory: “/path/to/data/” (globs for *.parquet and *.pq)

Parameters:

file_path – Path specification (file, glob pattern, or directory).

Returns:

Sorted list of resolved file paths.

Raises:

ValueError – If no matching files are found.

bridge.data.packing.paths.resolve_packed_parquet_paths(spec: str | pathlib.Path) → list[str]#

Resolve a packed parquet spec to a list of shard file paths.

Public wrapper around the internal _resolve_parquet_paths function. Use this to validate and resolve packed parquet specs before dataset creation.

Supports:

  • Single file: “data.idx.parquet”, “shard_0.parquet”

  • Glob pattern: “data*.idx.parquet”, “shard_*.parquet”

  • Directory: “/path/to/data/” (globs for *.parquet and *.pq)

Parameters:

spec – Path specification (file, glob pattern, or directory).

Returns:

Sorted list of resolved file paths.

Raises:

ValueError – If no matching files are found.

bridge.data.packing.paths._refresh_directory_metadata(spec: str) → None#

Force a listdir on the nearest existing ancestor to bust stale NFS directory-cache entries.

On NFS filesystems (e.g. Isilon NFSv4.0) a node that did not write a directory may cache a negative “not found” result for up to acdirmin seconds (~30 s by default). Calling os.listdir() on the nearest existing ancestor forces the NFS client to issue GETATTR+READDIR to the server, collapsing the negative-cache window within one RPC round-trip.

bridge.data.packing.paths.resolve_packed_parquet_paths_with_retry(
spec: str | pathlib.Path,
*,
max_attempts: int = 10,
backoff_s: float = 1.0,
) → list[str]#

Resolve packed parquet spec with NFS-aware retries and directory-metadata refresh.

On distributed NFS filesystems (e.g. Isilon NFSv4.0), a node that did not write a directory may see stale cached metadata for ~30 s after rank 0 writes it. Issuing os.listdir() on the nearest existing ancestor forces a fresh GETATTR/READDIR to the NFS server, collapsing the negative-cache window within one retry cycle. With max_attempts=10 and backoff_s=1.0 the total budget is 1+2+…+9 = 45 s, comfortably above the measured 29 s window. See NVIDIA-NeMo/Megatron-Bridge#4207.

Parameters:

  • spec – Path specification (file, glob pattern, or directory).

  • max_attempts – Maximum number of resolution attempts (default 10).

  • backoff_s – Base sleep duration in seconds; attempt N sleeps N*backoff_s (default 1.0).

Returns:

Sorted list of resolved file paths.

Raises:

ValueError – If no matching files are found after all attempts.