nemo_curator.utils.atomic_io

Module Contents

Functions

Name	Description
`_fsync_directory_best_effort`	Flush directory metadata when supported by the filesystem.
`_unlink_best_effort`	Remove a temporary file without masking the primary result.
`_write_json_temp_file`	Write JSON to a fsynced temporary file beside its destination.
`fsync_directory`	Flush directory metadata to disk.
`write_json_atomically`	Write JSON through a fsynced temp file and atomic rename.
`write_json_atomically_if_absent`	Atomically create a JSON file without replacing an existing file.

API

nemo_curator.utils.atomic_io._fsync_directory_best_effort(
    path: pathlib.Path
) -> None

Flush directory metadata when supported by the filesystem.

nemo_curator.utils.atomic_io._unlink_best_effort(
    path: pathlib.Path
) -> None

Remove a temporary file without masking the primary result.

nemo_curator.utils.atomic_io._write_json_temp_file(
    path: pathlib.Path,
    payload: typing.Any,
    indent: int | None = None,
    separators: tuple[str, str] | None = None,
    sort_keys: bool = True
) -> pathlib.Path

Write JSON to a fsynced temporary file beside its destination.

nemo_curator.utils.atomic_io.fsync_directory(
    path: pathlib.Path
) -> None

Flush directory metadata to disk.

nemo_curator.utils.atomic_io.write_json_atomically(
    path: pathlib.Path,
    payload: typing.Any,
    indent: int | None = None,
    separators: tuple[str, str] | None = None,
    sort_keys: bool = True
) -> None

Write JSON through a fsynced temp file and atomic rename.

Directory fsync is best-effort because the destination is already visible after os.replace and some shared filesystems do not support syncing a directory file descriptor.

nemo_curator.utils.atomic_io.write_json_atomically_if_absent(
    path: pathlib.Path,
    payload: typing.Any,
    indent: int | None = None,
    separators: tuple[str, str] | None = None,
    sort_keys: bool = True
) -> bool

Atomically create a JSON file without replacing an existing file.