aistore.sdk.obj.obj_file.object_file

Module Contents

Classes

Name	Description
`ObjectFileReader`	A sequential read-only file-like object extending `BufferedIOBase` for reading object data, with support for both
`ObjectFileWriter`	A file-like writer object for AIStore, extending `BufferedWriter`.

Data

logger

API

class aistore.sdk.obj.obj_file.object_file.ObjectFileReader(
    content_provider: aistore.sdk.obj.content_iterator.BaseContentIterProvider,
    max_resume: int
)

Bases: BufferedIOBase

A sequential read-only file-like object extending BufferedIOBase for reading object data, with support for both reading a fixed size of data and reading until the end of file (EOF).

When a read is requested, any remaining data from a previously fetched chunk is returned first. If the remaining data is insufficient to satisfy the request, the read() method fetches additional chunks from the provided iterator as needed, until the requested size is fulfilled or the end of the stream is reached.

In case of unexpected stream interruptions (e.g. ChunkedEncodingError, ConnectionError) or timeouts (e.g. ReadTimeout), the read() method automatically retries and resumes fetching data from the last successfully retrieved chunk. The max_resume parameter controls how many retry attempts are made before an error is raised.

Parameters:

content_provider

BaseContentIterProvider

A provider that creates iterators which can fetch object data from AIS in chunks.

max_resume

int

Maximum number of resumes allowed for an ObjectFileReader instance.

_content_iter

Optional[Generator[bytes, None, None]] = None

_remainder

Optional[memoryview] = None

_resume_position

= 0

_resume_total

= 0

aistore.sdk.obj.obj_file.object_file.ObjectFileReader.__enter__()

aistore.sdk.obj.obj_file.object_file.ObjectFileReader._expected_end_position() -> typing.Optional[int]

aistore.sdk.obj.obj_file.object_file.ObjectFileReader._handle_broken_stream(
    err: Exception
) -> typing.Optional[typing.Generator[bytes, None, None]]

Handle the broken stream/iterator by incrementing the resume count, logging a warning, and returning a newly instantiated iterator from the last known position.

Parameters:

err

Exception

The error that caused the resume attempt.

Returns: Optional[Generator[bytes, None, None]]

Optional[Generator[bytes, None, None]]: The new generator. None if the object is not cached.

Raises:

ObjectFileReaderMaxResumeError: If the maximum number of resume attempts is exceeded.

aistore.sdk.obj.obj_file.object_file.ObjectFileReader._reset(
    retain_resumes: bool = False
) -> None

aistore.sdk.obj.obj_file.object_file.ObjectFileReader.close() -> None

Close the file.

aistore.sdk.obj.obj_file.object_file.ObjectFileReader.read(
    size: typing.Optional[int] = -1
) -> bytes

Read up to ‘size’ bytes from the object. If size is -1, read until the end of the stream.

Parameters:

size

intDefaults to -1

The number of bytes to read. If -1, reads until EOF.

Returns: bytes

The read data as a bytes object.

Raises:

ObjectFileReaderStreamError: If a connection cannot be made.
ObjectFileReaderMaxResumeError: If the stream is interrupted more than the allowed maximum.
ValueError: I/O operation on a closed file.
Exception: Any other errors while streaming and reading.

aistore.sdk.obj.obj_file.object_file.ObjectFileReader.readable() -> bool

Return whether the file is readable.

class aistore.sdk.obj.obj_file.object_file.ObjectFileWriter(
    obj_writer: ObjectWriter,
    mode: str
)

Bases: BufferedWriter

A file-like writer object for AIStore, extending BufferedWriter.

Parameters:

obj_writer

ObjectWriter

The ObjectWriter instance for handling write operations.

mode

str

Specifies the mode in which the file is opened.

'w': Write mode. Opens the object for writing, truncating any existing content. Writing starts from the beginning of the object.
'a': Append mode. Opens the object for appending. Existing content is preserved, and writing starts from the end of the object.

_handle

= ''

aistore.sdk.obj.obj_file.object_file.ObjectFileWriter.__enter__(
    args = (),
    kwargs = {}
)

aistore.sdk.obj.obj_file.object_file.ObjectFileWriter.close() -> None

Close the writer and finalize the object.

aistore.sdk.obj.obj_file.object_file.ObjectFileWriter.flush() -> None

Flush the writer, ensuring the object is finalized.

This does not close the writer but makes the current state accessible.

Raises:

ValueError: I/O operation on a closed file.

aistore.sdk.obj.obj_file.object_file.ObjectFileWriter.write(
    buffer: bytes
) -> int

Write data to the object.

Parameters:

buffer

bytes

The data to write.

Returns: int

Number of bytes written.

Raises:

ValueError: I/O operation on a closed file.

aistore.sdk.obj.obj_file.object_file.logger = get_logger(__name__)