aistore.sdk.batch.multipart.multipart_stream_buffer

Module Contents

Classes

Name	Description
`MultipartStreamBuffer`	Manages reading content from an iterator and buffering it efficiently.

Data

BOUNDARY_MARGIN

API

class aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer(
    content_iter: typing.Iterator[bytes],
    boundary: bytes,
    max_buffer_size: int
)

Manages reading content from an iterator and buffering it efficiently.

This class handles the low-level details of reading from content iterators and managing the sliding window buffer. It provides an abstraction layer that prevents higher-level classes from directly manipulating the buffer.

Parameters:

content_iter

Iterator[bytes]

Iterator yielding chunks of content data

boundary

bytes

The multipart boundary marker to detect

max_buffer_size

int

Maximum size of the internal sliding window buffer

_boundary_safety_margin

= len(self._boundary) + BOUNDARY_MARGIN

_buffer

_whitespace_bytes

boundary

bytes

Get the boundary pattern.

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer._fill_buffer_safely(
    min_size: int = 0
) -> bool

Fill buffer ensuring we have at least min_size bytes available, with boundary safety margin.

This method ensures we always have enough data in the buffer to avoid boundary splits when consuming data.

Parameters:

min_size

intDefaults to 0

Minimum buffer size required

Returns: bool

True if sufficient data available, False if stream exhausted

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer._read_next_chunk() -> bool

Read one chunk from content iterator into buffer.

Returns: bool

True if chunk read successfully, False if iterator exhausted

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.consume_before_pattern(
    pos: int,
    pattern: bytes,
    clean_endings: bool = True
) -> bytes

Consume content before a pattern at a known position and return it.

Parameters:

pos

int

Known position of the pattern in the buffer

pattern

bytes

Pattern to search for in the buffer (for length calculation)

clean_endings

boolDefaults to True

Whether to strip whitespace and line endings from consumed content. Defaults to True.

Returns: bytes

Content found before the pattern, with optional cleaning applied. Returns empty bytes if position is invalid.

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.consume_data(
    length: int
) -> bytes

Consume and return data from buffer, ensuring boundary safety.

This method ensures that after consumption, we maintain enough buffer data to avoid boundary splits.

Parameters:

length

int

Number of bytes to consume

Returns: bytes

Consumed data

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.ensure_data_available(
    min_bytes: int = 0
) -> bool

Ensure at least min_bytes are available in buffer.

Parameters:

min_bytes

intDefaults to 0

Minimum buffer size required. Defaults to 0.

Returns: bool

True if sufficient buffer available, False if stream exhausted

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.find_boundary_position() -> int

Find boundary position in current buffer.

Returns: int

Position of boundary within buffer, -1 if not found or no boundary set

Raises:

ValueError: If no boundary has been set for this reader

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.find_pattern(
    pattern: bytes,
    start: int = 0
) -> int

Find pattern in current buffer data.

Parameters:

pattern

bytes

Pattern to search for

start

intDefaults to 0

Starting offset for search. Defaults to 0.

Returns: int

Position of pattern relative to buffer start, -1 if not found

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.get_boundary_size() -> int

Get the size of the boundary pattern.

Returns: int

Size of boundary in bytes, 0 if no boundary set

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.get_buffer_size() -> int

Get current buffer size for safe content calculations.

Returns: int

Current size of buffer

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.get_data_slice(
    start: int = 0,
    end: typing.Optional[int] = None
) -> bytes

Get a copy of data slice from buffer.

Parameters:

start

intDefaults to 0

Starting position. Defaults to 0.

end

Optional[int]Defaults to None

Ending position, None for end of buffer.

Returns: bytes

Copy of requested data range

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.get_safe_content_size() -> int

Get size of content that can be safely consumed without risking boundary split.

This ensures we don’t consume data that might contain a partial boundary that spans across buffer chunks.

Returns: int

Size of content consumable without a boundary split. Returns full buffer size if no boundary is set.

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.has_boundary() -> bool

Check if a boundary has been configured for this reader.

Returns: bool

True if boundary is set, False otherwise

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.has_data() -> bool

Check if buffer has any data or can get more.

Returns: bool

True if data is available or can be obtained

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.is_exhausted() -> bool

Check if both stream and buffer are exhausted.

Returns: bool

True if no more data can be obtained

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.read_until_boundary(
    clean_endings: bool = True
) -> typing.Iterator[bytes]

Read and consume all data until the next boundary is found.

Parameters:

clean_endings

boolDefaults to True

Whether to strip whitespace and line endings from consumed content. Defaults to True.

Raises:

ValueError: If no boundary has been set for this reader

aistore.sdk.batch.multipart.multipart_stream_buffer.MultipartStreamBuffer.skip_whitespace() -> int

Skip leading whitespace in buffer.

Returns: int

Number of whitespace bytes that were skipped

aistore.sdk.batch.multipart.multipart_stream_buffer.BOUNDARY_MARGIN = 8