aistore.sdk.batch.multipart.sliding_window_buffer

Module Contents

Classes

Name	Description
`SlidingWindowBuffer`	High-performance sliding window buffer optimized for streaming data processing.

API

class aistore.sdk.batch.multipart.sliding_window_buffer.SlidingWindowBuffer(
    max_size: int,
    boundary_size: int
)

High-performance sliding window buffer optimized for streaming data processing.

The buffer automatically manages memory by sliding the window when it reaches capacity, keeping only the most recent data plus a configurable overlap for cross-chunk pattern detection.

Parameters:

max_size

int

Maximum logical size before sliding window activation

boundary_size

int

Size of boundary pattern for overlap calculations

buffer

= bytearray(max_size)

end_pos

= 0

start_pos

= 0

total_processed

= 0

aistore.sdk.batch.multipart.sliding_window_buffer.SlidingWindowBuffer.__len__() -> int

Return current logical length of valid data in buffer.

Returns: int

Number of bytes of valid data

aistore.sdk.batch.multipart.sliding_window_buffer.SlidingWindowBuffer._compact_buffer() -> None

Compact buffer by moving valid data to beginning and updating position tracking.

aistore.sdk.batch.multipart.sliding_window_buffer.SlidingWindowBuffer._get_data(
    start: int = 0,
    end: typing.Optional[int] = None
) -> memoryview

Get a data slice without copying using memoryview for efficiency.

WARNING: The returned memoryview is only valid until the next call to append(), consume(), or any other method that modifies the buffer. For long-term storage, use get_data_bytes() instead.

Parameters:

start

intDefaults to 0

Starting position. Defaults to 0.

end

intDefaults to None

Ending position, None for end of buffer.

Returns: memoryview

Memory view of requested data range (TEMPORARY)

aistore.sdk.batch.multipart.sliding_window_buffer.SlidingWindowBuffer.append(
    data: bytes
) -> None

Append data to buffer with automatic sliding window management and compaction.

Parameters:

data

bytes

Data to append to the buffer

aistore.sdk.batch.multipart.sliding_window_buffer.SlidingWindowBuffer.clear() -> None

Clear the buffer and update total processed byte counter.

aistore.sdk.batch.multipart.sliding_window_buffer.SlidingWindowBuffer.consume(
    length: int
) -> bytes

Consume and return data from start of buffer, advancing the window.

Parameters:

length

int

Number of bytes to consume

Returns: bytes

Consumed data

aistore.sdk.batch.multipart.sliding_window_buffer.SlidingWindowBuffer.find(
    pattern: bytes,
    start: int = 0
) -> int

Find pattern in buffer and return absolute position in stream.

Parameters:

pattern

bytes

Pattern to search for

start

intDefaults to 0

Starting offset for search. Defaults to 0.

Returns: int

Absolute position if found, -1 if not found

aistore.sdk.batch.multipart.sliding_window_buffer.SlidingWindowBuffer.find_relative(
    pattern: bytes,
    start: int = 0
) -> int

Find pattern in buffer and return position relative to current buffer window.

Parameters:

pattern

bytes

Pattern to search for

start

intDefaults to 0

Starting offset for search. Defaults to 0.

Returns: int

Relative position if found, -1 if not found

aistore.sdk.batch.multipart.sliding_window_buffer.SlidingWindowBuffer.get_data_bytes(
    start: int = 0,
    end: typing.Optional[int] = None
) -> bytes

Get data as bytes by creating a copy of the requested range.

Parameters:

start

intDefaults to 0

Starting position. Defaults to 0.

end

intDefaults to None

Ending position, None for end of buffer

Returns: bytes

Copy of requested data range

aistore.sdk.batch.multipart.sliding_window_buffer.SlidingWindowBuffer.skip_leading_whitespace() -> int

Skip leading whitespace characters and return count of bytes skipped.

Returns: int

Number of whitespace bytes that were skipped and consumed