aistore.sdk.batch.extractor.archive_stream_extractor
aistore.sdk.batch.extractor.archive_stream_extractor
Module Contents
Classes
Data
API
Parent class for extracting batch archive streams from AIStore.
Integrates with MossReq/MossResp (Multi-Object Streaming Service) objects to provide proper metadata mapping.
Verify that the archive contains all requested entries.
Use the original request count because an incomplete multipart response can omit entries from both its metadata and its archive. If cont_on_err is enabled, logs the shortfall and allows continuation.
Parameters:
Number of archive entries processed.
MossReq — Original batch request.
Archive format, such as ‘tar’ or ‘zip’.
Raises:
BatchTruncatedError: If cont_on_err is False and the archive contains fewer entries than requested.
Get MossOut for the current file being extracted.
In multipart mode (when moss_resp is provided), uses the actual response metadata. In streaming mode, infers MossOut from the request as streaming mode streams only content (no metadata).
Parameters:
Index of the file in the batch
Length of file content in bytes (used to set size in streaming mode)
MossReq — Original batch request
Optional[ MossResp ] — Response metadata (None for streaming mode)
Archive member name used to identify server failure markers
Returns: MossOut
Response metadata for this file
Handle individual file extraction errors.
If cont_on_err is enabled, logs the error and allows continuation. Otherwise, raises a RuntimeError.
Parameters:
Name of the file that failed to extract
The exception that occurred
MossReq — Original batch request
Type of archive (e.g., ‘tar’, ‘zip’)
Raises:
RuntimeError: If cont_on_err is False
Extract from archive stream and yield individual file contents.
Sequentially streams the archive to avoid memory-intensive buffering.
Parameters:
HTTP response object containing connection for stream
Archive data stream or bytes
MossReq — Request that fetched the archive
Optional[ MossResp ] — Response metadata (None if streaming mode)
Raises:
RuntimeError: If stream extraction fails
Get formats supported by this extractor.
Returns: Tuple[str, ...]
Tuple[str]: Tuple of support formats