Python SDKPythonPythonAistoreSdkEtlWebserver

aistore.sdk.etl.webserver.utils

View as Markdown

Module Contents

Classes

NameDescription
_ResponseRawReaderWrap a requests.Response so close() releases the connection to the pool.

Functions

NameDescription
compose_etl_direct_put_urlCompose the final direct PUT URL by combining components from multiple URLs.
parse_etl_pipelineParse ETL pipeline from header value with validation.
serialize_classPickle and base64-encode a user-provided ETLServer subclass for transmission.

API

class aistore.sdk.etl.webserver.utils._ResponseRawReader(
    response: requests.Response
)

Wrap a requests.Response so close() releases the connection to the pool.

resp.raw.close() closes the socket but skips resp.close()‘s release_conn() call, which returns the socket to the keep-alive pool. Under early-close (mid-stream errors, client disconnects), that leaks connections. All reads are delegated to resp.raw; everything else falls through to resp.raw.

_raw
= response.raw
aistore.sdk.etl.webserver.utils._ResponseRawReader.__getattr__(
    name
)
aistore.sdk.etl.webserver.utils._ResponseRawReader.close()

Close the response so requests releases the connection to the pool.

aistore.sdk.etl.webserver.utils._ResponseRawReader.read(
    args = (),
    kwargs = {}
)

Read bytes from the underlying raw response stream.

aistore.sdk.etl.webserver.utils.compose_etl_direct_put_url(
    direct_put_url: str,
    host_target: str,
    obj_path: str
) -> str

Compose the final direct PUT URL by combining components from multiple URLs.

Scenarios:

  1. Pipeline stage: direct_put_url has no path → append object path.
  2. Offline transform: direct_put_url has a path → prepend host_target path (e.g. “/v1/etl/_object/<etl-name>/<etl-secret>/”) to validate request.

Returns: str: Complete direct PUT URL targeting the correct AIS node.

Parameters:

direct_put_url
str

Destination node’s direct PUT URL, possibly with path/query.

host_target
str

Base AIS target URL used for scheme and base path.

obj_path
str

Path of the object to PUT.

aistore.sdk.etl.webserver.utils.parse_etl_pipeline(
    pipeline_header: str
) -> typing.Tuple[str, str]

Parse ETL pipeline from header value with validation.

Parameters:

pipeline_header
str

Comma-separated pipeline URLs

Returns: str

Tuple of (first_url, remaining_pipeline_header)

Raises:

  • InvalidPipelineError: If pipeline header is malformed
aistore.sdk.etl.webserver.utils.serialize_class(
    cls: typing.Type[aistore.sdk.etl.webserver.base_etl_server.ETLServer],
    encoding: str = UTF_ENCODING
) -> str

Pickle and base64-encode a user-provided ETLServer subclass for transmission.

Parameters:

cls
Type[ETLServer]

A subclass of ETLServer to serialize.

encoding
strDefaults to UTF_ENCODING

The string encoding for the Base64 payload.

Returns: str

A Base64 string containing the pickled class.

Raises:

  • TypeError: If cls is not a subclass of ETLServer.