Python SDKPythonPythonAistoreSdkEtlWebserver

aistore.sdk.etl.webserver.utils

View as Markdown

Module Contents

Functions

NameDescription
compose_etl_direct_put_urlCompose the final direct PUT URL by combining components from multiple URLs.
parse_etl_pipelineParse ETL pipeline from header value with validation.
serialize_classPickle and base64-encode a user-provided ETLServer subclass for transmission.

API

aistore.sdk.etl.webserver.utils.compose_etl_direct_put_url(
    direct_put_url: str,
    host_target: str,
    obj_path: str
) -> str

Compose the final direct PUT URL by combining components from multiple URLs.

Scenarios:

  1. Pipeline stage: direct_put_url has no path → append object path.
  2. Offline transform: direct_put_url has a path → prepend host_target path (e.g. “/v1/etl/_object/<etl-name>/<etl-secret>/”) to validate request.

Returns: str: Complete direct PUT URL targeting the correct AIS node.

Parameters:

direct_put_url
str

Destination node’s direct PUT URL, possibly with path/query.

host_target
str

Base AIS target URL used for scheme and base path.

obj_path
str

Path of the object to PUT.

aistore.sdk.etl.webserver.utils.parse_etl_pipeline(
    pipeline_header: str
) -> typing.Tuple[str, str]

Parse ETL pipeline from header value with validation.

Parameters:

pipeline_header
str

Comma-separated pipeline URLs

Returns: str

Tuple of (first_url, remaining_pipeline_header)

Raises:

  • InvalidPipelineError: If pipeline header is malformed
aistore.sdk.etl.webserver.utils.serialize_class(
    cls: typing.Type[aistore.sdk.etl.webserver.base_etl_server.ETLServer],
    encoding: str = UTF_ENCODING
) -> str

Pickle and base64-encode a user-provided ETLServer subclass for transmission.

Parameters:

cls
Type[ETLServer]

A subclass of ETLServer to serialize.

encoding
strDefaults to UTF_ENCODING

The string encoding for the Base64 payload.

Returns: str

A Base64 string containing the pickled class.

Raises:

  • TypeError: If cls is not a subclass of ETLServer.