Python SDKPythonPythonAistoreSdkMultiobj

aistore.sdk.multiobj.object_group

View as Markdown

Module Contents

Classes

NameDescription
ObjectGroupA class representing multiple objects within the same bucket. Only one of obj_names, obj_range, or obj_template

API

class aistore.sdk.multiobj.object_group.ObjectGroup(
    bck: aistore.sdk.bucket.Bucket,
    obj_names: typing.Optional[typing.List] = None,
    obj_range: typing.Optional[aistore.sdk.multiobj.object_range.ObjectRange] = None,
    obj_template: typing.Optional[str] = None
)

Bases: AISSource

A class representing multiple objects within the same bucket. Only one of obj_names, obj_range, or obj_template should be provided.

Parameters:

bck
Bucket

Bucket the objects belong to

obj_names
list[str]Defaults to None

List of object names to include in this collection

obj_range
ObjectRangeDefaults to None

Range defining which object names in the bucket should be included

obj_template
strDefaults to None

String argument to pass as template value directly to api

client
RequestClient

The client bound to the bucket used by the ObjectGroup.

aistore.sdk.multiobj.object_group.ObjectGroup._parse_job_ids(
    response_text: str
) -> typing.List[str]

Parse job ID response that may contain comma-separated UUIDs. Used only for copy and archive operations which can return multiple UUIDs.

Parameters:

response_text
str

Raw response text

Returns: List[str]

List[str]: List of individual job IDs

aistore.sdk.multiobj.object_group.ObjectGroup.archive(
    archive_name: str,
    mime: str = '',
    to_bck: aistore.sdk.bucket.Bucket = None,
    include_source_name: bool = False,
    allow_append: bool = False,
    continue_on_err: bool = False
) -> typing.List[str]

Create or append to an archive

Parameters:

archive_name
str

Name of archive to create or append

mime
strDefaults to ''

MIME type of the content

to_bck
BucketDefaults to None

Destination bucket, defaults to current bucket

include_source_name
boolDefaults to False

Include the source bucket name in the archived objects’ names

allow_append
boolDefaults to False

Allow appending to an existing archive

continue_on_err
boolDefaults to False

Whether to continue if there is an error archiving a single object

Returns: List[str]

List[str]: List of job IDs that can be used to check the status of the operation

aistore.sdk.multiobj.object_group.ObjectGroup.copy(
    to_bck: aistore.sdk.bucket.Bucket,
    prepend: str = '',
    continue_on_error: bool = False,
    dry_run: bool = False,
    force: bool = False,
    latest: bool = False,
    sync: bool = False,
    num_workers: int = None
) -> typing.List[str]

Copies a list or range of objects in a bucket

Parameters:

to_bck
Bucket

Destination bucket

prepend
strDefaults to ''

Value to prepend to the name of copied objects

continue_on_error
boolDefaults to False

Whether to continue if there is an error copying a single object

dry_run
boolDefaults to False

Skip performing the copy and just log the intended actions

force
boolDefaults to False

Force this job to run over others in case it conflicts (see “limited coexistence” and xact/xreg/xreg.go)

latest
boolDefaults to False

GET the latest object version from the associated remote bucket

sync
boolDefaults to False

synchronize destination bucket with its remote (e.g., Cloud or remote AIS) source

num_workers
intDefaults to None

Number of concurrent workers (readers). Defaults to the number of target mountpaths if omitted or zero. A value of -1 indicates no workers at all (i.e., single-threaded execution). Any positive value will be adjusted not to exceed the number of target CPUs.

Returns: List[str]

List[str]: List of job IDs that can be used to check the status of the operation

Raises:

  • aistore.sdk.errors.AISError: All other types of errors with AIStore
  • requests.ConnectionError: Connection error
  • requests.ConnectionTimeout: Timed out connecting to AIStore
  • requests.exceptions.HTTPError: Service unavailable
  • requests.RequestException: “There was an ambiguous exception that occurred while handling…”
  • requests.ReadTimeout: Timed out receiving response from AIStore
aistore.sdk.multiobj.object_group.ObjectGroup.delete()

Deletes a list or range of objects in a bucket

Returns:

Job ID (as str) that can be used to check the status of the operation

Raises:

  • aistore.sdk.errors.AISError: All other types of errors with AIStore
  • requests.ConnectionError: Connection error
  • requests.ConnectionTimeout: Timed out connecting to AIStore
  • requests.exceptions.HTTPError: Service unavailable
  • requests.RequestException: “There was an ambiguous exception that occurred while handling…”
  • requests.ReadTimeout: Timed out receiving response from AIStore
aistore.sdk.multiobj.object_group.ObjectGroup.evict()

Evicts a list or range of objects in a bucket so that they are no longer cached in AIS NOTE: only Cloud buckets can be evicted.

Returns:

Job ID (as str) that can be used to check the status of the operation

Raises:

  • aistore.sdk.errors.AISError: All other types of errors with AIStore
  • requests.ConnectionError: Connection error
  • requests.ConnectionTimeout: Timed out connecting to AIStore
  • requests.exceptions.HTTPError: Service unavailable
  • requests.RequestException: “There was an ambiguous exception that occurred while handling…”
  • requests.ReadTimeout: Timed out receiving response from AIStore
aistore.sdk.multiobj.object_group.ObjectGroup.list_all_objects_iter(
    prefix: str = '',
    props: str = 'name,size'
) -> typing.Iterable[aistore.sdk.obj.object.Object]

Implementation of the abstract method from AISSource that provides an iterator of all the objects in this bucket matching the specified prefix.

Parameters:

prefix
strDefaults to ''

Limit objects selected by a given string prefix

props
strDefaults to 'name,size'

By default, will include all object properties. Pass in None to skip and avoid the extra API call.

aistore.sdk.multiobj.object_group.ObjectGroup.list_names() -> typing.List[str]

List all the object names included in this group of objects

Returns: List[str]

List of object names

aistore.sdk.multiobj.object_group.ObjectGroup.list_urls(
    prefix: str = '',
    etl: typing.Optional[aistore.sdk.etl.ETLConfig] = None
) -> typing.Iterable[str]

Implementation of the abstract method from AISSource that provides an iterator of full URLs to every object in this bucket matching the specified prefix Args: prefix (str, optional): Limit objects selected by a given string prefix etl (Optional[ETLConfig], optional): An optional ETL configuration. If provided, the URLs will include ETL processing parameters. Defaults to None.

aistore.sdk.multiobj.object_group.ObjectGroup.prefetch(
    blob_threshold: typing.Optional[int] = None,
    num_workers: typing.Optional[int] = None,
    latest: bool = False,
    continue_on_error: bool = False
)

Prefetches a list or range of objects in a bucket so that they are cached in AIS NOTE: only Cloud buckets can be prefetched.

Parameters:

latest
boolDefaults to False

GET the latest object version from the associated remote bucket

continue_on_error
boolDefaults to False

Whether to continue if there is an error prefetching a single object

blob_threshold
intDefaults to None

Utilize built-in blob-downloader for remote objects greater than the specified (threshold) size in bytes

num_workers
intDefaults to None

Number of concurrent workers (readers). Defaults to the number of target mountpaths if omitted or zero. A value of -1 indicates no workers at all (i.e., single-threaded execution). Any positive value will be adjusted not to exceed the number of target CPUs.

Returns:

Job ID (as str) that can be used to check the status of the operation

Raises:

  • aistore.sdk.errors.AISError: All other types of errors with AIStore
  • requests.ConnectionError: Connection error
  • requests.ConnectionTimeout: Timed out connecting to AIStore
  • requests.exceptions.HTTPError: Service unavailable
  • requests.RequestException: “There was an ambiguous exception that occurred while handling…”
  • requests.ReadTimeout: Timed out receiving response from AIStore
aistore.sdk.multiobj.object_group.ObjectGroup.transform(
    to_bck: aistore.sdk.bucket.Bucket,
    etl_name: str,
    timeout: str = DEFAULT_ETL_TIMEOUT,
    prepend: str = '',
    ext: typing.Optional[typing.Dict[str, str]] = None,
    continue_on_error: bool = False,
    dry_run: bool = False,
    force: bool = False,
    latest: bool = False,
    sync: bool = False,
    num_workers: typing.Optional[int] = None,
    etl_pipeline: typing.Optional[typing.List[str]] = None
)

Performs ETL operation on a list or range of objects in a bucket, placing the results in the destination bucket

Parameters:

to_bck
Bucket

Destination bucket

etl_name
str

Name of existing ETL to apply

timeout
strDefaults to DEFAULT_ETL_TIMEOUT

Timeout of the ETL job (e.g. 5m for 5 minutes)

prepend
strDefaults to ''

Value to prepend to the name of resulting transformed objects

ext
Dict[str, str]Defaults to None

Dict mapping each extension to the extension that will replace it (i.e. {“jpg”: “txt”})

continue_on_error
boolDefaults to False

Whether to continue if there is an error transforming a single object

dry_run
boolDefaults to False

Skip performing the transform and just log the intended actions

force
boolDefaults to False

Force this job to run over others in case it conflicts (see “limited coexistence” and xact/xreg/xreg.go)

latest
boolDefaults to False

GET the latest object version from the associated remote bucket

sync
boolDefaults to False

synchronize destination bucket with its remote (e.g., Cloud or remote AIS) source

num_workers
intDefaults to None

Number of concurrent workers (readers). Defaults to the number of target mountpaths if omitted or zero. A value of -1 indicates no workers at all (i.e., single-threaded execution). Any positive value will be adjusted not to exceed the number of target CPUs.

etl_pipeline
List[str]Defaults to None

List of ETL names to be used for the transformation pipeline.

Returns:

Job ID (as str) that can be used to check the status of the operation

Raises:

  • aistore.sdk.errors.AISError: All other types of errors with AIStore
  • requests.ConnectionError: Connection error
  • requests.ConnectionTimeout: Timed out connecting to AIStore
  • requests.exceptions.HTTPError: Service unavailable
  • requests.RequestException: “There was an ambiguous exception that occurred while handling…”
  • requests.ReadTimeout: Timed out receiving response from AIStore