aistore.botocore_patch.botocore

Allow use of Amazon’s boto3 and botocore libraries with AIStore.

If you would like to interact with AIStore using the popular boto3 (or botocore) libraries, you can do so by importing this module (you’ll also need to install wrapt):

import wrapt
from aistore.botocore_patch import botocore

By way of explanation: by default, botocore (used by boto3) does not respect HTTP redirects that make use of the location header. Since AIStore uses them for load balancing, trying to use boto without patching it first results in various ClientErrors.

Importing this module patches the botocore library to respect the location header under the same conditions used by AIStore: the response code is 301, 302 or 307, and the “location” header is set.

Module Contents

Functions

Name	Description
`_ais_redirect_wrapper`	Intercept S3 redirections from AIStore and patch them to:
`_apply_patches`	When botocore.utils is imported, patch it to handle redirects.

API

aistore.botocore_patch.botocore._ais_redirect_wrapper(
    wrapped,
    instance,
    args,
    kwargs
)

Intercept S3 redirections from AIStore and patch them to:

Respect HTTP 301, 302 or 307 redirects using the ‘Location’ header
Fail after 3 redirects for a given operation (rather than just one)

We make no attempt to differentiate semantically between different 3XX codes; we just reuse the same method and request context, and hit the url specified by the “location” response header.

We also don’t try to handle caching via ‘Cache-Control’ or ‘Expires’ (though we might, later).

aistore.botocore_patch.botocore._apply_patches(
    module
)

When botocore.utils is imported, patch it to handle redirects.

We can’t future proof against everything, but we make a reasonable effort to find all likely “S3RegionRedirector” type classes below (at time of writing there are two, “S3RegionRedirector” and “S3RegionRedirectorv2”.

If the S3RegionRedirector.redirect_on_error method bound to each fails to match our signature, we’ll bail.