AIStoreOracle Cloud Infrastructure (“OCI”) has been supported via OCI’s Amazon S3 Compatibility API for quite some time. As such, AIStore’s support for the S3 protocol has enabled OCI’s Object Storage to be utilized as a backend. But OCI also provides a more optimized API. AIStore has now added support for using OCI Object Storage as a backend via this OCI-native protocol.
As with other backends, OCI Object Storage utilizing their OCI-native protocol is enabled
by including oci in the AIS_BACKEND_PROVIDERS environment variable. Doing so will
trigger the OCI Object Storage being linked and configured utilizing the OCI config file
found at ~/.oci/config. See AIS Getting Started for a helpful guide on
deployment steps.
You can override the location of this config file by setting the OCI_CLI_CONFIG_FILE
environment variable. Several of the values searched for in this configuration file may
also be overridden by setting various other environment variables:
Note that the OCI_PRIVATE_KEY environment variable should actually contain the (unencrypted)
contents of the key_file. For an (unencrypted) private key file located at ~/.oci/prikey.pem,
the OCI_PRIVATE_KEY environment variable may be set simply by:
While all existing buckets implicitly belong to specific OCI Compartments, in order to perform
operations like listing all buckets, AIStore must also be provided the OCID for a particular
Compartment by setting the OCI_COMPARTMENT_OCID environment variable to it’s OCID.
Here we will deploy a single storage target single proxy (gateway) AIStore instance with our desired OCI backend. First things first, ensure you have a properly formatted OCI CLI Config File (as described in OCI Config File Setup):
Be sure to set the CompartmentID’s OCID environment variable:
Now launch a simple AIStore instance:
As with other supported backends, the OCI Object Storage buckets are referenced with either the
oc: or oci: prefix. For a comparison between OCI CLI commands and their equivalent using the
AIStore CLI, see the following:
Note the addition of the --all on the ais ls command. This may be required if AIStore has
yet to cache anything from any bucket from the configured OCI Object Storage. Similarly, the
ais object show command may need to be adorned with --not-cached if that particular object
has yet to be accessed via AIStore.
Several behaviors of access to OCI Object Storage may be tuned via environment variables:
To demonstrate the advantage of “multi part upload/download” tuning, let’s start with a simple test file that is practical for accessing OCI Object Storage remotely:
Next, let’s shrink some of those “MULTI_PART” environment variables considerably:
Restart your local AIStore instance and perform a PUT operation on that test file. Note that the
test file z is much larger than the value of OCI_MAX_UPLOAD_SEGMENT_SIZE, thus a Multi-Part
Upload will be used.
Next, let’s up those thresholds to well exceed the size of the test file z:
Restart your local AIStore instance and repeat the PUT operation. Here, the Multi-Part Upload functionality will be bypassed.
With the Multi-Part Download threshold also forcing single-threaded downloading for now, we will see something like:
Finally, let’s return those threshold values to the original values:
Restart your local AIStore instance one last time and repeat the GET operation. Here, the Multi-Part Download functionality will be enabled.
To summarize, we crafted a modest test file comparing upload (PUT) and download (GET) performance both single-threaded and multi-threaded techniques by adjusting the thresholds at which the “Multi-Part” functionality was triggered. These results are summarized here:
Native support for OCI Object Storage is now available as an AIStore backend. This support includes tunable optimizations for uploading and downloading of large objects between an AIStore (it’s client and it’s local cache) and the OCI Object Storage infrastructure.
Looking ahead, we will be focusing on leveraging the native OCI Object Storage support to produce more optimal performance relative to what is possible when utilizing the S3 protocol translation/proxy capability,
