APIs, SDKs, and Compatibility

TensorFlow integration

View as Markdown

Table of Contents

Overview

AIS cluster provides out-of-the-box integration with TensorFlow TFRecord format

  • Creating TensorFlow datasets from TFRecords stored in AIS cluster with tf.data.TFRecordDataset API. See S3 compatibility docs
  • Creating TensorFlow datasets from TAR files stored in AIS cluster with tf.data.TFRecordDataset API. The conversion is executed remotely, on the fly in the cluster.

Examples

Create TensorFlow dataset from TFRecords stored in AIS

1import tensorflow as tf
2import os
3
4os.environ["S3_ENDPOINT"] = CLUSTER_ENDPOINT
5
6# (...)
7
8train_dataset = tf.data.TFRecordDataset(filenames=[
9    "s3://tf/train-1.tfrecord",
10    "s3://tf/train-2.tfrecord",
11]).map(record_parser).batch(BATCH_SIZE)
12
13# (...)
14
15model.fit(train_dataset, ...)

Create TensorFlow dataset from TARs stored in AIS

1import tensorflow as tf
2import os
3
4os.environ["S3_ENDPOINT"] = CLUSTER_ENDPOINT
5
6# (...)
7
8# ?uuid query param to convert TAR to a transformed data.
9
10train_dataset = tf.data.TFRecordDataset(filenames=[
11    "s3://tf/train-1.tar?uuid=<uuid of tensorflow transformer>",
12    "s3://tf/train-2.tar?uuid=<uuid of tensorflow transformer>",
13]).map(record_parser).batch(BATCH_SIZE)
14
15# (...)
16
17model.fit(train_dataset, ...)