TensorFlow integration

Overview

AIS cluster provides out-of-the-box integration with TensorFlow TFRecord format

Creating TensorFlow datasets from TFRecords stored in AIS cluster with tf.data.TFRecordDataset API. See S3 compatibility docs
Creating TensorFlow datasets from TAR files stored in AIS cluster with tf.data.TFRecordDataset API. The conversion is executed remotely, on the fly in the cluster.

Examples

Create TensorFlow dataset from TFRecords stored in AIS

1 import tensorflow as tf
2 import os
3 
4 os.environ["S3_ENDPOINT"] = CLUSTER_ENDPOINT
5 
6 # (...)
7 
8 train_dataset = tf.data.TFRecordDataset(filenames=[
9     "s3://tf/train-1.tfrecord",
10     "s3://tf/train-2.tfrecord",
11 ]).map(record_parser).batch(BATCH_SIZE)
12 
13 # (...)
14 
15 model.fit(train_dataset, ...)

Create TensorFlow dataset from TARs stored in AIS

1 import tensorflow as tf
2 import os
3 
4 os.environ["S3_ENDPOINT"] = CLUSTER_ENDPOINT
5 
6 # (...)
7 
8 # ?uuid query param to convert TAR to a transformed data.
9 
10 train_dataset = tf.data.TFRecordDataset(filenames=[
11     "s3://tf/train-1.tar?uuid=<uuid of tensorflow transformer>",
12     "s3://tf/train-2.tar?uuid=<uuid of tensorflow transformer>",
13 ]).map(record_parser).batch(BATCH_SIZE)
14 
15 # (...)
16 
17 model.fit(train_dataset, ...)

TensorFlow integration

TensorFlow integration

Table of Contents

Overview

Examples

Create TensorFlow dataset from TFRecords stored in AIS

Create TensorFlow dataset from TARs stored in AIS

Table of Contents

Overview

Examples

Create TensorFlow dataset from TFRecords stored in AIS

Create TensorFlow dataset from TARs stored in AIS

1	import tensorflow as tf
2	import os
3
4	os.environ["S3_ENDPOINT"] = CLUSTER_ENDPOINT
5
6	# (...)
7
8	train_dataset = tf.data.TFRecordDataset(filenames=[
9	"s3://tf/train-1.tfrecord",
10	"s3://tf/train-2.tfrecord",
11	]).map(record_parser).batch(BATCH_SIZE)
12
13	# (...)
14
15	model.fit(train_dataset, ...)