MXNet Plugin API reference¶
-
class
nvidia.dali.plugin.mxnet.
DALIClassificationIterator
(pipelines, size, data_name='data', label_name='softmax_label', data_layout='NCHW', fill_last_batch=True, auto_reset=False, squeeze_labels=True, dynamic_shape=False, last_batch_padded=False)¶ DALI iterator for classification tasks for MXNet. It returns 2 outputs (data and label) in the form of MXNet’s DataBatch of NDArrays.
Calling
DALIClassificationIterator(pipelines, size, data_name, label_name, data_layout)
is equivalent to calling
DALIGenericIterator(pipelines, [(data_name, DALIClassificationIterator.DATA_TAG), (label_name, DALIClassificationIterator.LABEL_TAG)], size, data_layout)
Please keep in mind that NDArrays returned by the iterator are still owned by DALI. They are valid till the next iterator call. If the content needs to be preserved please copy it to another NDArray.
- Parameters
pipelines (list of nvidia.dali.pipeline.Pipeline) – List of pipelines to use
size (int) – Number of samples in the epoch (Usually the size of the dataset). Providing -1 means that the iterator will work until StopIteration is raised from the inside of iter_setup(). The options fill_last_batch, last_batch_padded and auto_reset don’t work in such case. It works with only one pipeline inside the iterator.
data_name (str, optional, default = 'data') – Data name for provided symbols.
label_name (str, optional, default = 'softmax_label') – Label name for provided symbols.
data_layout (str, optional, default = 'NCHW') – Either ‘NHWC’ or ‘NCHW’ - layout of the pipeline outputs.
fill_last_batch (bool, optional, default = True) – Whether to fill the last batch with data up to ‘self.batch_size’. The iterator would return the first integer multiple of self._num_gpus * self.batch_size entries which exceeds ‘size’. Setting this flag to False will cause the iterator to return exactly ‘size’ entries.
auto_reset (bool, optional, default = False) – Whether the iterator resets itself for the next epoch or it requires reset() to be called separately.
squeeze_labels (bool, optional, default = True) – Whether the iterator should squeeze the labels before copying them to the ndarray.
dynamic_shape (bool, optional, default = False) – Whether the shape of the output of the DALI pipeline can change during execution. If True, the mxnet.ndarray will be resized accordingly if the shape of DALI returned tensors changes during execution. If False, the iterator will fail in case of change.
last_batch_padded (bool, optional, default = False) – Whether the last batch provided by DALI is padded with the last sample or it just wraps up. In the conjunction with fill_last_batch it tells if the iterator returning last batch with data only partially filled with data from the current epoch is dropping padding samples or samples from the next epoch. If set to False next epoch will end sooner as data from it was consumed but dropped. If set to True next epoch would be the same length as the first one.
Example
With the data set
[1,2,3,4,5,6,7]
and the batch size 2:fill_last_batch = False, last_batch_padded = True -> last batch =
[7]
, next iteration will return[1, 2]
fill_last_batch = False, last_batch_padded = False -> last batch =
[7]
, next iteration will return[2, 3]
fill_last_batch = True, last_batch_padded = True -> last batch =
[7, 7]
, next iteration will return[1, 2]
fill_last_batch = True, last_batch_padded = False -> last batch =
[7, 1]
, next iteration will return[2, 3]
-
DATA_TAG
= 'data'¶
-
LABEL_TAG
= 'label'¶
-
next
()¶ Returns the next batch of data.
-
reset
()¶ Resets the iterator after the full epoch. DALI iterators do not support resetting before the end of the epoch and will ignore such request.
-
property
size
¶
-
class
nvidia.dali.plugin.mxnet.
DALIGenericIterator
(pipelines, output_map, size, data_layout='NCHW', fill_last_batch=True, auto_reset=False, squeeze_labels=True, dynamic_shape=False, last_batch_padded=False)¶ General DALI iterator for MXNet. It can return any number of outputs from the DALI pipeline in the form of MXNet’s DataBatch of NDArrays.
Please keep in mind that NDArrays returned by the iterator are still owned by DALI. They are valid till the next iterator call. If the content needs to be preserved please copy it to another NDArray.
- Parameters
pipelines (list of nvidia.dali.pipeline.Pipeline) – List of pipelines to use
output_map (list of (str, str)) – List of pairs (output_name, tag) which maps consecutive outputs of DALI pipelines to proper field in MXNet’s DataBatch. tag is one of DALIGenericIterator.DATA_TAG and DALIGenericIterator.LABEL_TAG mapping given output for data or label correspondingly. output_names should be distinct.
size (int) – Number of samples in the epoch (Usually the size of the dataset). Providing -1 means that the iterator will work until StopIteration is raised from the inside of iter_setup(). The options fill_last_batch, last_batch_padded and auto_reset don’t work in such case. It works with only one pipeline inside the iterator.
data_layout (str, optional, default = 'NCHW') – Either ‘NHWC’ or ‘NCHW’ - layout of the pipeline outputs.
fill_last_batch (bool, optional, default = True) – Whether to fill the last batch with data up to ‘self.batch_size’. The iterator would return the first integer multiple of self._num_gpus * self.batch_size entries which exceeds ‘size’. Setting this flag to False will cause the iterator to return exactly ‘size’ entries.
auto_reset (bool, optional, default = False) – Whether the iterator resets itself for the next epoch or it requires reset() to be called separately.
squeeze_labels (bool, optional, default = True) – Whether the iterator should squeeze the labels before copying them to the ndarray.
dynamic_shape (bool, optional, default = False) – Whether the shape of the output of the DALI pipeline can change during execution. If True, the mxnet.ndarray will be resized accordingly if the shape of DALI returned tensors changes during execution. If False, the iterator will fail in case of change.
last_batch_padded (bool, optional, default = False) – Whether the last batch provided by DALI is padded with the last sample or it just wraps up. In the conjunction with fill_last_batch it tells if the iterator returning last batch with data only partially filled with data from the current epoch is dropping padding samples or samples from the next epoch. If set to False next epoch will end sooner as data from it was consumed but dropped. If set to True next epoch would be the same length as the first one. For this happen, the option pad_last_batch in the reader need to be set to True as well.
Example
With the data set
[1,2,3,4,5,6,7]
and the batch size 2:fill_last_batch = False, last_batch_padded = True -> last batch =
[7]
, next iteration will return[1, 2]
fill_last_batch = False, last_batch_padded = False -> last batch =
[7]
, next iteration will return[2, 3]
fill_last_batch = True, last_batch_padded = True -> last batch =
[7, 7]
, next iteration will return[1, 2]
fill_last_batch = True, last_batch_padded = False -> last batch =
[7, 1]
, next iteration will return[2, 3]
-
DATA_TAG
= 'data'¶
-
LABEL_TAG
= 'label'¶
-
next
()¶ Returns the next batch of data.
-
reset
()¶ Resets the iterator after the full epoch. DALI iterators do not support resetting before the end of the epoch and will ignore such request.
-
property
size
¶
-
class
nvidia.dali.plugin.mxnet.
DALIGluonIterator
(pipelines, size, output_types=None, auto_reset=False, fill_last_batch=True, last_batch_padded=False)¶ General DALI iterator for MXNet with Gluon API. It can return any number of outputs from the DALI pipeline in the form of per GPU tuples. These tuples consisting of NDArrays (for outputs marked as DALIGluonIterator.DENSE_TAG) and list of NDArrays (for output marked as DALIGluonIterator.SPARSE_TAG).
Please keep in mind that NDArrays returned by the iterator are still owned by DALI. They are valid till the next iterator call. If the content needs to be preserved please copy it to another NDArray.
- Parameters
pipelines (list of nvidia.dali.pipeline.Pipeline) – List of pipelines to use
size (int) – Number of samples in the epoch (Usually the size of the dataset). Providing -1 means that the iterator will work until StopIteration is raised from the inside of iter_setup(). The options fill_last_batch, last_batch_padded and auto_reset don’t work in such case. It works with only one pipeline inside the iterator.
output_types (list of str, optional, default = None) – List of tags indicating whether the pipeline(s) output batch is uniform (all the samples have the same size) or not. Batch output marked as the former will be returned as a single NDArray, the latter will be returned as a list of NDArray. Must be either DALIGluonIterator.DENSE_TAG or DALIGluonIterator.SPARSE_TAG. Length of output_types must match the number of output of the pipeline(s). If not set, all outputs are considered to be marked with DALIGluonIterator.DENSE_TAG.
auto_reset (bool, optional, default = False) – Whether the iterator resets itself for the next epoch or it requires reset() to be called separately.
fill_last_batch (bool, optional, default = True) – Whether to fill the last batch with data up to ‘self.batch_size’. The iterator would return the first integer multiple of self._num_gpus * self.batch_size entries which exceeds ‘size’. Setting this flag to False will cause the iterator to return exactly ‘size’ entries.
last_batch_padded (bool, optional, default = False) – Whether the last batch provided by DALI is padded with the last sample or it just wraps up. In the conjunction with fill_last_batch it tells if the iterator returning last batch with data only partially filled with data from the current epoch is dropping padding samples or samples from the next epoch. If set to False next epoch will end sooner as data from it was consumed but dropped. If set to True next epoch would be the same length as the first one. For this happen, the option pad_last_batch in the reader need to be set to True as well.
Example
With the data set
[1,2,3,4,5,6,7]
and the batch size 2:fill_last_batch = False, last_batch_padded = True -> last batch =
[7]
, next iteration will return[1, 2]
fill_last_batch = False, last_batch_padded = False -> last batch =
[7]
, next iteration will return[2, 3]
fill_last_batch = True, last_batch_padded = True -> last batch =
[7, 7]
, next iteration will return[1, 2]
fill_last_batch = True, last_batch_padded = False -> last batch =
[7, 1]
, next iteration will return[2, 3]
-
DENSE_TAG
= 'dense'¶
-
SPARSE_TAG
= 'sparse'¶
-
next
()¶ Returns the next batch of data.
-
reset
()¶ Resets the iterator after the full epoch. DALI iterators do not support resetting before the end of the epoch and will ignore such request.
-
property
size
¶
-
nvidia.dali.plugin.mxnet.
feed_ndarray
(dali_tensor, arr)¶ Copy contents of DALI tensor to MXNet’s NDArray.
- Parameters
dali_tensor (nvidia.dali.backend.TensorCPU or nvidia.dali.backend.TensorGPU) – Tensor from which to copy
arr (mxnet.nd.NDArray) – Destination of the copy
-
nvidia.dali.plugin.mxnet.
get_mx_array
(shape, ctx=None, dtype=None)¶
-
nvidia.dali.plugin.mxnet.
non_empty
(shape)¶