PaddlePaddle Plugin API reference

class nvidia.dali.plugin.paddle.DALIClassificationIterator(pipelines, size, auto_reset=False, fill_last_batch=True, dynamic_shape=False, last_batch_padded=False)

DALI iterator for classification tasks for Paddle. It returns 2 outputs (data and label) in the form of LoDTensor.

Calling

DALIClassificationIterator(pipelines, size)

is equivalent to calling

DALIGenericIterator(pipelines, ["data", "label"], size)

Please keep in mind that Tensors returned by the iterator are still owned by DALI. They are valid till the next iterator call. If the content needs to be preserved please copy it to another tensor.

Parameters

  • pipelines (list of nvidia.dali.pipeline.Pipeline) – List of pipelines to use

  • size (int) – Number of samples in the epoch (Usually the size of the dataset). Providing -1 means that the iterator will work until StopIteration is raised from the inside of iter_setup(). The options fill_last_batch, last_batch_padded and auto_reset don’t work in such case. It works with only one pipeline inside the iterator.

  • auto_reset (bool, optional, default = False) – Whether the iterator resets itself for the next epoch or it requires reset() to be called separately.

  • fill_last_batch (bool, optional, default = True) – Whether to return a fraction of a full batch of data such that the total entries returned by the iterator == ‘size’. Setting this flag to False will cause the iterator to return the first integer multiple

  • dynamic_shape (bool, optional, default = False) – Whether the shape of the output of the DALI pipeline can change during execution. If True, the LoDtensor will be resized accordingly if the shape of DALI returned tensors changes during execution. If False, the iterator will fail in case of change.

  • last_batch_padded (bool, optional, default = False) – Whether the last batch provided by DALI is padded with the last sample or it just wraps up. In the conjunction with fill_last_batch it tells if the iterator returning last batch with data only partially filled with data from the current epoch is dropping padding samples or samples from the next epoch. If set to False next epoch will end sooner as data from it was consumed but dropped. If set to True next epoch would be the same length as the first one.

Example

With the data set [1,2,3,4,5,6,7] and the batch size 2:

fill_last_batch = False, last_batch_padded = True -> last batch = [7], next iteration will return [1, 2]

fill_last_batch = False, last_batch_padded = False -> last batch = [7], next iteration will return [2, 3]

fill_last_batch = True, last_batch_padded = True -> last batch = [7, 7], next iteration will return [1, 2]

fill_last_batch = True, last_batch_padded = False -> last batch = [7, 1], next iteration will return [2, 3]

next()

Returns the next batch of data.

reset()

Resets the iterator after the full epoch. DALI iterators do not support resetting before the end of the epoch and will ignore such request.

class nvidia.dali.plugin.paddle.DALIGenericIterator(pipelines, output_map, size, auto_reset=False, fill_last_batch=True, dynamic_shape=False, last_batch_padded=False)

General DALI iterator for Paddle. It can return any number of outputs from the DALI pipeline in the form of Paddle’s Tensors.

Please keep in mind that Tensors returned by the iterator are still owned by DALI. They are valid till the next iterator call. If the content needs to be preserved please copy it to another tensor.

Parameters

  • pipelines (list of nvidia.dali.pipeline.Pipeline) – List of pipelines to use

  • output_map (list of str or pair of type (str, int)) – The strings maps consecutive outputs of DALI pipelines to user specified name. Outputs will be returned from iterator as dictionary of those names. Each name should be distinct. Item can also be a pair of (str, int), where the int value specifies the LoD level of the resulting LoDTensor.

  • size (int) – Number of samples in the epoch (Usually the size of the dataset). Providing -1 means that the iterator will work until StopIteration is raised from the inside of iter_setup(). The options fill_last_batch, last_batch_padded and auto_reset don’t work in such case. It works with only one pipeline inside the iterator.

  • auto_reset (bool, optional, default = False) – Whether the iterator resets itself for the next epoch or it requires reset() to be called separately.

  • fill_last_batch (bool, optional, default = True) – Whether to return a fraction of a full batch of data such that the total entries returned by the iterator == ‘size’. Setting this flag to False will cause the iterator to return the first integer multiple of self._num_gpus * self.batch_size which exceeds ‘size’.

  • dynamic_shape (bool, optional, default = False) – Whether the shape of the output of the DALI pipeline can change during execution. If True, the LoDTensor will be resized accordingly if the shape of DALI returned tensors changes during execution. If False, the iterator will fail in case of change.

  • last_batch_padded (bool, optional, default = False) – Whether the last batch provided by DALI is padded with the last sample or it just wraps up. In the conjunction with fill_last_batch it tells if the iterator returning last batch with data only partially filled with data from the current epoch is dropping padding samples or samples from the next epoch. If set to False next epoch will end sooner as data from it was consumed but dropped. If set to True next epoch would be the same length as the first one. For this happen, the option pad_last_batch in the reader need to be set to True as well.

Example

With the data set [1,2,3,4,5,6,7] and the batch size 2:

fill_last_batch = False, last_batch_padded = True -> last batch = [7], next iteration will return [1, 2]

fill_last_batch = False, last_batch_padded = False -> last batch = [7], next iteration will return [2, 3]

fill_last_batch = True, last_batch_padded = True -> last batch = [7, 7], next iteration will return [1, 2]

fill_last_batch = True, last_batch_padded = False -> last batch = [7, 1], next iteration will return [2, 3]

next()

Returns the next batch of data.

reset()

Resets the iterator after the full epoch. DALI iterators do not support resetting before the end of the epoch and will ignore such request.

nvidia.dali.plugin.paddle.feed_ndarray(dali_tensor, ptr)

Copy contents of DALI tensor to Paddle’s Tensor.

Parameters
nvidia.dali.plugin.paddle.lod_tensor_clip(lod_tensor, size)
nvidia.dali.plugin.paddle.recursive_length(tensor, lod_level)
nvidia.dali.plugin.paddle.to_paddle_type(tensor)

Get paddle dtype for given tensor or tensor list

Parameters

tensor – tensor or tensor list

Returns: fluid.core.VarDesc.VarType