{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Serialization\n",
    "\n",
    "## Overview\n",
    "\n",
    "This sample shows how to serialize the pipeline to a string.\n",
    "\n",
    "## Serialization\n",
    "\n",
    "In order to use C API or TensorFlow plugin (or just to save the pipeline with a model, so the training process is fully reproducible) we need to serialize the pipeline. \n",
    "\n",
    "Let us make a simple pipeline reading from MXNet recordIO format (for example of using other data formats please see other [examples](../index.rst)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nvidia.dali import pipeline_def, Pipeline\n",
    "import nvidia.dali.fn as fn\n",
    "import nvidia.dali.types as types\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import os.path\n",
    "\n",
    "test_data_root = os.environ[\"DALI_EXTRA_PATH\"]\n",
    "base = os.path.join(test_data_root, \"db\", \"recordio\")\n",
    "\n",
    "idx_files = [base + \"/train.idx\"]\n",
    "rec_files = [base + \"/train.rec\"]\n",
    "\n",
    "\n",
    "@pipeline_def\n",
    "def example_pipe():\n",
    "    encoded, labels = fn.readers.mxnet(path=rec_files, index_path=idx_files)\n",
    "    images = fn.decoders.image(encoded, device=\"mixed\", output_type=types.RGB)\n",
    "    images = fn.resize(\n",
    "        images,\n",
    "        interp_type=types.INTERP_LINEAR,\n",
    "        resize_shorter=fn.random.uniform(range=(256.0, 480.0)),\n",
    "    )\n",
    "    images = fn.crop_mirror_normalize(\n",
    "        images, dtype=types.FLOAT, crop=(224, 224), mean=[0.0, 0.0, 0.0], std=[1.0, 1.0, 1.0]\n",
    "    )\n",
    "    return images, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 16\n",
    "\n",
    "pipe = example_pipe(batch_size=batch_size, num_threads=2, device_id=0, seed=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now serialize this pipeline, using `serialize` function of the `Pipeline` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = pipe.serialize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to deserialize our pipeline in Python, we need to create another pipeline, this time using the generic `Pipeline` class. We give the same seed to the new pipeline, in order to compare the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe2 = Pipeline(batch_size=batch_size, num_threads=2, device_id=0, seed=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now use the serialized form of `pipe` object to make `pipe2` a copy of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe2.deserialize_and_build(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can compare the results of the 2 pipelines - original and deserialized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe.build()\n",
    "original_pipe_out = pipe.run()\n",
    "serialized_pipe_out = pipe2.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_difference(batch_1, batch_2):\n",
    "    return [np.sum(np.abs(batch_1.at(i) - batch_2.at(i))) for i in range(batch_size)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_images, _ = original_pipe_out\n",
    "serialized_images, _ = serialized_pipe_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0,\n",
       " 0.0]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "check_difference(original_images.as_cpu(), serialized_images.as_cpu())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Both pipelines give exactly the same results."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}