Heart Rate Estimation --------------------- .. _heartratenet: Data Input for Heart Rate Estimation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. _dataset_format: HeartRateNet expects directories of images in the format shown below. The images and ground truth labels are then converted to TFRecords for training. HeartRateNet Data Format ^^^^^^^^^^^^^^^^^^^^^^^^ .. code:: Subject_001/ ground_truth.csv image_timestamps.csv images/ 0000.bmp 0001.bmp . . N.bmp . . Subject_M/ ground_truth.csv image_timestamps.csv images/ 0000.bmp 0001.bmp . . Y.bmp HeartRateNet dataset conversion also requires an experiment spec to generate the TFRecords. The following gives a break down on the configuration file. Creating a Configuration File To Generate TFRecords ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ | **Field** | **Description** | **Data Type and Constraints** | **Recommended/Typical Value** | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`train_subjects` | List of the train subjects used to generate the train.tfrecord. | List of Strings | | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`validation_subjects` | List of the validation subjects used to generate the validation.tfrecord. | List of Strings | | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`test_subjects` | List of the test subjects used to generate the test.tfrecord. | List of Strings | | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`input_directory_path` | Data input directory path. | String | | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`data_directory_output_path` | Place to save the generated TFRecords. | String | | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`image_crop_size` | Image crop size. Image input size to model. | UnsignedInt | 72 | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`last_frame` | Last frame to use in the video. | String / UnsignedInt | 'all' or 300 | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`start_frame` | Frame number to begin preprocessing. | UnsignedInt | 60 | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`minimum_frames_to_preprocess` | Minimum frames required for band pass filter to work. | UnsignedInt | 34 | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`image_fps` | Frames per second of the captured video. | UnsignedInt | 30 | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`ppg_sampling_frequency` | Sampling frequency of the pulse oximeter. | UnsignedInt | 60 | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`lower_cutoff_frequency` | Lower cutoff frequency of predicted PPG signal. | Float | 0.67 | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`upper_cutoff_freq` | Upper cutoff frequency of predicted PPG signal. | Float | 4.0 | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`scale_factor` | Parameter used for face detection. | Float | 1.3 | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`min_neighbors` | Parameter used for face detection. | UnsignedInt | 5 | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`face_bbox_extension_factor` | Parameter used for face detection. | Float | 0.3 | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`low_heart_bpm_cutoff` | Lower beats per minute cut off limit. Used for alignment of green channel | Float | 20.0 | | | and is subtracted from the ground truth heart rate. | | | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`hight_heart_bpm_cutoff` | Upper beats per minute cut off limit. Used for alignment of green channel | Float | 20.0 | | | and is added to the ground truth heart rate. | | | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ |:code:`model` | Model to generate the TFRecords for. | String | HRNet_release | +-------------------------------------+-----------------------------------------------------------------------+-------------------------------+----------------------------------------+ Generating TFRecords ^^^^^^^^^^^^^^^^^^^^ To generate the TFRecords you can use the following command: .. code:: tlt heartratenet dataset_convert -e Required Arguments ****************** * :code:`-e, --experiment_spec_file`: Path to dataset spec. Sample Usage ************ Here is an example of using the dataset_convert to generate TFRecords: .. code:: tlt heartratenet dataset_convert -e /workspace/examples/heartratenet/specs/heartratenet_data_generation.yaml Creating a Configuration File to Train and Evaluate Heart Rate Network ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. _creating_a_configuration_file_two_branch_network: The configuration file for HeartRateNet has 5 key components in order to train and evaluate the models. The :code:`heartratenet train` and :code:`heartratenet evaluate` commands share the same configuration file. The key parts the configuration files are given below. * :code:`dataloader` allows you to configure the dataset. * :code:`model` allows you to configure the model. * :code:`loss` allows you to configure the loss. * :code:`optimizer` allows you to configure the optimizer. .. code:: __class_name__: HeartRateNetTrainer checkpoint_dir: "/workspace/tlt-experiments/heartratenet/model/" results_dir: "/workspace/tlt-experiments/heartratenet/" random_seed: 32 log_every_n_secs: 20 checkpoint_n_epoch: 1 num_epoch: 20 summary_every_n_steps: 1 infrequency_summary_every_n_steps: 0 last_step: 1 evaluation_window: 10 model_type: HRNet_release dataloader: __class_name__: HeartRateNetDataloader image_info: num_channels: 3 image_height: 72 image_width: 72 data_format: channels_last dataset_info: tfrecords_directory_path: /workspace/tlt-experiments/heartratenet/data tfrecord_folder_name: '' tfrecord_train_file_name: train.tfrecord tfrecord_test_file_name: test.tfrecord tfrecord_validation_file_name: validation.tfrecord model_info: model_type: HRNet_release model: __class_name__: HeartRateNet model_parameters: input_size: 72 data_format: channels_first conv_dropout_rate: 0.0 fully_connected_dropout_rate: 0.0 use_batch_norm: False model_type: HRNet_release frozen_blocks: 0 loss: __class_name__: HeartRateNetLoss loss_function_name: MSE optimizer: __class_name__: AdaDeltaOptimizer rho: 0.95 epsilon: 1.0e-7 learning_rate_schedule: __class_name__: ConstantLearningRateSchedule learning_rate: 1.0 +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ | **Field** | **Description** | **Data Type and Constraints** | **Recommended/Typical Value** | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |__class_name__ | Module name | String | HeartRateNetTrainer | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`checkpoint_dir` | The checkpoint dir to save the model | String | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`results_dir` | Directory to save the results | String | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`random_seed` | The random seed to use | Unsigned Int | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`log_every_n_secs` | Time in seconds to log the loss | Unsigned Int | 3000 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`checkpoint_n_epoch` | Frequency to save the current model | Unsigned Int | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`num_epoch` | Number of epochs to train the model for | Unsigned Int | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`summary_every_n_steps` | Frequency to give summary | Unsigned Int | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`evaluation_window` | The evaluation window on the predicted blood volume pulse | Unsigned Int | 10 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`model_type` | Model to use | String | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`dataloader` | Dataloader configuration | Message | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`model` | Model configuration | Message | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`loss` | Loss configuration | Message | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`optimizer` | Optimizer configuration | Message | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ Dataloader ********** The dataloader configuration defines how the data will get loaded. A detailed description can be found in the table below. +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ | **Field** | **Description** | **Data Type and Constraints** | **Recommended/Typical Value** | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`num_channels` | Number of channels used in video. | Unsigned Int | 3 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`image_height` | The image height. | Unsigned Int | 72 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`image_width` | The image width. | Unsigned Int | 72 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`data_format` | Channel format of data | String | channels_first | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`tfrecords_directory_path` | Path to the tfrecods directory | String | /workspace/tlt-experiments \ | | | | | /heartratenet/data | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`tfrecord_folder_name` | Folder name of the tfrecords | String | "" | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`tfrecord_train_file_name` | Tfrecord filename | String | train.tfrecord | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`tfrecord_test_file_name` | Tfrecord filename | String | test.tfrecord | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`tfrecord_validation_file_nam`| Tfrecord filename | String | validation.tfrecord | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`model_type` | Model to use | String | HRNet_release | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ Model ***** The model configuration allows you to customize your HeartRateNet model. +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ | **Field** | **Description** | **Data Type and Constraints** | **Recommended/Typical Value** | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`input_size` | Image input size to the model | Unsigned Int | 72 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`data_format` | Data format of the images | String | channels_first | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`conv_dropout_rate` | Convolution dropout rate | Float | 0.1 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`fully_connected_dropout_rate`| Fully connected layer dropout | Float | 0.1 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`use_batch_norm` | Flag to use batch normalization | Bool | False | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`model_type` | Model to use | String | HRNet_release | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`frozen_blocks` | How many layers to freeze in the model starting from the first layer | Unsigned Int | 0 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ Loss **** The loss configuration allows you to pick between different losses. +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ | **Field** | **Description** | **Data Type and Constraints** | **Recommended/Typical Value** | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`loss_function_name` | Loss function to use | String | MSE | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ Optimizer ********* The optimizer configuration allows you to configure the optimizer. +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ | **Field** | **Description** | **Data Type and Constraints** | **Recommended/Typical Value** | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |__class_name__ | Optimizer class name to use | String | AdaDeltaOptimizer | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`rho` | Parameter for optimizer | Float | 0.95 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`learning_rate` | Learning rate to use | Float | 1.0 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ Training the Model ^^^^^^^^^^^^^^^^^^ Train the HeartRateNet model using this command: .. code:: tlt heartratenet train -e -k -r -ll Required Arguments ****************** * :code:`-r, --results_dir`: Folder path to where training logs are saved. * :code:`-k, --key`: Provide the encryption key to decrypt the model. * :code:`-e, --experiment_spec_file`: Experiment specification file. Optional Arguments ****************** * :code:`-ll, --log_level`: Log level to use. Sample Usage ************ Here's an example of using the train command on a HeartRateNet model: .. code:: tlt heartratenet train -e /workspace/examples/heartratenet/specs/.yaml -k 'test' -r /workspace/tlt-experiments/gesturenet/ Evaluating the Model ^^^^^^^^^^^^^^^^^^^^ To run evaluation for a HeartRateNet model, use this command: .. code:: tlt heartratenet evaluate -e -m -k -r Required Arguments ****************** * :code:`-e, --experiment_spec_file`: Experiment spec file to set up evaluation experiment. * :code:`-m, --model_folder_path`: The model folder path to where the models are saved. * :code:`-k, --key`: The encryption key used to train the model. * :code:`-r, --results_dir`: The directory to save the evaluation results. Optional Arguments ****************** * :code:`-ll, --log_level`: Log level to use. Running Inference on the Model ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To run inference for HeartRateNet model, use this command: .. code:: tlt heartratenet inference -m --subject_infer_dir --subject -r -c -ll -k Required Arguments ****************** * :code:`-m, --model_full_path`: Path to the model to use. * :code:`--subject_infer_dir`: Path to the subject inference directory. * :code:`-k, --key`: Encryption key that was used to train the model. * :code:`--subject`: Subject to run inference on. * :code:`-r, --results_dir`: Directory to save the results. Optional Arguments ****************** * :code:`-c, --channels_format`: Channel format of data input. * :code:`-ll, --log_level`: Log level to use. Exporting the Model ^^^^^^^^^^^^^^^^^^^ HeartRateNet includes :code:`heartratenet export` command to export the trained model to TLT format. To generate encoded tlt model you can use the following commands: .. code:: tlt heartratenet export -m -k -o --log_level Required Arguments ****************** * :code:`-m, --model_filename`: Absolute path to the desired model file. * :code:`-k, --key`: Encryption key to use. * :code:`-o, --out_file`: Absolute path for the desired exported model. Optional Arguments ****************** * :code:`-t, --export_type`: Export type to use. * :code:`-ll, --log_level`: Log level to use.