Heart Rate Estimation --------------------- .. _heartratenet: Data Input for Heart Rate Estimation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. _dataset_format: See the :ref:`Data Annotation Format ` page for more information about the data format for Heart Rate Estimation. Creating a Configuration File to Generate TFRecords ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ HeartRateNet dataset conversion also requires an experiment spec to generate the TFRecords. A detailed description is included in the table below. +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ | **Field** | **Description** | **Data Type and Constraints** | **Recommended/Typical Value** | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`train_subjects` | A list of training subjects used to generate the :code:`train.tfrecord` | List of Strings | | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`validation_subjects` | A list of the validation subjects used to generate the :code:`validation.tfrecord` | List of Strings | | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`test_subjects` | A list of the test subjects used to generate the :code:`test.tfrecord` | List of Strings | | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`input_directory_path` | The data input directory path | String | | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`data_directory_output_path` | Place to save the generated TFRecords | String | | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`image_crop_size` | The image crop size, which is the image input size to the model | UnsignedInt | 72 | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`last_frame` | The last frame to use in the video | String / UnsignedInt | 'all' or 300 | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`start_frame` | The frame number at which to begin preprocessing | UnsignedInt | 60 | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`minimum_frames_to_preprocess` | The minimum frames required for the band pass filter to work | UnsignedInt | 34 | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`image_fps` | The frames per second of the captured video | UnsignedInt | 30 | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`ppg_sampling_frequency` | The sampling frequency of the pulse oximeter | UnsignedInt | 60 | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`lower_cutoff_frequency` | The lower cutoff frequency of the predicted PPG signal | Float | 0.67 | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`upper_cutoff_freq` | The upper cutoff frequency of the predicted PPG signal | Float | 4.0 | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`scale_factor` | A parameter used for face detection | Float | 1.3 | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`min_neighbors` | A parameter used for face detection | UnsignedInt | 5 | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`face_bbox_extension_factor` | A parameter used for face detection | Float | 0.3 | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`low_heart_bpm_cutoff` | The lower beats per minute cutoff limit. Used for alignment of the | Float | 20.0 | | | green channel and is subtracted from the ground-truth heart rate. | | | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`hight_heart_bpm_cutoff` | The upper beats per minute cutoff limit. Used for alignment of the | Float | 20.0 | | | green channel and is added to the ground truth heart rate. | | | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`model` | The model to generate the TFRecords for | String | HRNet_release | +-------------------------------------+----------------------------------------------------------------------------------------+-------------------------------+-------------------------------+ Generating TFRecords ^^^^^^^^^^^^^^^^^^^^ To generate TFRecords, use the following command: .. code:: tlt heartratenet dataset_convert -e Required Arguments ****************** * :code:`-e, --experiment_spec_file`: The path to the dataset spec Sample Usage ************ Here is an example of using :code:`dataset_convert` to generate TFRecords: .. code:: tlt heartratenet dataset_convert -e /workspace/examples/heartratenet/specs/heartratenet_data_generation.yaml Creating a Configuration File to Train and Evaluate Heart Rate Network ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. _creating_a_configuration_file_two_branch_network: The configuration file for HeartRateNet has four key components for training and evaluating the models. The key parts of the configuration files are given below. .. Note:: The :code:`heartratenet train` and :code:`heartratenet evaluate` commands share the same configuration file. * :code:`dataloader`: Configures the dataset. * :code:`model`: Configures the model. * :code:`loss`: Configures the loss. * :code:`optimizer`: Configures the optimizer. .. code:: __class_name__: HeartRateNetTrainer checkpoint_dir: "/workspace/tlt-experiments/heartratenet/model/" results_dir: "/workspace/tlt-experiments/heartratenet/" random_seed: 32 log_every_n_secs: 20 checkpoint_n_epoch: 1 num_epoch: 20 summary_every_n_steps: 1 infrequency_summary_every_n_steps: 0 last_step: 1 evaluation_window: 10 low_freq_cutoff: 0.67 high_freq_cutoff: 4.0 fps: 20.0 model_type: HRNet_release dataloader: __class_name__: HeartRateNetDataloader image_info: num_channels: 3 image_height: 72 image_width: 72 data_format: channels_first dataset_info: tfrecords_directory_path: /workspace/tlt-experiments/heartratenet/data tfrecord_folder_name: '' tfrecord_train_file_name: train.tfrecord tfrecord_test_file_name: test.tfrecord tfrecord_validation_file_name: validation.tfrecord model_info: model_type: HRNet_release model: __class_name__: HeartRateNet model_parameters: input_size: 72 data_format: channels_first conv_dropout_rate: 0.0 fully_connected_dropout_rate: 0.0 use_batch_norm: False model_type: HRNet_release frozen_blocks: 0 loss: __class_name__: HeartRateNetLoss loss_function_name: MSE optimizer: __class_name__: AdaDeltaOptimizer rho: 0.95 epsilon: 1.0e-7 learning_rate_schedule: __class_name__: ConstantLearningRateSchedule learning_rate: 1.0 +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ | **Field** | **Description** | **Data Type and Constraints** | **Recommended/Typical Value** | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`__class_name__` | The module name | String | HeartRateNetTrainer | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`checkpoint_dir` | The checkpoint directory to save the model to | String | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`results_dir` | The directory to save the results to | String | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`random_seed` | The random seed to use | Unsigned Int | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`log_every_n_secs` | The time interval in seconds to log the loss | Unsigned Int | 3000 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`checkpoint_n_epoch` | The frequency for saving the current model | Unsigned Int | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`num_epoch` | The number of epochs for training the model | Unsigned Int | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`summary_every_n_steps` | The frequency for giving summaries | Unsigned Int | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`evaluation_window` | The evaluation window on the predicted blood volume pulse in seconds | Unsigned Int | 10 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`low_freq_cutoff` | Heart rate high frequency cut off in Hz | Float | 0.67 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`high_freq_cutoff` | Heart rate low frequency cut off in Hz | Float | 0.40 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`fps` | Video frame rate in seconds | Float | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`model_type` | The model to use | String | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`dataloader` | The dataloader configuration | Message | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`model` | The model configuration | Message | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`loss` | The loss configuration | Message | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`optimizer` | The optimizer configuration | Message | | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ Dataloader ********** The dataloader configuration defines how the data will be loaded. +-------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ | **Field** | **Description** | **Data Type and Constraints** | **Recommended/Typical Value** | +-------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`num_channels` | The number of channels used in the video | Unsigned Int | 3 | +-------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`image_height` | The image height | Unsigned Int | 72 | +-------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`image_width` | The image width | Unsigned Int | 72 | +-------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`data_format` | The channel format of the data | String | channels_first | +-------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`tfrecords_directory_path` | The path to the tfrecods directory | String | /workspace/tlt-experiments \ | | | | | /heartratenet/data | +-------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`tfrecord_folder_name` | The folder name for the tfrecords | String | "" | +-------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`tfrecord_train_file_name` | The Tfrecord train filename | String | train.tfrecord | +-------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`tfrecord_test_file_name` | The Tfrecord test filename | String | test.tfrecord | +-------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`tfrecord_validation_file_name`| The Tfrecord validation filename | String | validation.tfrecord | +-------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`model_type` | The Model to use | String | HRNet_release | +-------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ Model ***** The model configuration allows you to customize your HeartRateNet model. +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ | **Field** | **Description** | **Data Type and Constraints** | **Recommended/Typical Value** | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`input_size` | The size of the image input to the model | Unsigned Int | 72 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`data_format` | The data format of the images | String | channels_first | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`conv_dropout_rate` | The convolution dropout rate | Float | 0.1 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`fully_connected_dropout_rate`| The fully connected layer dropout rate | Float | 0.1 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`use_batch_norm` | A flag specifying whether to use batch normalization | Bool | False | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`model_type` | The model to use | String | HRNet_release | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`frozen_blocks` | The number of layers to freeze in the model starting from the first layer | Unsigned Int | 0 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ Loss **** The loss configuration allows you to pick between different losses. +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ | **Field** | **Description** | **Data Type and Constraints** | **Recommended/Typical Value** | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`loss_function_name` | The loss function to use | String | MSE | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ Optimizer ********* The optimizer configuration allows you to configure the optimizer. +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ | **Field** | **Description** | **Data Type and Constraints** | **Recommended/Typical Value** | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`__class_name__` | The optimizer class name to use | String | AdaDeltaOptimizer | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`rho` | The parameter for the optimizer | Float | 0.95 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ |:code:`learning_rate` | The learning rate to use | Float | 1.0 | +------------------------------------+--------------------------------------------------------------------------------+-------------------------------+-------------------------------+ Training the Model ^^^^^^^^^^^^^^^^^^ Train the HeartRateNet model using the :code:`train` command: .. code:: tlt heartratenet train -e -k -r -ll Required Arguments ****************** * :code:`-r, --results_dir`: The folder path where training logs are saved * :code:`-k, --key`: The encryption key to decrypt the model * :code:`-e, --experiment_spec_file`: The experiment specification file Optional Arguments ****************** * :code:`-ll, --log_level`: The log level to use. Sample Usage ************ Here's an example of using the :code:`train` command on a HeartRateNet model: .. code:: tlt heartratenet train -e /workspace/examples/heartratenet/specs/.yaml -k 'test' -r /workspace/tlt-experiments/gesturenet/ Evaluating the Model ^^^^^^^^^^^^^^^^^^^^ Use the :code:`evaluate` command to run evaluation for a HeartRateNet model: .. code:: tlt heartratenet evaluate -e -m -k -r Required Arguments ****************** * :code:`-e, --experiment_spec_file`: The experiment spec file to set up evaluation experiment * :code:`-m, --model_folder_path`: The model folder path where the models are saved * :code:`-k, --key`: The encryption key used to train the model * :code:`-r, --results_dir`: The directory to save the evaluation results Optional Arguments ****************** * :code:`-ll, --log_level`: The log level to use. Running Inference on the Model ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To run inference for a HeartRateNet model, use this command: .. code:: tlt heartratenet inference -m --subject_infer_dir --subject -f