Heart Rate Estimation

See the Data Annotation Format page for more information about the data format for Heart Rate Estimation.

HeartRateNet dataset conversion also requires an experiment spec to generate the TFRecords. A detailed description is included in the table below.

Field Description Data Type and Constraints Recommended/Typical Value
train_subjects A list of training subjects used to generate the train.tfrecord List of Strings
validation_subjects A list of the validation subjects used to generate the validation.tfrecord List of Strings
test_subjects A list of the test subjects used to generate the test.tfrecord List of Strings
input_directory_path The data input directory path String
data_directory_output_path Place to save the generated TFRecords String
image_crop_size The image crop size, which is the image input size to the model UnsignedInt 72
last_frame The last frame to use in the video String / UnsignedInt ‘all’ or 300
start_frame The frame number at which to begin preprocessing UnsignedInt 60
minimum_frames_to_preprocess The minimum frames required for the band pass filter to work UnsignedInt 34
image_fps The frames per second of the captured video UnsignedInt 30
ppg_sampling_frequency The sampling frequency of the pulse oximeter UnsignedInt 60
lower_cutoff_frequency The lower cutoff frequency of the predicted PPG signal Float 0.67
upper_cutoff_freq The upper cutoff frequency of the predicted PPG signal Float 4.0
scale_factor A parameter used for face detection Float 1.3
min_neighbors A parameter used for face detection UnsignedInt 5
face_bbox_extension_factor A parameter used for face detection Float 0.3
low_heart_bpm_cutoff The lower beats per minute cutoff limit. Used for alignment of the green channel and is subtracted from the ground-truth heart rate. Float 20.0
hight_heart_bpm_cutoff The upper beats per minute cutoff limit. Used for alignment of the green channel and is added to the ground truth heart rate. Float 20.0
model The model to generate the TFRecords for String HRNet_release

To generate TFRecords, use the following command:

Copy
Copied!
            

tao heartratenet dataset_convert -e <experiment_spec_file>

Required Arguments

  • -e, --experiment_spec_file: The path to the dataset spec

Sample Usage

Here is an example of using dataset_convert to generate TFRecords:

Copy
Copied!
            

tao heartratenet dataset_convert -e /workspace/examples/heartratenet/specs/heartratenet_data_generation.yaml


The configuration file for HeartRateNet has four key components for training and evaluating the models. The key parts of the configuration files are given below.

Note

The heartratenet train and heartratenet evaluate commands share the same configuration file.

  • dataloader: Configures the dataset.

  • model: Configures the model.

  • loss: Configures the loss.

  • optimizer: Configures the optimizer.

Copy
Copied!
            

__class_name__: HeartRateNetTrainer checkpoint_dir: "/workspace/tao-experiments/heartratenet/model/" results_dir: "/workspace/tao-experiments/heartratenet/" random_seed: 32 log_every_n_secs: 20 checkpoint_n_epoch: 1 num_epoch: 20 summary_every_n_steps: 1 infrequency_summary_every_n_steps: 0 last_step: 1 evaluation_window: 10 low_freq_cutoff: 0.67 high_freq_cutoff: 4.0 fps: 20.0 model_type: HRNet_release dataloader: __class_name__: HeartRateNetDataloader image_info: num_channels: 3 image_height: 72 image_width: 72 data_format: channels_first dataset_info: tfrecords_directory_path: /workspace/tao-experiments/heartratenet/data tfrecord_folder_name: '' tfrecord_train_file_name: train.tfrecord tfrecord_test_file_name: test.tfrecord tfrecord_validation_file_name: validation.tfrecord model_info: model_type: HRNet_release model: __class_name__: HeartRateNet model_parameters: input_size: 72 data_format: channels_first conv_dropout_rate: 0.0 fully_connected_dropout_rate: 0.0 use_batch_norm: False model_type: HRNet_release frozen_blocks: 0 loss: __class_name__: HeartRateNetLoss loss_function_name: MSE optimizer: __class_name__: AdaDeltaOptimizer rho: 0.95 epsilon: 1.0e-7 learning_rate_schedule: __class_name__: ConstantLearningRateSchedule learning_rate: 1.0

Field Description Data Type and Constraints Recommended/Typical Value
__class_name__ The module name String HeartRateNetTrainer
checkpoint_dir The checkpoint directory to save the model to String
results_dir The directory to save the results to String
random_seed The random seed to use Unsigned Int
log_every_n_secs The time interval in seconds to log the loss Unsigned Int 3000
checkpoint_n_epoch The frequency for saving the current model Unsigned Int
num_epoch The number of epochs for training the model Unsigned Int
summary_every_n_steps The frequency for giving summaries Unsigned Int
evaluation_window The evaluation window on the predicted blood volume pulse in seconds Unsigned Int 10
low_freq_cutoff Heart rate high frequency cut off in Hz Float 0.67
high_freq_cutoff Heart rate low frequency cut off in Hz Float 0.40
fps Video frame rate in seconds Float
model_type The model to use String
dataloader The dataloader configuration Message
model The model configuration Message
loss The loss configuration Message
optimizer The optimizer configuration Message

Dataloader

The dataloader configuration defines how the data will be loaded.

Field Description Data Type and Constraints Recommended/Typical Value
num_channels The number of channels used in the video Unsigned Int 3
image_height The image height Unsigned Int 72
image_width The image width Unsigned Int 72
data_format The channel format of the data String channels_first
tfrecords_directory_path The path to the tfrecods directory String /workspace/tao-experiments /heartratenet/data
tfrecord_folder_name The folder name for the tfrecords String “”
tfrecord_train_file_name The Tfrecord train filename String train.tfrecord
tfrecord_test_file_name The Tfrecord test filename String test.tfrecord
tfrecord_validation_file_name The Tfrecord validation filename String validation.tfrecord
model_type The Model to use String HRNet_release

Model

The model configuration allows you to customize your HeartRateNet model.

Field Description Data Type and Constraints Recommended/Typical Value
input_size The size of the image input to the model Unsigned Int 72
data_format The data format of the images String channels_first
conv_dropout_rate The convolution dropout rate Float 0.1
fully_connected_dropout_rate The fully connected layer dropout rate Float 0.1
use_batch_norm A flag specifying whether to use batch normalization Bool False
model_type The model to use String HRNet_release
frozen_blocks The number of layers to freeze in the model starting from the first layer Unsigned Int 0

Loss

The loss configuration allows you to pick between different losses.

Field Description Data Type and Constraints Recommended/Typical Value
loss_function_name The loss function to use String MSE

Optimizer

The optimizer configuration allows you to configure the optimizer.

Field Description Data Type and Constraints Recommended/Typical Value
__class_name__ The optimizer class name to use String AdaDeltaOptimizer
rho The parameter for the optimizer Float 0.95
learning_rate The learning rate to use Float 1.0

Train the HeartRateNet model using the train command:

Copy
Copied!
            

tao heartratenet train -e <experiment_spec> -k <key> -r <results_dir> -ll <log_level>

Required Arguments

  • -r, --results_dir: The folder path where training logs are saved

  • -k, --key: The encryption key to decrypt the model

  • -e, --experiment_spec_file: The experiment specification file

Optional Arguments

  • -ll, --log_level: The log level to use.

Sample Usage

Here’s an example of using the train command on a HeartRateNet model:

Copy
Copied!
            

tao heartratenet train -e /workspace/examples/heartratenet/specs/<config_file>.yaml -k 'test' -r /workspace/tao-experiments/gesturenet/


Use the evaluate command to run evaluation for a HeartRateNet model:

Copy
Copied!
            

tao heartratenet evaluate -e <experiment_spec_file> -m <model_folder_path> -k <key> -r <results_dir>

Required Arguments

  • -e, --experiment_spec_file: The experiment spec file to set up evaluation experiment

  • -m, --model_folder_path: The model folder path where the models are saved

  • -k, --key: The encryption key used to train the model

  • -r, --results_dir: The directory to save the evaluation results

Optional Arguments

  • -ll, --log_level: The log level to use.

To run inference for a HeartRateNet model, use this command:

Copy
Copied!
            

tao heartratenet inference -m <model_full_path> --subject_infer_dir <subject_infer_dir> --subject <subject> -f <video fps> -r <results_dir> -c <channels_format> -ll <log_level> -k <key>

Required Arguments

  • -m, --model_full_path: Path to the model to use.

  • --subject_infer_dir: Path to the subject inference directory.

  • -f, --fps: FPS for input video

  • -k, --key: Encryption key that was used to train the model.

  • --subject: Subject to run inference on.

  • -r, --results_dir: Directory to save the results.

Optional Arguments

  • -i, --input_size: Size of input crop to network

  • -lf, --low_freq: Low band for band pass filter

  • -hf, --high_freq: High band for band pass filter

  • -c, --channels_format: Channel format of data input.

  • -ll, --log_level: Log level to use.

HeartRateNet includes the heartratenet export command to export the trained model to TAO format. To generate an encoded TAO model, use the following commands:

Copy
Copied!
            

tao heartratenet export -m <model_filename> -k <key> -o <output_file> --log_level <log_level>

Required Arguments

  • -m, --model_filename: The absolute path to the desired model file

  • -k, --key: The encryption key to use

  • -o, --out_file: The absolute path for the desired exported model

Optional Arguments

  • -t, --export_type: The export type to use

  • -ll, --log_level: The log level to use

The pretrained model for HeartRateNet provided through NGC is available by default with DeepStream 6.0.

For more details, refer to DeepStream TAO Integration for HeartRateNet.

© Copyright 2023, NVIDIA.. Last updated on Sep 5, 2023.