NVIDIA TAO Toolkit v4.0
NVIDIA TAO Release tlt.40

Heart Rate Estimation

See the Data Annotation Format page for more information about the data format for Heart Rate Estimation.

HeartRateNet dataset conversion also requires an experiment spec to generate the TFRecords. A detailed description is included in the table below.

Field

Description

Data Type and Constraints

Recommended/Typical Value

train_subjects

A list of training subjects used to generate the train.tfrecord

List of Strings

validation_subjects

A list of the validation subjects used to generate the validation.tfrecord

List of Strings

test_subjects

A list of the test subjects used to generate the test.tfrecord

List of Strings

input_directory_path

The data input directory path

String

data_directory_output_path

Place to save the generated TFRecords

String

image_crop_size

The image crop size, which is the image input size to the model

UnsignedInt

72

last_frame

The last frame to use in the video

String / UnsignedInt

‘all’ or 300

start_frame

The frame number at which to begin preprocessing

UnsignedInt

60

minimum_frames_to_preprocess

The minimum frames required for the band pass filter to work

UnsignedInt

34

image_fps

The frames per second of the captured video

UnsignedInt

30

ppg_sampling_frequency

The sampling frequency of the pulse oximeter

UnsignedInt

60

lower_cutoff_frequency

The lower cutoff frequency of the predicted PPG signal

Float

0.67

upper_cutoff_freq

The upper cutoff frequency of the predicted PPG signal

Float

4.0

scale_factor

A parameter used for face detection

Float

1.3

min_neighbors

A parameter used for face detection

UnsignedInt

5

face_bbox_extension_factor

A parameter used for face detection

Float

0.3

low_heart_bpm_cutoff

The lower beats per minute cutoff limit. Used for alignment of the green channel and is subtracted from the ground-truth heart rate.

Float

20.0

hight_heart_bpm_cutoff

The upper beats per minute cutoff limit. Used for alignment of the green channel and is added to the ground truth heart rate.

Float

20.0

model

The model to generate the TFRecords for

String

HRNet_release

To generate TFRecords, use the following command:

Copy
Copied!
            

tao heartratenet dataset_convert -e <experiment_spec_file>

Required Arguments

  • -e, --experiment_spec_file: The path to the dataset spec

Sample Usage

Here is an example of using dataset_convert to generate TFRecords:

Copy
Copied!
            

tao heartratenet dataset_convert -e /workspace/examples/heartratenet/specs/heartratenet_data_generation.yaml


The configuration file for HeartRateNet has four key components for training and evaluating the models. The key parts of the configuration files are given below.

Note

The heartratenet train and heartratenet evaluate commands share the same configuration file.

  • dataloader: Configures the dataset.

  • model: Configures the model.

  • loss: Configures the loss.

  • optimizer: Configures the optimizer.

Copy
Copied!
            

__class_name__: HeartRateNetTrainer checkpoint_dir: "/workspace/tao-experiments/heartratenet/model/" results_dir: "/workspace/tao-experiments/heartratenet/" random_seed: 32 log_every_n_secs: 20 checkpoint_n_epoch: 1 num_epoch: 20 summary_every_n_steps: 1 infrequency_summary_every_n_steps: 0 last_step: 1 evaluation_window: 10 low_freq_cutoff: 0.67 high_freq_cutoff: 4.0 fps: 20.0 model_type: HRNet_release dataloader: __class_name__: HeartRateNetDataloader image_info: num_channels: 3 image_height: 72 image_width: 72 data_format: channels_first dataset_info: tfrecords_directory_path: /workspace/tao-experiments/heartratenet/data tfrecord_folder_name: '' tfrecord_train_file_name: train.tfrecord tfrecord_test_file_name: test.tfrecord tfrecord_validation_file_name: validation.tfrecord model_info: model_type: HRNet_release model: __class_name__: HeartRateNet model_parameters: input_size: 72 data_format: channels_first conv_dropout_rate: 0.0 fully_connected_dropout_rate: 0.0 use_batch_norm: False model_type: HRNet_release frozen_blocks: 0 loss: __class_name__: HeartRateNetLoss loss_function_name: MSE optimizer: __class_name__: AdaDeltaOptimizer rho: 0.95 epsilon: 1.0e-7 learning_rate_schedule: __class_name__: ConstantLearningRateSchedule learning_rate: 1.0

Field

Description

Data Type and Constraints

Recommended/Typical Value

__class_name__

The module name

String

HeartRateNetTrainer

checkpoint_dir

The checkpoint directory to save the model to

String

results_dir

The directory to save the results to

String

random_seed

The random seed to use

Unsigned Int

log_every_n_secs

The time interval in seconds to log the loss

Unsigned Int

3000

checkpoint_n_epoch

The frequency for saving the current model

Unsigned Int

num_epoch

The number of epochs for training the model

Unsigned Int

summary_every_n_steps

The frequency for giving summaries

Unsigned Int

evaluation_window

The evaluation window on the predicted blood volume pulse in seconds

Unsigned Int

10

low_freq_cutoff

Heart rate high frequency cut off in Hz

Float

0.67

high_freq_cutoff

Heart rate low frequency cut off in Hz

Float

0.40

fps

Video frame rate in seconds

Float

model_type

The model to use

String

dataloader

The dataloader configuration

Message

model

The model configuration

Message

loss

The loss configuration

Message

optimizer

The optimizer configuration

Message

Dataloader

The dataloader configuration defines how the data will be loaded.

Field

Description

Data Type and Constraints

Recommended/Typical Value

num_channels

The number of channels used in the video

Unsigned Int

3

image_height

The image height

Unsigned Int

72

image_width

The image width

Unsigned Int

72

data_format

The channel format of the data

String

channels_first

tfrecords_directory_path

The path to the tfrecods directory

String

/workspace/tao-experiments /heartratenet/data

tfrecord_folder_name

The folder name for the tfrecords

String

“”

tfrecord_train_file_name

The Tfrecord train filename

String

train.tfrecord

tfrecord_test_file_name

The Tfrecord test filename

String

test.tfrecord

tfrecord_validation_file_name

The Tfrecord validation filename

String

validation.tfrecord

model_type

The Model to use

String

HRNet_release

Model

The model configuration allows you to customize your HeartRateNet model.

Field

Description

Data Type and Constraints

Recommended/Typical Value

input_size

The size of the image input to the model

Unsigned Int

72

data_format

The data format of the images

String

channels_first

conv_dropout_rate

The convolution dropout rate

Float

0.1

fully_connected_dropout_rate

The fully connected layer dropout rate

Float

0.1

use_batch_norm

A flag specifying whether to use batch normalization

Bool

False

model_type

The model to use

String

HRNet_release

frozen_blocks

The number of layers to freeze in the model starting from the first layer

Unsigned Int

0

Loss

The loss configuration allows you to pick between different losses.

Field

Description

Data Type and Constraints

Recommended/Typical Value

loss_function_name

The loss function to use

String

MSE

Optimizer

The optimizer configuration allows you to configure the optimizer.

Field

Description

Data Type and Constraints

Recommended/Typical Value

__class_name__

The optimizer class name to use

String

AdaDeltaOptimizer

rho

The parameter for the optimizer

Float

0.95

learning_rate

The learning rate to use

Float

1.0

Train the HeartRateNet model using the train command:

Copy
Copied!
            

tao heartratenet train -e <experiment_spec> -k <key> -r <results_dir> -ll <log_level>

Required Arguments

  • -r, --results_dir: The folder path where training logs are saved

  • -k, --key: The encryption key to decrypt the model

  • -e, --experiment_spec_file: The experiment specification file

Optional Arguments

  • -ll, --log_level: The log level to use.

Sample Usage

Here’s an example of using the train command on a HeartRateNet model:

Copy
Copied!
            

tao heartratenet train -e /workspace/examples/heartratenet/specs/<config_file>.yaml -k 'test' -r /workspace/tao-experiments/gesturenet/


Use the evaluate command to run evaluation for a HeartRateNet model:

Copy
Copied!
            

tao heartratenet evaluate -e <experiment_spec_file> -m <model_folder_path> -k <key> -r <results_dir>

Required Arguments

  • -e, --experiment_spec_file: The experiment spec file to set up evaluation experiment

  • -m, --model_folder_path: The model folder path where the models are saved

  • -k, --key: The encryption key used to train the model

  • -r, --results_dir: The directory to save the evaluation results

Optional Arguments

  • -ll, --log_level: The log level to use.

To run inference for a HeartRateNet model, use this command:

Copy
Copied!
            

tao heartratenet inference -m <model_full_path> --subject_infer_dir <subject_infer_dir> --subject <subject> -f <video fps> -r <results_dir> -c <channels_format> -ll <log_level> -k <key>

Required Arguments

  • -m, --model_full_path: Path to the model to use.

  • --subject_infer_dir: Path to the subject inference directory.

  • -f, --fps: FPS for input video

  • -k, --key: Encryption key that was used to train the model.

  • --subject: Subject to run inference on.

  • -r, --results_dir: Directory to save the results.

Optional Arguments

  • -i, --input_size: Size of input crop to network

  • -lf, --low_freq: Low band for band pass filter

  • -hf, --high_freq: High band for band pass filter

  • -c, --channels_format: Channel format of data input.

  • -ll, --log_level: Log level to use.

HeartRateNet includes the heartratenet export command to export the trained model to TAO format. To generate an encoded TAO model, use the following commands:

Copy
Copied!
            

tao heartratenet export -m <model_filename> -k <key> -o <output_file> --log_level <log_level>

Required Arguments

  • -m, --model_filename: The absolute path to the desired model file

  • -k, --key: The encryption key to use

  • -o, --out_file: The absolute path for the desired exported model

Optional Arguments

  • -t, --export_type: The export type to use

  • -ll, --log_level: The log level to use

The pretrained model for HeartRateNet provided through NGC is available by default with DeepStream 6.0.

For more details, refer to DeepStream TAO Integration for HeartRateNet.

© Copyright 2022, NVIDIA.. Last updated on Mar 23, 2023.