Heart Rate Estimation¶

Data Input for Heart Rate Estimation¶

HeartRateNet expects directories of images in the format shown below. The images and ground truth labels are then converted to TFRecords for training.

HeartRateNet Data Format¶

Subject_001/
    ground_truth.csv
    image_timestamps.csv
    images/
        0000.bmp
        0001.bmp
            .
            .
        N.bmp
.
.
Subject_M/
    ground_truth.csv
    image_timestamps.csv
    images/
        0000.bmp
        0001.bmp
            .
            .
        Y.bmp

HeartRateNet dataset conversion also requires an experiment spec to generate the TFRecords. The following gives a break down on the configuration file.

Creating a Configuration File To Generate TFRecords¶

Field	Description \| Data Type and Constraints \| Recommended/Typical Value
`train_subjects`	List of the train subjects used to generate the train.tfrecord. \| List of Strings \|
`validation_subjects`	List of the validation subjects used to generate the validation.tfrecord. \| List of Strings \|
`test_subjects`	List of the test subjects used to generate the test.tfrecord. \| List of Strings \|
`input_directory_path`	Data input directory path. \| String \|
`data_directory_output_path`	Place to save the generated TFRecords. \| String \|
`image_crop_size`	Image crop size. Image input size to model. \| UnsignedInt \| 72
`last_frame`	Last frame to use in the video. \| String / UnsignedInt \| ‘all’ or 300
`start_frame`	Frame number to begin preprocessing. \| UnsignedInt \| 60
`minimum_frames_to_preprocess`	Minimum frames required for band pass filter to work. \| UnsignedInt \| 34
`image_fps`	Frames per second of the captured video. \| UnsignedInt \| 30
`ppg_sampling_frequency`	Sampling frequency of the pulse oximeter. \| UnsignedInt \| 60
`lower_cutoff_frequency`	Lower cutoff frequency of predicted PPG signal. \| Float \| 0.67
`upper_cutoff_freq`	Upper cutoff frequency of predicted PPG signal. \| Float \| 4.0
`scale_factor`	Parameter used for face detection. \| Float \| 1.3
`min_neighbors`	Parameter used for face detection. \| UnsignedInt \| 5
`face_bbox_extension_factor`	Parameter used for face detection. \| Float \| 0.3
`low_heart_bpm_cutoff`	Lower beats per minute cut off limit. Used for alignment of green channel \| Float \| 20.0 and is subtracted from the ground truth heart rate. \| \|
`hight_heart_bpm_cutoff`	Upper beats per minute cut off limit. Used for alignment of green channel \| Float \| 20.0 and is added to the ground truth heart rate. \| \|
`model`	Model to generate the TFRecords for. \| String \| HRNet_release

Generating TFRecords¶

To generate the TFRecords you can use the following command:

tlt heartratenet dataset_convert -e <experiment_spec_file>

Required Arguments¶

-e, --experiment_spec_file: Path to dataset spec.

Sample Usage¶

Here is an example of using the dataset_convert to generate TFRecords:

tlt heartratenet dataset_convert -e /workspace/examples/heartratenet/specs/heartratenet_data_generation.yaml

Creating a Configuration File to Train and Evaluate Heart Rate Network¶

The configuration file for HeartRateNet has 5 key components in order to train and evaluate the models. The heartratenet train and heartratenet evaluate commands share the same configuration file. The key parts the configuration files are given below.

dataloader allows you to configure the dataset.
model allows you to configure the model.
loss allows you to configure the loss.
optimizer allows you to configure the optimizer.

__class_name__: HeartRateNetTrainer
checkpoint_dir: "/workspace/tlt-experiments/heartratenet/model/"
results_dir: "/workspace/tlt-experiments/heartratenet/"
random_seed: 32
log_every_n_secs: 20
checkpoint_n_epoch: 1
num_epoch: 20
summary_every_n_steps: 1
infrequency_summary_every_n_steps: 0
last_step: 1
evaluation_window: 10
model_type: HRNet_release
dataloader:
    __class_name__: HeartRateNetDataloader
    image_info:
        num_channels: 3
        image_height: 72
        image_width: 72
        data_format: channels_last
    dataset_info:
        tfrecords_directory_path: /workspace/tlt-experiments/heartratenet/data
        tfrecord_folder_name: ''
        tfrecord_train_file_name: train.tfrecord
        tfrecord_test_file_name: test.tfrecord
        tfrecord_validation_file_name: validation.tfrecord
    model_info:
        model_type: HRNet_release
model:
    __class_name__: HeartRateNet
    model_parameters:
        input_size: 72
        data_format: channels_first
        conv_dropout_rate: 0.0
        fully_connected_dropout_rate: 0.0
        use_batch_norm: False
        model_type: HRNet_release
        frozen_blocks: 0
loss:
    __class_name__: HeartRateNetLoss
    loss_function_name: MSE
optimizer:
    __class_name__: AdaDeltaOptimizer
    rho: 0.95
    epsilon: 1.0e-7
    learning_rate_schedule:
        __class_name__: ConstantLearningRateSchedule
        learning_rate: 1.0

Field	Description	Data Type and Constraints	Recommended/Typical Value
__class_name__	Module name	String	HeartRateNetTrainer
`checkpoint_dir`	The checkpoint dir to save the model	String
`results_dir`	Directory to save the results	String
`random_seed`	The random seed to use	Unsigned Int
`log_every_n_secs`	Time in seconds to log the loss	Unsigned Int	3000
`checkpoint_n_epoch`	Frequency to save the current model	Unsigned Int
`num_epoch`	Number of epochs to train the model for	Unsigned Int
`summary_every_n_steps`	Frequency to give summary	Unsigned Int
`evaluation_window`	The evaluation window on the predicted blood volume pulse	Unsigned Int	10
`model_type`	Model to use	String
`dataloader`	Dataloader configuration	Message
`model`	Model configuration	Message
`loss`	Loss configuration	Message
`optimizer`	Optimizer configuration	Message

Dataloader¶

The dataloader configuration defines how the data will get loaded. A detailed description can be found in the table below.

Field	Description	Data Type and Constraints	Recommended/Typical Value
`num_channels`	Number of channels used in video.	Unsigned Int	3
`image_height`	The image height.	Unsigned Int	72
`image_width`	The image width.	Unsigned Int	72
`data_format`	Channel format of data	String	channels_first
`tfrecords_directory_path`	Path to the tfrecods directory	String	/workspace/tlt-experiments /heartratenet/data
`tfrecord_folder_name`	Folder name of the tfrecords	String	“”
`tfrecord_train_file_name`	Tfrecord filename	String	train.tfrecord
`tfrecord_test_file_name`	Tfrecord filename	String	test.tfrecord
`tfrecord_validation_file_nam`	Tfrecord filename	String	validation.tfrecord
`model_type`	Model to use	String	HRNet_release

Model¶

The model configuration allows you to customize your HeartRateNet model.

Field	Description	Data Type and Constraints	Recommended/Typical Value
`input_size`	Image input size to the model	Unsigned Int	72
`data_format`	Data format of the images	String	channels_first
`conv_dropout_rate`	Convolution dropout rate	Float	0.1
`fully_connected_dropout_rate`	Fully connected layer dropout	Float	0.1
`use_batch_norm`	Flag to use batch normalization	Bool	False
`model_type`	Model to use	String	HRNet_release
`frozen_blocks`	How many layers to freeze in the model starting from the first layer	Unsigned Int	0

Loss¶

The loss configuration allows you to pick between different losses.

Field	Description	Data Type and Constraints	Recommended/Typical Value
`loss_function_name`	Loss function to use	String	MSE

Optimizer¶

The optimizer configuration allows you to configure the optimizer.

Field	Description	Data Type and Constraints	Recommended/Typical Value
__class_name__	Optimizer class name to use	String	AdaDeltaOptimizer
`rho`	Parameter for optimizer	Float	0.95
`learning_rate`	Learning rate to use	Float	1.0

Training the Model¶

Train the HeartRateNet model using this command:

tlt heartratenet train -e <experiment_spec>
                   -k <key>
                   -r <results_dir>
                   -ll <log_level>

Required Arguments¶

-r, --results_dir: Folder path to where training logs are saved.
-k, --key: Provide the encryption key to decrypt the model.
-e, --experiment_spec_file: Experiment specification file.

Optional Arguments¶

-ll, --log_level: Log level to use.

Sample Usage¶

Here’s an example of using the train command on a HeartRateNet model:

tlt heartratenet train -e /workspace/examples/heartratenet/specs/<config_file>.yaml
                   -k 'test'
                   -r /workspace/tlt-experiments/gesturenet/

Evaluating the Model¶

To run evaluation for a HeartRateNet model, use this command:

tlt heartratenet evaluate -e <experiment_spec_file>
                    -m <model_folder_path>
                    -k <key>
                    -r <results_dir>

Required Arguments¶

-e, --experiment_spec_file: Experiment spec file to set up evaluation experiment.
-m, --model_folder_path: The model folder path to where the models are saved.
-k, --key: The encryption key used to train the model.
-r, --results_dir: The directory to save the evaluation results.

Optional Arguments¶

-ll, --log_level: Log level to use.

Running Inference on the Model¶

To run inference for HeartRateNet model, use this command:

tlt heartratenet inference -m <model_full_path>
                       --subject_infer_dir <subject_infer_dir>
                       --subject <subject>
                       -r <results_dir>
                       -c <channels_format>
                       -ll <log_level>
                       -k <key>

Required Arguments¶

-m, --model_full_path: Path to the model to use.
--subject_infer_dir: Path to the subject inference directory.
-k, --key: Encryption key that was used to train the model.
--subject: Subject to run inference on.
-r, --results_dir: Directory to save the results.

Optional Arguments¶

-c, --channels_format: Channel format of data input.
-ll, --log_level: Log level to use.

Exporting the Model¶

HeartRateNet includes heartratenet export command to export the trained model to TLT format. To generate encoded tlt model you can use the following commands:

tlt heartratenet export -m <model_filename>
                    -k <key>
                    -o <output_file>
                    --log_level <log_level>

Required Arguments¶

-m, --model_filename: Absolute path to the desired model file.
-k, --key: Encryption key to use.
-o, --out_file: Absolute path for the desired exported model.

Optional Arguments¶

-t, --export_type: Export type to use.
-ll, --log_level: Log level to use.