BEVFusion

BEVFusion is an 3D object-detection model that is included in the TAO. It supports the following tasks:

• convert

• train

• evaluate

• inference

The sample inferenc result:

TAO BEVFusion Inferece Image

These tasks may be invoked from the TAO Launcher using the following convention on the command line:

            

            
tao model bevfusion <sub_task> <args_per_subtask>
        

where args_per_subtask are the command-line arguments required for a given subtask. Each of these subtasks are explained as follows.

The dataset for BEVFusion contains point cloud data, rgb image and the corresponding annotations of 3D objects. The directory structure should be organized as KITTI directory structure.

            

            
/kitti
    /training
        /calib
          000000.txt
          000001.txt
            ...
          N.txt
        /image_2
          000000.png
          000001.png
            ...
          N.png
        /label_2
          000000.txt
          000001.txt
            ...
          N.txt
        /velodyne
          000000.bin
          000001.bin
            ...
          N.bin
    /ImageSets
        train.txt
        val.txt
        test.txt
        

Each .bin file should comply with the format described above. Each .txt label file should comply to the KITTI format.

Below is a sample BEVFusion spec file. It has six components -model, inference, evaluate, dataset and train-as well as several global parameters, which are described below. The format of the spec file is a YAML file.

Here’s a sample of the BEVFusion spec file:

            

            
results_dir: /results/bevfusion
dataset:
  type: KittiPersonDataset
  root_dir: /data/
  gt_box_type: camera
  default_cam_key: CAM2
  train_dataset:
    repeat_time: 2
    ann_file: /data/kitti_person_infos_train.pkl
    data_prefix:
      pts: training/velodyne_reduced
      img: training/image_2
    batch_size: 4
    num_workers: 8
  val_dataset:
    ann_file: /data/kitti_person_infos_val.pkl
    data_prefix:
      pts: training/velodyne_reduced
      img: training/image_2
    batch_size: 2
    num_workers: 4
  test_dataset:
    ann_file: /data/kitti_person_infos_val.pkl
    data_prefix:
      pts: training/velodyne_reduced
      img: training/image_2
    batch_size: 4
    num_workers: 4
model:
  type: BEVFusion
  point_cloud_range: [0, -40, -3, 70.4, 40, 1]
  voxel_size: [0.05, 0.05, 0.1]
  grid_size: [1440, 1440, 41]
train:
  num_gpus: 1
  num_nodes: 1
  validation_interval: 1
  num_epochs: 5
  optimizer:
    type: AdamW
    lr:  0.0002
  lr_scheduler:
    - type: LinearLR
      start_factor: 0.33333333
      by_epoch: False
      begin: 0
      end: 500
    - type: CosineAnnealingLR
      T_max: 10
      begin: 0
      end: 10
      by_epoch: True
      eta_min_ratio: 1e-4
    - type: CosineAnnealingMomentum
      eta_min: 0.8947
      begin: 0
      end: 2.4
      by_epoch: True
    - type: CosineAnnealingMomentum
      eta_min: 1
      begin: 2.4
      end: 10
      by_epoch: True
inference:
  num_gpus: 1
  conf_threshold: 0.3
  checkpoint: /results/train/bevfusion_model.pth
evaluate:
  num_gpus: 1
  checkpoint: /results/train/bevfusion_model.pth
        

Data Preprocessor Config

The dataset configuration (data_preprocessor) defines the data source and pre-processing hyperparameters.

Dataset Config

The dataset configuration (dataset) defines the dataset directories, annotation file and batch size for either train, val or test.

Model Config

The model configuration (model) defines the BEVFusion model structure. This model is used for training, evaluation, and inference. A detailed description is included in the table below.

Image Backbone Config

The backbone configuration (img_backbone) defines the backbone structure. A detailed description is included in the table below. Currently, BEVFusion only supports Swin-Transformers and ResNet50 image backbone.

Image Neck Config

The neck configuration (img_neck) defines the image neck structure. A detailed description is included in the table below. Currently, BEVFusion only supports GeneralizedLSSFPN image backbone.

View Transform Config

The configuration (view_transform) defines the view transform structure for camera input. A detailed description is included in the table below. Currently, BEVFusion only supports DepthLSSTransform and LSSTransform image backbone.

Lidar Backbone Config

The backbone configuration (lidar_backbone) defines the image backbone structure. A detailed description is included in the table below. Currently, BEVFusion only supports SECOND lidar backbone at the moment.

Lidar Encoder Config

The encoder configuration (pts_middle_encoder) defines the lidar encoder structure. A detailed description is included in the table below. Currently, BEVFusion only supports BEVFusionSparseEncoder structure at the moment.

Lidar Neck Config

The configuration (pts_neck) defines the lidar neck structure. A detailed description is included in the table below. Currently, BEVFusion only supports SECONDFPN structure at the moment.

Fusion Layer Config

The configuration (fusion_layer) defines the fusion layer structure. A detailed description is included in the table below. Currently, BEVFusion only supports ConvFuser structure at the moment.

BBoxHead Config

The configuration (bbox_head) defines the bbox prediction head structure. A detailed description is included in the table below. Currently, BEVFusion only supports BEVFusionHead structure at the moment.

Train Config

The train configuration defines the hyperparameters of the training process.

            

            
train:
  precision: 'fp16'
  num_gpus: 1
  checkpoint_interval: 10
  validation_interval: 10
  num_epochs: 50
  optim:
    type: "AdamW"
    lr: 0.0001
    weight_decay: 0.05
        

Optimizer config

The optim parameter defines the config for the optimizer in training, including the learning rate, learning scheduler, and weight decay.

Evaluation Config

The evaluate parameter defines the hyperparameters of the evaluation process.

            

            
evaluate:
  checkpoint: /path/to/model.pth
  num_gpus: 1
        

Inference Config

The inference parameter defines the hyperparameters of the inference process.

            

            
inference:
  checkpoint: /path/to/model.pth
  num_gpus: 1
        

To train a BEVFusion model, use this command:

            

            
tao model bevfusion train [-h] -e <experiment_spec>
                          [-r <results_dir>]
        

Required Arguments

• -e, --experiment_spec: The experiment specification file to set up the training experiment

Optional Arguments

• -r, --results_dir: The path to the folder where the experiment outputs should be written. If this argument is not specified, the results_dir from the spec file is used.

• --gpus: The number of GPUs used to run training

• --num_nodes: The number of nodes used to run training. If this value is larger than 1, distributed multi-node training is enabled.

• -h, --help: Show this help message and exit.

Sample Usage

Here’s an example of the train command:

            

            
tao bevfusion model train -e /path/to/spec.yaml
        

To run evaluation with a BEVFusion model, use this command:

            

            
tao model bevfusion evaluate [-h] -e <experiment_spec>
                             [-r <results_dir>]
        

Required Arguments

• -e, --experiment_spec: The experiment spec file to set up the evaluation experiment

Optional Arguments

• -r, --results_dir: The directory where the evaluation result is stored

Sample Usage

Here’s an example of using the evaluate command:

            

            
tao model bevfusion evaluate -e /path/to/spec.yaml -r /path/to/results/ evaluate.checkpoint=/path/to/model.pth
        

Use the following command to run inference on BEVFusion with .pth:

            

            
tao model bevfusion inference [-h] -e <experiment spec file>
                              [-r <results_dir>]
        

Required Arguments

• -e, --experiment_spec: The experiment spec file to set up the inference experiment

Optional Arguments

• -r, --results_dir: The directory where the inference result is stored

Sample Usage

Here’s an example of using the inference command:

            

            
tao model bevfusion inference -e /path/to/spec.yaml -r /path/to/results/ inference.checkpoint=/path/to/model.pth