> This page is for version 9.2 (default).
> For other versions, use one of these documentation indexes:
> - 9.2 (default): https://docs.nvidia.com/deepstream/9.2/llms.txt
> - Archives: https://docs.nvidia.com/deepstream/archives/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/deepstream/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/deepstream/_mcp/server.

# DeepStream 3D Action Recognition App

The `deepstream-3d-action-recognition` sample application is provided at `app/sample_apps/deepstream-3d-action-recognition` for your reference. This example demonstrates a sequence batching based 3D or 2D model inference pipeline for action recognition. The image below shows the architecture of this reference app.

![DeepStream 3D Action Recognition Application Architecture](/deepstream/_fern-img/6a228b2b880ee2fe7324da9875f1f3e7ae6b07faa53ebf72b5c6713e8ba34f52.webp)

## This 3D/2D model is pretrained by NVIDIA TAO toolkit. The 3D model has NCDHW (NCSHW) input and the 2D model has NSHW shapes.

N: Max batch size of total number of ROIs in all streams, value > 0.
C: Channel numbers, must be 3.
D/S: sequence length of consecutive frames, value > 1
H: height, value > 0
W: width, value > 0
2D S: channels x sequence_length, reshaped from \[C, D]

A custom sequence preprocessing lib: `libnvds_custom_sequence_preprocess.so` is also provided at `sources/apps/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess` to demonstrate how to implement a sequence batching and preprocessing methods with `Gst-nvdspreprocess` plugin. This custom lib normalizes each incoming ROI cropped image and accumulates the data into buffer sequence for temporal batching. When temporal batching is ready, it continues to do spatial batching on multi-ROIs and multi-streams. Finally it returns the temporal and spatial batched buffer(tensor) to `Gst-nvdspreprocess` plugin which would attach the buffer as preprocess input metadata and deliver to downstream `Gst-nvinfer` plugin to do inference.

### Getting Started

#### Prerequisites

* Go to the folder `sources/apps/sample_apps/deepstream-3d-action-recognition`.

* Search and Download 3D and 2D RGB based `tao_iva_action_recognition_pretrained` models from NGC [https://ngc.nvidia.com/catalog/models/nvidia:tao:actionrecognitionnet](https://ngc.nvidia.com/catalog/models/nvidia:tao:actionrecognitionnet) (Version 5):

  * `resnet18_3d_rgb_hmdb5_32`
  * `resnet18_2d_rgb_hmdb5_32`

* These Models support following classes : `push`; `fall_floor`; `walk`; `run`; `ride_bike`.

* Update source streams `uri-list` in action recognition config file: `deepstream_action_recognition_config.txt`.

```
uri-list=file:///path/to/sample_action1.mov;file:///path/to/sample_action2.mov;file:///path/to/sample_action3.mov;file:///path/to/sample_action4.mov;
```

* Export DISPLAY environment to correct display. e.g. `export DISPLAY=:0.0`.

#### Run 3D Action Recognition Examples

* Make sure 3D preprocess config and 3D inference config are enabled in `deepstream_action_recognition_config.txt`.

```
# Enable 3D preprocess and inference
preprocess-config=config_preprocess_3d_custom.txt
infer-config=config_infer_primary_3d_action.txt
```

* Run the following command:

```
$ deepstream-3d-action-recognition -c deepstream_action_recognition_config.txt
```

* Run with DS-Triton, update application config file `deepstream_triton_action_recognition_config.txt`.

```
preprocess-config=config_preprocess_3d_custom.txt
triton-infer-config=config_triton_infer_primary_3d_action.txt
```

* Run 3D test with DS-Triton:

  ```text
  $ ./deepstream-3d-action-recognition -c deepstream_triton_action_recognition_config.txt
  ```

  Check `sources/TritonOnnxYolo/README` for more details how to switch action recognition DS-Triton tests between CAPI and gRPC.

#### Run 2D Action Recognition Examples

* Make sure 2D preprocess config and 2D inference config are enabled in `deepstream_action_recognition_config.txt`.

  ```text
  # Enable 2D preprocess and inference
  preprocess-config=config_preprocess_2d_custom.txt
  infer-config=config_infer_primary_2d_action.txt
  ```

* Run the following command:

```
$ deepstream-3d-action-recognition -c deepstream_action_recognition_config.txt
```

* Run with DS-Triton, update application config file `deepstream_triton_action_recognition_config.txt`.

  ```text
  preprocess-config=config_preprocess_2d_custom.txt
  triton-infer-config=config_triton_infer_primary_2d_action.txt
  ```

* Run 2D test with DS-Triton:

  ```text
  $ ./deepstream-3d-action-recognition -c deepstream_triton_action_recognition_config.txt
  ```

  Check `sources/TritonOnnxYolo/README` for more details how to switch action recognition DS-Triton tests between CAPI and gRPC.

### DeepStream 3D Action Recognition App Configuration Specifications

deepstream-3d-action-recognition `[action-recognition]` group settings

#### deepstream-3d-action-recognition `[action-recognition]` group settings

The table below demonstrates the group settings for `deepstream_action_recognition_config.txt` as an example.

**3D action recognition Supported Settings**

| Property            | Meaning                                              | Type and Range                             | Example                                                                |
| ------------------- | ---------------------------------------------------- | ------------------------------------------ | ---------------------------------------------------------------------- |
| uri-list            | source video file or stream list                     | Semicolon delimited string list            | file:///path/to/sample_action1.mp4;file:///path/to/sample_action2.mp4; |
| display-sync        | Indicate display synchronization on timestamp or not | Boolean                                    | display-sync=1                                                         |
| preprocess-config   | Gst-nvdspreprocess plugin config file path           | String                                     | preprocess-config=config_preprocess_3d_custom.txt                      |
| infer-config        | Gst-nvinfer plugin config file path                  | String                                     | infer-config=config_infer_primary_2d_action.txt                        |
| muxer-height        | Gst-nvstreammux height                               | Unsigned Integer                           | muxer-height=720                                                       |
| muxer-width         | Gst-nvstreammux width                                | Unsigned Integer                           | muxer-width=1280                                                       |
| muxer-batch-timeout | Gst-nvstreammux batched push timeout in usec         | Unsigned Integer                           | muxer-batch-timeout=40000                                              |
| tiler-height        | Gst-nvmultistreamtiler height                        | Unsigned Integer                           | tiler-height=720                                                       |
| tiler-width         | Gst-nvmultistreamtiler width                         | Unsigned Integer                           | tiler-width=1280                                                       |
| debug               | Log print debug level                                | Integer, 0: disabled. 1: debug. 2: verbose | debug=0                                                                |
| enable-fps          | Indicate whether print fps on screen                 | Boolean                                    | enable-fps=1                                                           |

Custom sequence preprocess lib user settings `[user-configs]` for `gst-nvdspreprocess`

#### Custom sequence preprocess lib user settings `[user-configs]` for `gst-nvdspreprocess`

The table below demonstrates the `config_preprocess_3d_custom.txt` setting of `libnvds_custom_sequence_preprocess.so` as an example.

**user-configs properties for custom sequence preprocess**

| Property              | Meaning                                               | Type and Range                  | Example                                                    |
| --------------------- | ----------------------------------------------------- | ------------------------------- | ---------------------------------------------------------- |
| channel-scale-factors | scale factor list for each channel                    | Semicolon delimited float array | channel-scale-factors= 0.007843137;0.007843137;0.007843137 |
| channel-mean-offsets  | data mean offsets for each channel                    | Semicolon delimited float array | channel-mean-offsets=127.5;127.5;127.5                     |
| stride                | sequence sliding stride for each batched sequence     | Unsigned Integer, value >= 1    | stride=1                                                   |
| subsample             | Subsample rates for inference images in each sequence | Unsigned Integer, value >= 0    | subsample=0                                                |

#### Custom `lib` and `gst-nvdspreprocess` Settings for Action Recognition

* You'll need to set input order as CUSTOM `network-input-order=2` for this custom sequence preprocess lib.
* 3D models NCDHW(NCSHW) require network-input-shape with 5-dimension shape. For example:

```
   network-input-shape= 4;3;32;224;224

It means max_batch_size: 4, channels 3, sequence_len: 32, height 224, width 224.
```

* 2D models NSHW require network-input-shape with 4-dimension shape. For example:

```
   network-input-shape= 4;96;224;224

It means max_batch_size: 4, channels 3, sequence_len: 32, height 224, width 224. where 96 = channels x sequence_len.
```

* Assume incoming frame numbers are 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15... When `subsample=1`, the preprocessing custom lib will pick up frame numbers: 1,3,5,7,9... to preprocess sequentially and  pass it onto inference as a next-step.
* Assuming same incoming frame numbers above, For example, when `subsample=0, stride=1`, the 2 consecutive sliding sequences are:

```
Batch A: [1,2,3,4,5...]
Batch B: [2,3,4,5,6...]
```

When `subsample=0, stride=2`, the 2 consecutive sliding sequences are:

```text
Batch A: [1,2,3,4,5...]
Batch B: [3,4,5,6,7...]
```

When `subsample=1, stride=2`, the subsample is performed first, and sliding sequences are on top of subsample results. The processing frame numbers after subsample are: 1,3,5,7,9,11,13,15,17,19... The consecutive sliding sequences on top of them are:

```text
Batch A: [1,3,5,7,9...]
Batch B: [5,7,9,11,13...] # 1st frame sliding from frame 1 of Batch A to frame 5
Batch C: [9,11,13,15,17...] # 1st frame sliding from frame 5 of Batch C to frame 9
```

The image below shows the frame batches with different subsample and stride settings.

![3D Action Sequence batching](/deepstream/_fern-files/nvidia-deepstream.docs.buildwithfern.com/87fc281a353a4497e1aea347c56529d8f0f019462c3c3db94c789f2f86e9bec3/assets/images/content/DS_Action_Recognition_Sequence_Batching.png)

### Build Custom sequence preprocess lib and application From Source

* Go to the folder `sources/apps/sample_apps/deepstream-3d-action-recognition`.
* Run the following commands:

```
$ make
$ make install
```

* Check the source code and comments to learn about implementation of other order formats, for example, NSCHW (NDCHW).