Zero Shot Detection Workflow
Overview
Jetson Platform Services provide several reference workflows that illustrate how various foundational and AI services can be integrated to create end to end applications. This section described the zero shot detection workflow, that integrates the corresponding AI service with associated pieces including VST, Redis and monitoring. The resulting application provides a web based UI with which a user can interact with the system, and leverages VST for stream input and management. Users can use this as a reference to carve out relevant pieces for creating their own applications utilizing the zero shot detection AI service.
To get the docker compose and config files for this workflow, download the Jetson Platform Services resources bundle from NGC or SDK Manager.
To launch this workflow, first follow steps on the Zero Shot Detection with Jetson Platform Services page to ensure the Zero Shot Detection service is functional on your system.
After verifying the Zero Shot Detection service works as expected, bring it down with sudo docker compose down from the same folder it was launched.
Getting Started
Ensure the necessary foundation services are running
sudo systemctl restart jetson-monitoring
sudo systemctl restart jetson-sys-monitoring
sudo systemctl restart jetson-gpu-monitoring
Verify all the lines (for monitoring services) in the platform ingress config file are uncommented in the following file:
/opt/nvidia/jetson/services/ingress/config/platform-nginx.conf
This will ensure that monitoring dashboards are accessible through the API Gateway.
sudo systemctl restart jetson-ingress
sudo systemctl restart jetson-redis
sudo systemctl restart jetson-vst
Note that we will also start jetson-vst as this will be the source of our live streams in this workflow example.
Launch the workflow
cd ~/zero_shot_detection/example_2
sudo docker compose up
Once launched, SDR will automatically add an available video stream from VST to the Zero Shot Detection service. The detection classes can be controlled through the REST API and the outputs viewable on RTSP and Redis as shown in the Zero Shot Detection with Jetson Platform Services page. The following sections will explain in more depth how the Zero Shot Detection service can integrate with the rest of Jetson Platform Services.
Video Ingestion with SDR & VST
It is typical for generative AI video applications to be developed accepting file input for video source. In production, these applications need to interact with video streams from the real world, including from cameras and network streams. The Video Storage Toolkit microservice provides an out of the box mechanism to ingest video, including camera discovery, camera reconnection, monitoring and camera events notification. In the process, VST provides a proxy RTSP link for camera video sources that services can query for based on VST APIs.
VST supports the ONVIF protocol for discovering cameras. These cameras could be connected directly to the Jetson system through a dedicated power over Ethernet (PoE) switch; or to the network that the device is connected to. Another option is for users to manually add devices based on its IP address.
Given that users can add/remove cameras dynamically, applications need to be notified of camera changes so that they can incorporate addition or deletion of the new streams. Instead of applications having to poll VST APIs periodically to register the changes, camera event information is sent over the Redis message bus, that other services can subscribe to. Camera event information contains name, resolution and RTSP URL endpoint information that can be used by the application to start processing (or omit) video streams from the camera.
Along with the Zero Shot Detection service, docker composes file in this workflow also launches the Sensor Distribution and Routing (SDR) service. In the Zero Shot Detection with Jetson Platform Services page, example curl commands were shown to manually add and remove streams from the Zero Shot Detection service. By launching SDR alongside our AI service, SDR will listen to the VST stream add/remove events on Redis and then call the add/remove APIs of the Zero Shot Detection service to control the input stream. This allows camera changes in VST to dynamically update the stream input to the Zero Shot Detection service without user intervention.
External API access through Ingress, Firewall and IoTGateway
AI services using REST APIs for configuration and output of results can leverage the combination of Ingress, Firewall and IoTGateway modules offered as part of Jetson Platform services to enable secure and remote access of the APIs. The Zero Shot Detection example for instance uses REST APIs to control stream input and detection classes.
With APIs at the core Jetson Platform Services, Ingress provides a centralized means of accessing them by providing a common endpoint with designated routes to each microservice. For example, the Zero Shot Detection API can be accessed in two ways:
Name |
Local URI |
API Gateway URI |
|---|---|---|
REST API |
In the API Gateway URI, 30080 is the network port where Ingress is running, and zero_shot_detection is the route registered by the Zero Shot detection service through the Ingress configuration file.
The route to expose the Zero Shot detection REST API at set in the
zero_shot_detection-nginx.conf
location /zero_shot_detection/ {
rewrite ^/zero_shot_detection/?(.*)$ /$1 break;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
access_log /var/log/nginx/access.log timed_combined;
proxy_pass http://localhost:5010;
}
When following the setup on the Zero Shot Detection with Jetson Platform Services page, the zero_shot_detection-nginx.conf is copied to the ingress configuration folder and the jetson-ingress service is then restarted for the route to take effect.
Developers can similarly register their own AI application microservice at system deployment time by providing the ‘route’ along with the internal port in which their application is serving its APIs. Upon receiving incoming requests for that path, the ingress microservice automatically forwards the request to the right microservice.
A standard pattern used in securing APIs is to to leverage Ingress in combination with the Firewall platform service, so that only the ingress port is externally accessible, and all the internal microservices (such as Redis) and internal ports being served by the various microservices are encapsulated from the outside world.
Finally, the IOTGateway enables AI service APIs to be accessible remotely by having requests from clients routed through the Reference Cloud software (Reference Cloud for Jetson devices running Jetson Platform Services) offered as part of Jetson Platform Services. Rich clients such as a mobile app can access this functionality from anywhere, as is the norm with product grade systems. The cloud ensures that this interaction is secure by providing user authentication and authorization as functionality out of the box.
Metadata Output with Redis
Once SDR has added a stream to the Zero Shot Detection service and the detection classes have been set, the Zero Shot Detection model will begin inferencing and generating detection metadata.
The redis-cli can be used from the command line to view the metadata output.
redis-cli XREAD COUNT 1 BLOCK 5000 STREAMS owl $
The detection metadata is output in Metropolis Minimal Schema:
{
"metadata": {
"version": "4.0",
"id": 154,
"@timestamp": "2024-05-03T17:16:47.885Z",
"sensorId": 1,
"objects": [
"46|816.09375|497.34375|1107.1875|739.6875|a giraffe"
]
}
}
For each object detected, it will have six properties, separated by the ‘|’ character. The properties are in the following order:
Object ID
Bounding box top Left X coordinate
Bounding box top left Y coordinate
Bounding box bottom right X coordinate
Bounding box bottom right Y coordinate
Detection class
From Python, the metadata can be read using the redis-py package and then used to implement a custom analytic microservice.
Monitoring
Jetson Platform Services provides monitoring as a ready to use platform service, enabling developers to track system utilization and performance KPIs. Given generative AI applications push the system capabilities both in terms of GPU utilization and memory availability, the Prometheus based monitoring dashboards provide a ready means to observe utilization of these metrics as applications are run. These metrics are updated live, presenting to users a real time view of the utilization as the application executes and starts processing different inputs.
View the Usage & Configuration page for more details on the monitoring service and capabilities.
Secure Storage
AI applications use and generate various data, including model, inference output etc. Jetson Platform Services offers encryption of storage used by microservices that can be leveraged by AI applications so that their data (model, weights, input, output) is safe at rest. The storage platform service describes how to enable encryption and also how microservices can request storage quotas from attached external storage such as hard disk.
Further Customization & Integration
To facilitate integration of Generative AI applications (typically written in Python) into systems using Jetson Platform Services, we highlight a collection of Python modules that developers can leverage. The source code for the Zero Shot Detection service available on GitHub illustrates the use of these modules to achieve this integration.
The list of these Python modules include:
jetson-containers : provides containers, models, vectorDB, and other building blocks for generative AI applications on Jetson
jetson-utils : NVIDIA provided python module for hardware accelerated image and multimedia operations relating supporting RTSP stream input/output, video decode, and efficient video overlay creation
jetson-platform-services : Open Source GitHub repository with reference AI services and workflows.
moj-utils: New NVIDIA provided python module for integration with rest of Jetson Platform Services
The mmj_utils module can be installed from GitHub:
pip install git+https://github.com/NVIDIA-AI-IOT/mmj_utils
The module provides utilities to integrate Python applications with Jetson Platform Services including:
Support for Video Storage Toolkit (VST): VST APIs enable among other functionality, stream discovery and introspection. The the vst submodule provides a class abstraction to invoke the relevant APIs to discover, add, remove streams among other operations
Metadata creation: The Zero Shot Detection services uses Metropolis schema as the standardized spec to capture model output; the schema_gen submodule provides a ready JSON schema generator based on object detection output from models like NanoOWL.
Overlay Generation: the DetectionOverlayCUDA submodule enables hardware accelerated bounding box and text overlay generation to facilitate visualization of model output in live video