Performance

Request Type Selection

Choose the right request type for your workload:

• request_type: "query": Best for the following:
  • Single items (1 items)
  • Low-latency applications requiring immediate responses
  • Interactive applications
  • Sending video data as part of the request
• request_type: "bulk_text": Best for the following:
  • Large batches of text-only inputs (up to 64 strings)
  • High-throughput text processing
  • When you have many text captions to process at once
• request_type: "bulk_video": Best for the following:
  • Large batches of video-only inputs (up to 64 videos)
  • Maximum throughput for video processing
  • Batch processing workflows

Bulk Video Throughput Performance

The following benchmarks were conducted using a pipelined version with load-balanced queues, where decoding (pyNvVideoCodec) runs in parallel with CV-CUDA pre-processing and inference (TRT fp16). This setup utilizes best-effort load balancing, MPS (Multi-Process Service), and IPC (Inter-Process Communication). The processes involved are 2 downloaders, 11 decoders, 2 preprocessors, and 2 embedders.

• A100 GPU:
  • Configuration: 15-second, 1080p video in bulk_video mode with a request size of 64.
  • Throughput: 16 videos/second
• H100 GPU:
  • Configuration: 15-second, 1080p video in bulk_video mode with a request size of 64.
  • Throughput: 28.57 videos/second

Additional GPUs

• Expect proportional scaling based on decode and compute capacity across supported GPUs (e.g. L40s, L4, H20, L20). For GPUs with less memory, reduce batch/request sizes to avoid memory pressure.

Latency Performance

The following table shows representative single-embedding latency (seconds) per request_type=query for text and video, measured on MSRVTT clips.

Resource Management

Memory Optimization

• Monitor GPU memory usage with larger batch sizes.
• The service automatically scales workers based on available resources.
• Use CUDA_VISIBLE_DEVICES to control GPU allocation.

Concurrency

• The service handles multiple concurrent requests efficiently.
• For sustained high throughput, maintain a consistent request rate.
• Consider connection pooling for client applications.

Error Handling

• The service includes automatic retry logic for transient failures.
• Monitor the X-Restart-Count header for service health.
• Implement exponential backoff in client code for failed requests.