> This page is for version 1.0.0.
> For other versions, use one of these documentation indexes:
> - 1.1.0 (Latest) (default): https://docs.nvidia.com/nim/cosmos-embed1/1.1.0/llms.txt
> - 1.0.0: https://docs.nvidia.com/nim/cosmos-embed1/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/cosmos-embed1/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/cosmos-embed1/_mcp/server.

# Performance

## Request Type Selection

Choose the right request type for your workload:

- `request_type: "query"`: Best for the following:
      - Single items (1 items)
      - Low-latency applications requiring immediate responses
      - Interactive applications
      - Sending video data as part of the request
- `request_type: "bulk_text"`: Best for the following:
      - Large batches of text-only inputs (up to 64 strings)
      - High-throughput text processing
      - When you have many text captions to process at once
- `request_type: "bulk_video"`: Best for the following:
      - Large batches of video-only inputs (up to 64 videos)
      - Maximum throughput for video processing
      - Batch processing workflows

## Bulk Video Throughput Performance

The following benchmarks were conducted using a pipelined version with load-balanced queues, where decoding (pyNvVideoCodec) runs in parallel with CV-CUDA pre-processing and inference (TRT fp16). This setup utilizes best-effort load balancing, MPS (Multi-Process Service), and IPC (Inter-Process Communication). The processes involved are 2 downloaders, 11 decoders, 2 preprocessors, and 2 embedders.

- **A100 GPU**:
      - **Configuration**: 15-second, 1080p video in `bulk_video` mode with a request size of 64.
      - **Throughput**: 16 videos/second
- **H100 GPU**:
      - **Configuration**: 15-second, 1080p video in `bulk_video` mode with a request size of 64.
      - **Throughput**: 28.57 videos/second

## Additional GPUs

- Expect proportional scaling based on decode and compute capacity across supported GPUs (e.g. L40s, L4, H20, L20). For GPUs with less memory, reduce batch/request sizes to avoid memory pressure.

## Latency Performance

The following table shows representative single-embedding latency (seconds) per `request_type=query` for text and video, measured on MSRVTT clips.

| GPU                   | Video (s) | Text (s) |
| --------------------- | --------- | -------- |
| NVIDIA H100 80GB HBM3 | 0.0431    | 0.0058   |
| NVIDIA H100 NVL       | 0.0433    | 0.0055   |
| NVIDIA H100 PCIe      | 0.0446    | 0.0058   |
| NVIDIA L40S           | 0.0511    | 0.0061   |
| NVIDIA A100 80GB PCIe | 0.0563    | 0.0064   |
| NVIDIA A100-SXM4-80GB | 0.0613    | 0.0069   |
| NVIDIA A100-SXM4-40GB | 0.0672    | 0.0074   |
| NVIDIA A100-PCIE-40GB | 0.0683    | 0.0093   |
| NVIDIA H20            | 0.0705    | 0.0077   |
| NVIDIA L4             | 0.1096    | 0.0085   |
| NVIDIA L20            | 0.0738    | 0.0064   |
| NVIDIA RTX 6000 Ada   | 0.0527    | 0.0067   |

## Resource Management

**Memory Optimization**

- Monitor GPU memory usage with larger batch sizes.
- The service automatically scales workers based on available resources.
- Use `CUDA_VISIBLE_DEVICES` to control GPU allocation.

**Concurrency**

- The service handles multiple concurrent requests efficiently.
- For sustained high throughput, maintain a consistent request rate.
- Consider connection pooling for client applications.

**Error Handling**

- The service includes automatic retry logic for transient failures.
- Monitor the `X-Restart-Count` header for service health.
- Implement exponential backoff in client code for failed requests.