> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/vss/dev/system-components/analytics-microservices/alerts-microservice/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/vss/_mcp/server. # Alerts Microservice The Alerts Microservice is a modular, agent-native, configuration-driven service for alert-related workflows, including: * **Alert verification, contextualization, and classification**: Process curated (short) clips from the VSS video pipeline (for example, RTVI CV and video analytics) through a VLM to confirm, enrich, or classify upstream alerts. * **Real-time alerts**: Register video streams for continuous alert generation via periodic VLM inference through the RTVI VLM microservice. * **Alert-on-demand**: Process short clips on demand through a VLM using preconfigured alert settings. ## Key Features * **Event-Driven Processing**: Consumes alerts from Kafka in `nv.Incident` and `nv.Behavior` message formats in both Protobuf and JSON * **HTTP API Support**: RESTful endpoints for various functionalities (`POST /api/v1/realtime`, `POST /api/v1/verification/config`, `POST /api/v1/verification/ondemand`, `POST /api/v1/incidents`, `POST /api/v1/alerts`). * **Agent Skills**: VSS Agent skills for full alert lifecycle management, including microservice deployment, alert creation, retrieval, and configuration. * **Diverse VLM Sources**: RTVI VLM microservice (default), NVIDIA VLM NIM (for example, Cosmos Reason, Qwen VL), remote model endpoints (for example, GPT from OpenAI). * **Automatic Video Resolution**: Retrieves video segments from VIOS based on alert timestamps and sensor information. * **Configurable Prompts**: Alert-type-specific prompts with template placeholders for dynamic content injection from alert payloads. * **Configurable VLM Parameters**: Tune video preprocessing and inference settings to match deployment requirements and VLM backend capabilities. * **VLM Warmup**: Automatically sends multiple warmup inferences (default 3) to the VLM backend at startup, reducing initial request latency. Warmup polls the VLM NIM health endpoint until ready, then sends configurable rounds of test video inference before alert processing begins. Controlled via the `VLM_WARMUP_ENABLED` environment variable (enabled by default). * **Structured VLM Responses**: Standardized response format with verdict (confirmed/rejected/unverified), reasoning trace, and HTTP-like status codes. * **Custom VLM Response Processing**: User-defined hooks for bespoke VLM response parsing. * **Flexible Output Options**: Persist results to Elasticsearch for analytics, with optional Kafka publishing. * **Custom Category Names**: Configure user-friendly display names for alert categories in output (for example, `"collision"` displays as `"Vehicle Collision"` in Elasticsearch and UI). * **Alert Enrichment**: Optionally generate detailed event descriptions after successful verification, providing additional context such as objects involved, sequence of events, and environmental factors. ## Alert Verification Alert verification uses a VLM to confirm, enrich, and classify alerts generated upstream in the VSS pipeline. Each alert type is handled through a designated VLM prompt. The service supports alerts in [NvSchema](/system-components/middleware/nvschema) format (`nv.Incident` and `nv.Behavior`) via Kafka or HTTP API. Upon receiving an alert, it identifies the alert or incident type from the `category` attribute, retrieves the video URL for the alert time window from VIOS based on sensor ID and timestamps, and passes that URL to a VLM-based verification backend (RTVI VLM by default; NVIDIA Cosmos Reason NIM is also supported) through an OpenAI-compatible API. The VLM downloads the video and analyzes it against configurable prompts. Upstream alerts are typically generated by video analytics based on events detected within a camera's field of view. The alert verification workflow uses a 6-step process: 1. **Alert Config Creation**: Alert rules are defined using the `POST /api/v1/verification/config` API call, which specifies the alert category, VLM prompt, and VLM invocation parameters. Alternatively, rules can be defined in a config file at deployment time. 2. **Alert Ingestion**: Alerts are received via HTTP API or Kafka topics. The system validates the message schema, applies normalization, and queues alerts for processing. 3. **Video URL Retrieval**: The service extracts sensor ID, start time, and end time from the alert, then retrieves the corresponding video URL from VIOS via the REST API. 4. **VLM Analysis**: The video URL and a dynamically constructed prompt (based on alert type and context) are sent to the VLM backend. The VLM downloads the video, analyzes it, and returns a verdict with reasoning. 5. **Result Publishing**: Verified alerts are persisted to Elasticsearch and optionally published to Kafka (enabled via configuration). Results include the original alert fields plus verification metadata. 6. **Alert Retrieval**: Verified alerts are retrieved using the [video analytics microservice API](/system-components/middleware/api-gateway-mcp/video-analytics-api/video-analytics-api-reference) call to `GET /alerts/`. For a complete description of APIs, see the [API Reference](/appendix/api-reference). ## Real-Time Alerts The Alerts Microservice enables users to create real-time alert rules, which are registered with the RTVI VLM microservice for continuous processing of input streams using designated prompts. Users configure the prompt along with invocation parameters such as chunk duration, frame resolution, and frame rate. Generated alerts are published to a configurable Kafka topic and follow the `nv.Incident` structure defined in NvSchema. Real-time alerts follow the process below: 1. **Alert Config Creation**: Alert rules are defined using the `POST /api/v1/realtime` API. 2. **Alert Publish and Subscribe**: Generated alerts are published to a configurable Kafka topic that downstream consumers can subscribe to. 3. **Alert Storage**: The system can be configured to persist alerts from Kafka to a database such as Elasticsearch. Refer to the alerts developer profile for an example. 4. **Alert Retrieval**: Stored incidents are queried with `GET /api/v1/realtime/incidents`. Because RT-VLM can emit one incident per video chunk while a single event stays visible, the same endpoint can group consecutive confirmed chunks into one event on request with `consolidate=true`. This is an API-only option; see [Realtime incident consolidation](/system-components/analytics-microservices/alerts-microservice/alert-verification-api#realtime-incident-consolidation). Always-on alerts are a refinement of the real-time alerts capability: users define VLM prompts and configuration in a file, and the rule is applied automatically to every newly added stream. This saves users from invoking the alert config creation API explicitly. ## Alert-on-Demand Third-party CV applications that generate alerts for verification can use the alert-on-demand feature to run VLM verification on an associated video snippet stored in object storage. Alert-on-demand uses the process below: 1. **Alert creation and media storage**: The CV application identifies an alert condition and creates a corresponding video snippet, which it stores in object storage (such as MinIO). 2. **Alert delivery**: The application delivers the alert to the Alerts Microservice through the `POST /api/v1/verification/ondemand` API; the alert is captured using the NvSchema `nv.Incident` payload schema. 3. **Alert processing and storage**: The Alerts Microservice downloads the video (which may be remote), encodes it in base64 format, and submits it to the underlying VLM along with the prompt and VLM invocation parameters; the returned VLM response is stored in Elasticsearch for retrieval through APIs. ## Setup The Alerts Microservice supports the following deployment options: * Docker Compose deployment * Kubernetes deployment > **Note** > > For detailed deployment instructions, refer to the Docker Compose Deployment and Kubernetes Deployment sections. **Prerequisites** Before deploying the Alerts Microservice, ensure the following requirements are met: * A valid NGC API key is required for accessing NVIDIA services. * A VLM backend (e.g., NVIDIA Cosmos3 Nano Reasoner NIM) must be deployed and accessible. * VIOS service must be available for video retrieval. * Kafka is required if using event-driven ingestion. * Elasticsearch (optional) for result persistence and analytics. ## Configuring Prompts Prompts control how the VLM analyzes video content for each alert type. They are seeded from a JSON file (`alert_type_config.json`) at startup into the alert-config store, which is backed by Elasticsearch. Each verification reads the store, so a record changed there takes effect on the next alert. > **Note** > > Editing `alert_type_config.json` requires a container restart, because the file is only read at startup. Changing a prompt without a restart is done through `PUT /api/v1/verification/config/{alert_type}`, which writes to the store the pipeline reads. ### Prompt Structure Each alert type requires a configuration with the following components: * **user** (mandatory): The primary instruction describing how to analyze the video and what decision to make. * **system** (optional): Contextual setup that shapes the VLM's behavior, tone, and domain knowledge. When an alert type has none, the service default (`prompt.default_system_prompt`) is used for the VLM call. * **enrichment** (optional): A prompt to generate detailed event descriptions after successful verification. When defined, a second VLM call provides additional context such as objects involved, sequence of events, and environmental factors. * **output_category** (optional): A user-friendly display name for the alert category in output. When specified, this name appears in Elasticsearch and downstream systems instead of the internal `alert_type` value (e.g., `"collision"` displays as `"Vehicle Collision"`). * **vlm_params** (optional): Per-alert-type VLM parameter overrides (for example `model`, `max_tokens`, `temperature`, `num_frames`). Specified fields override the global `vlm` config; unspecified fields fall back to the global defaults. Unknown fields raise a startup validation error so configuration typos fail fast. Records are keyed by alert type and selected on that key during processing. Prompts and `vlm_params` may both be updated at runtime via `PUT /api/v1/verification/config/{alert_type}` (see the [Alert Verification API](/system-components/analytics-microservices/alerts-microservice/alert-verification-api)) without restarting the container. At request time the service merges all sources with the following precedence: 1. Runtime overrides set via the API and persisted in the alert-config store as `alert_config:{alert_type}` — applied without restart. 2. Static `vlm_params` from `alert_type_config.json` — requires container restart to take effect. 3. Global `vlm` config in `config.yaml` — requires container restart to take effect. ### Prompt Templating Prompts support dynamic placeholders that are substituted with values from the alert payload at runtime: * Placeholders use `{field.path}` syntax with dot notation for nested fields. * Examples: `{info.primaryObjectId}`, `{place.name}`, `{objectIds}` * Missing fields render as `` for debugging. * Array values are automatically joined as comma-separated strings. ### Alert Type Matching The `category` field in the incoming alert is matched against the `alert_type` in the prompt configuration. When a match is found, the corresponding prompts are selected and rendered with alert context. ### VLM Response Format The VLM must respond using the following format for proper parsing: ```text reasoning trace here verdict (A/B or true/false) ``` The answer section should contain a clear verdict that the system can map to confirmed/rejected status. ### Example Configuration Below is an example prompt configuration from `alert_type_config.json`: ```json { "version": "1.0", "alerts": [ { "alert_type": "collision", "output_category": "Vehicle Collision", "prompts": { "system": "You are an expert AI assistant for video analysis. Your task is to determine whether a surveillance video depicts a **collision event** or **no collision**, based on the definitions below...", "user": "Based on the video, which category best describes what occurred at {place.name}:\n(A) Collision (physical contact or impact detected)\n(B) No collision (no contact or impact)...", "enrichment": "Provide a detailed description of this collision event. Include: 1) Objects/vehicles involved and their identifiers 2) Sequence of events leading to the collision 3) Estimated speeds and trajectories 4) Visible damage or safety concerns 5) Environmental factors such as lighting, weather, or obstacles" }, "vlm_params": { "max_tokens": 2048, "temperature": 0.4, "num_frames": 10 } }, { "alert_type": "Stop Anomaly Module", "output_category": "Abnormal Vehicle Stop", "prompts": { "system": "You are an expert AI assistant for video analysis. Your task is to determine whether a surveillance video depicts **normal stopping behavior** or a **stop anomaly**, based on the strict definitions below...", "user": "Based on the video, which category best describes the observed behavior at {place.name} (if available):\n(A) Stop anomaly (unexpected or unsafe stop)\n(B) Normal stop (safe and expected behavior)..." } } ] } ``` In this example: * The `collision` alert type uses `output_category` to display as `"Vehicle Collision"` in output, and includes an `enrichment` prompt for detailed event descriptions. * The `Stop Anomaly Module` uses `output_category` but no enrichment prompt (enrichment is optional per alert type). * `vlm_params` can be added to any alert type to override the global VLM defaults; `collision` shows it as an illustration, but each alert type may define its own subset of overrides independently. > **Note** > > For prompts that ask for alert enhancement, such as an `enrichment` prompt or structured output read by a [pluggable response parser](#pluggable-response-parser), set `vlm_params.max_tokens` large enough for the expected length of the VLM response. A response that reaches the limit is cut off and may fail parsing. ## Pluggable Response Parser By default, the Alerts Microservice extracts a binary YES/NO verdict from the VLM response, which is appropriate for alert verification but limiting for other use cases such as classification, analytics, enrichment, or counting. A pluggable response parser lets you replace the built-in verification parsing with a custom class that returns arbitrary structured data. ### When to Use * **Binary verification** (confirm/reject an alert): use the built-in parser. No configuration needed. * **Classification** (PPE violation type, hazard category, etc.): use a pluggable parser. * **Analytics** (object counts, zone occupancy, etc.): use a pluggable parser. * **Enrichment** (scene description, recommended action, etc.): use a pluggable parser. ### Parser Contract A parser is any Python class that exposes a `parse(self, raw_response: str) -> dict` method. No imports from Alert Bridge are required. ```python class MyParser: def parse(self, raw_response: str) -> dict: """Receive raw VLM response string, return a flat key-value dict.""" ... ``` Requirements: * The class must be instantiable with no arguments (`def __init__(self) -> None`). Classes with required `__init__` parameters or abstract base classes fail startup validation. * Implementations **must be thread-safe**: the parser instance is shared across all worker threads (up to `alert_agent.num_workers`) and the async dispatcher. Do not mutate instance state inside `parse()`; treat any attribute set in `__init__` as read-only. Read-only caches (compiled regexes, LRU caches on pure functions) are fine. * Return a flat dictionary. Values may be primitives (string, int, bool, float) or stringified JSON for nested structures. The merge helper coerces non-string values to strings on the wire. ### Configuration Set `vlm.response_parser` in `config.yml` to the dotted Python path of the parser class: ```yaml vlm: response_parser: "parsers.my_classifier.MyClassifier" ``` Alert Bridge imports the module and instantiates the class at startup. Validation errors (missing module, missing class, no `parse` method, required `__init__` parameters) abort startup with a descriptive error message. ### Example Parser A PPE violation classifier that strips markdown code fences before parsing JSON: ````python import json import re _RE_MD_FENCE = re.compile(r'^```(?:\w+)?\s*\n(.*?)```\s*$', re.DOTALL) class PPEClassifier: def parse(self, raw_response: str) -> dict: text = raw_response.strip() m = _RE_MD_FENCE.match(text) if m: text = m.group(1).strip() data = json.loads(text) return { "classification": data.get("label", "unknown"), "severity": data.get("severity", "unknown"), "confidence": data.get("confidence", 0), "reasoning": data.get("reasoning", ""), } ```` ### Output Schema Parser output is JSON-serialized into `info["vlm_response"]`. The outer `info` shape (transport metadata such as `videoSource`, `verificationResponseCode`, `sensorId`) is identical to verification mode, so existing Elasticsearch mappings, Kibana dashboards, and downstream consumers require no changes — only an additive `vlm_response` field in the `mdx-vlm-incidents` index. Successful pluggable-parser response: ```json { "info": { "sensorId": "warehouse-cam-03", "category": "ppe-violation", "verdict": "", "vlm_response": "{\"classification\": \"no-spotter\", \"severity\": \"high\", \"confidence\": \"0.95\", \"reasoning\": \"...\"}", "videoSource": "rtsp://cam/stream", "verificationResponseCode": "200", "verificationResponseStatus": "OK" } } ``` The verification path (`info["reasoning"]`) and the pluggable-parser path (`info["vlm_response"]`) are disjoint — a given message contains exactly one of the two body fields, never both. ### Error Handling If the parser raises an exception or returns a non-dict value, Alert Bridge emits a distinct error event: | Field | Value on parser failure | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `verificationResponseCode` | `"500"` | | `verificationResponseStatus` | `"Pluggable parser failed: {ErrorType}: {message}"` | | `verdict` | `"verification-failed"` | | `errorSource` | `"pluggable_parser"` | | `vlm_response` | Not emitted (no valid parser output to serialize). The raw VLM response is logged at WARN level for operator inspection. | The `"Pluggable parser failed: ..."` status prefix lets operators distinguish parser crashes from upstream VLM service failures (`"VLM service internal error"`, `"VLM response validation failed"`). Parser-error events are routed through the error publish path; they never land on the success topic. ### Integration Steps (End-to-End) The following walkthrough takes a PPE-violation classifier from zero to a working pipeline. Adapt the field names to your own use case. **Step 1 — Design the VLM output JSON shape** Decide what fields you want the VLM to produce. For a PPE classifier: ```text { "label": "no-spotter", // one of [no-helmet, no-vest, no-spotter, compliant] "severity": "high", // one of [low, medium, high] "confidence": 0.95, // float between 0 and 1 "reasoning": "..." // short explanation } ``` **Step 2 — Update** `alert_type_config.json` **with a prompt that produces that shape** The prompt MUST instruct the VLM to return JSON with the exact field names your parser will read. Co-design the prompt and parser together. ```json { "version": "1.0", "alerts": [ { "alert_type": "ppe-violation", "output_category": "PPE Violation", "prompts": { "system": "You are a safety classification system. Always respond in valid JSON only, no other text.", "user": "Classify this scenario at {place.name}. Return JSON with exactly these fields:\n- label: one of [no-helmet, no-vest, no-spotter, compliant]\n- severity: one of [low, medium, high]\n- confidence: float between 0 and 1\n- reasoning: short explanation" } } ] } ``` > **Note** > > When using a pluggable parser, the prompt no longer needs the `......` Cosmos Reason format — that format is for the built-in verification parser. Tell the VLM to return plain JSON instead. VLMs commonly wrap JSON in markdown code fences (``\`\`\`json ... \`\`\``); the parser must strip them. **Step 3 — Write the parser class** (`parsers/ppe_classifier.py`) ````python import json import re _RE_MD_FENCE = re.compile(r'^```(?:\w+)?\s*\n(.*?)```\s*$', re.DOTALL) class PPEClassifier: def parse(self, raw_response: str) -> dict: text = raw_response.strip() m = _RE_MD_FENCE.match(text) if m: text = m.group(1).strip() data = json.loads(text) return { "classification": data.get("label", "unknown"), "severity": data.get("severity", "unknown"), "confidence": data.get("confidence", 0), "reasoning": data.get("reasoning", ""), } ```` The output keys (`classification`, `severity`, `confidence`, `reasoning`) are what end up serialized inside `info["vlm_response"]` — you can rename them freely; consumers only see the parser's output. **Step 4 — Mount the parser directory into the container** Place an empty `__init__.py` and your parser file under a host directory and mount it to `/app/parsers/` (see [Deployment](#deployment) below for the full layout). **Step 5 — Set** `vlm.response_parser` **in** `config.yml` ```yaml vlm: response_parser: "parsers.ppe_classifier.PPEClassifier" ``` The path is the dotted module name relative to `/app/` — `parsers.ppe_classifier` resolves to `/app/parsers/ppe_classifier.py` and `PPEClassifier` is the class inside that file. **Step 6 — Recreate the container** Run the `docker compose ... up --detach --pull always --force-recreate --build` command from the [Alert Verification deployment workflow](/vision-agent/agent-workflows/alert-verification#step-1-deploy-the-agent) again. Docker Compose recreates Alert Bridge to apply the new volume mount; a `docker compose restart` does not apply Compose configuration changes. On startup, look for the log line `Pluggable response parser active: 'parsers.ppe_classifier.PPEClassifier'`. A startup error here means the dotted path or class signature is wrong; the container will exit instead of running with a broken parser. **Step 7 — Verify with a test alert** Submit a test alert via the HTTP API with `category` matching your `alert_type`: ```bash curl -X POST http://:9080/api/v1/verification/ondemand \ -H "Content-Type: application/json" \ -d '{ "sensorId": "warehouse-cam-03", "timestamp": "2026-03-17T10:00:00Z", "end": "2026-03-17T10:00:10Z", "category": "ppe-violation", "place": {"name": "warehouse-zone-A"}, "info": { "media_urls": ["http://:30888/vst/sim/media/ppe-clip.mp4"], "media_type": "video" } }' ``` Then inspect the `mdx-vlm-incidents` Elasticsearch index — see [Consuming the Output Downstream](#consuming-the-output-downstream) below. ### Consuming the Output Downstream Pluggable parser output lands in `info["vlm_response"]` as a **JSON-encoded string** (per the NvSchema `map` contract). Downstream consumers must deserialize it before reading individual fields. **Reading from Elasticsearch** The `mdx-vlm-incidents` index stores each verified alert as a document. Query by category and decode `info.vlm_response` client-side: ```bash curl -s "http://:9200/mdx-vlm-incidents/_search?pretty" \ -H "Content-Type: application/json" -d '{ "size": 5, "query": {"term": {"info.category": "ppe-violation"}}, "_source": ["sensorId", "info.vlm_response", "info.verdict", "info.verificationResponseStatus"] }' ``` Each `info.vlm_response` value comes back as a JSON-encoded string; deserialize once to access the parser's structured fields. **Reading from Kafka** When `event_bridge.sinkType: "kafka"` is enabled, verified alerts are published to `mdx-vlm-incidents`. A minimal Python consumer: ```python import json from kafka import KafkaConsumer consumer = KafkaConsumer( "mdx-vlm-incidents", bootstrap_servers="localhost:9092", value_deserializer=lambda v: json.loads(v.decode()), ) for msg in consumer: info = msg.value.get("info", {}) parsed = json.loads(info.get("vlm_response", "{}")) # deserialize the string print(info.get("category"), parsed.get("classification"), parsed.get("severity")) ``` The double-decode (Kafka payload → outer message → inner `vlm_response` string) is intentional: the inner JSON is opaque to the alert pipeline, which lets parser authors evolve fields without breaking the schema. **Viewing in Kibana** The `vlm_response` field appears as a single string column by default. To filter or chart on individual parser fields, define a Kibana **scripted field** (or use a runtime field on Elasticsearch 7.11+) that parses the JSON: ```text def m = /\"severity\"\s*:\s*\"([^\"]+)\"/.matcher(doc['info.vlm_response.keyword'].value); return m.find() ? m.group(1) : null; ``` Repeat for each field you want to chart. For richer use cases, run a small ingest pipeline that pre-extracts parser fields into top-level Elasticsearch fields; that approach is out of scope here but follows standard ES ingest-node patterns. ### Deployment Custom parser modules are not included in the Alert Bridge image or mounted by the shipped Compose configuration. Create a host directory for them, for example: ```text /absolute/path/to/alert-parsers/ __init__.py # empty, required for Python package my_classifier.py # your parser ``` In `deploy/docker/services/alert/compose.yml`, add that directory to the `alert-bridge` service as a read-only volume: ```yaml services: alert-bridge: volumes: - /absolute/path/to/alert-parsers:/app/parsers:ro ``` To add or change parsers, place a `.py` file in the mounted host directory, update `vlm.response_parser` in `config.yml`, and recreate the container. No image rebuild is required. > **Note** > > When migrating from the legacy `vlm.custom_parser_module` registry mechanism, `vlm.response_parser` takes precedence on the default VLM path. Configuring both keys produces a startup WARN. Prefer `vlm.response_parser` for new integrations and remove `vlm.custom_parser_module` once migrated. ## Alert Messages and Schema Format The Alerts Microservice supports NvSchema message formats for both alerts and incidents: * `nv.Incident`: For incident-type events requiring verification. * `nv.Behavior`: For behavior/alert-type events requiring verification. Both Protobuf and JSON formats are supported for message serialization. > **Note** > > For the complete schema definitions of `nv.Incident` and `nv.Behavior` messages, refer to the [Protobuf Schema](/system-components/middleware/nvschema/protobuf-schema) documentation. ### Alert Ingestion Endpoints **HTTP API** * `POST /api/v1/alerts`: Submit alerts in `nv.Behavior` format (JSON or Protobuf). * `POST /api/v1/incidents`: Submit incidents in `nv.Incident` format (JSON or Protobuf). Responses: 202 Accepted (queued), 422 (validation error), 415 (unsupported media type), 500 (internal error). **Kafka Topics** (configurable) * Input: `mdx-incidents` (nv.Incident), `mdx-alerts` (nv.Behavior) * Output: `mdx-vlm-incidents`, `mdx-vlm-alerts` (when Kafka sink enabled) ### Key Fields The following table describes key fields in alert messages: | Field | Description | | ------------ | ------------------------------------------------------------------------------- | | `sensorId` | Camera or sensor identifier used for video retrieval. | | `timestamp` | Event start time in ISO 8601 format. | | `end` | Event end time in ISO 8601 format. | | `objectIds` | List of tracked object IDs involved in the event. | | `category` | Alert type used for prompt matching (e.g., "collision", "Stop Anomaly Module"). | | `place.name` | Location identifier (e.g., intersection name, zone). | | `info` | Additional metadata; extended with VLM verification results in responses. | ### Verification Response Verified alerts include the original fields plus an extended `info` section containing: | Field | Description | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `verificationResponseCode` | HTTP-like numeric code: 200 = success, 4xx = client errors, 5xx = server/VLM errors. | | `verificationResponseStatus` | Error description when non-200 status. `"OK"` on success. Pluggable-parser failures are prefixed with `"Pluggable parser failed: ..."` so they can be told apart from upstream VLM service failures. | | `verdict` | One of: `"confirmed"` (alert verified true), `"rejected"` (alert verified false), `"verification-failed"` (verification incomplete due to error). Empty string (`""`) when a pluggable response parser is configured — verdict is verification-only. | | `reasoning` | VLM reasoning trace extracted from the `` section. Emitted only on the default verification path; replaced by `vlm_response` when a pluggable parser is configured. | | `vlm_response` | JSON-serialized parser output. Emitted only when `vlm.response_parser` is configured (see [Pluggable Response Parser](#pluggable-response-parser)). Disjoint from `reasoning`: a given message contains exactly one of the two. | | `errorSource` | Structured error classification on failure. One of: `"pluggable_parser"` (parser raised or returned non-dict), `"vlm_schema"` (default-path response validation failure), `"vlm_api"` (VLM endpoint failure such as timeout, connection error, 5xx), `"media_download"` (on-demand media fetch failure). Absent on success paths. | ### Example Request ```json { "sensorId": "Lafayette_Agnew", "timestamp": "2025-09-11T00:08:27.822Z", "end": "2025-09-11T00:09:22.122Z", "objectIds": [ "958741182", "958750871", "958834290", "958730631" ], "place": { "name": "city=Montague/intersection=Lafayette_Agnew" }, "analyticsModule": { "id": "Collision Detection Module", "description": "Potential collision detected between 4 vehicles" }, "category": "collision", "isAnomaly": true, "info": { "location": "42.48837572978232,-90.73894264480816,0.0", "primaryObjectId": "958750871" } } ``` ### Example Response ```json { "sensorId": "Lafayette_Agnew", "timestamp": "2025-09-11T00:08:27.822Z", "end": "2025-09-11T00:09:22.122Z", "objectIds": [ "958741182", "958750871", "958834290", "958730631" ], "place": { "name": "city=Montague/intersection=Lafayette_Agnew" }, "analyticsModule": { "id": "Collision Detection Module", "description": "Potential collision detected between 4 vehicles" }, "category": "collision", "isAnomaly": true, "info": { "location": "42.48837572978232,-90.73894264480816,0.0", "primaryObjectId": "958750871", "verificationResponseCode": "200", "verificationResponseStatus": "OK", "reasoning": "The video shows vehicle 958750871 approaching the intersection...", "verdict": "confirmed" } } ``` ## Scaling and Performance Tuning One setting scales the Alerts Microservice: `alert_agent.processes`, the number of pipeline processes one instance runs. Each process uses about one CPU core and holds its own Kafka consumers, event loop, and clients. The concurrency caps toward the VLM and VIOS backends are profile defaults sized to those backends and are described under [Profile defaults](#profile-defaults). This section covers when more processes help, how to configure them, what startup validates, and how to read the metrics that show which stage is limiting throughput. ### Pipeline Modes `alert_agent.pipeline_mode` selects how each message is dispatched inside a process. The shipped profile uses `event_loop`, and multi-process scaling requires it. | Mode | Dispatch | Use | | --------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------- | | `sync` | Inline in the batch worker thread, sized by `alert_agent.num_workers` | Reference and rollback mode | | `thread_bridge` | Dispatch thread pool with a blocking wait, sized by `alert_agent.async_dispatch_workers` | Legacy asynchronous mode | | `event_loop` | One coroutine per message on a persistent event loop, with asynchronous VLM, VIOS, and Elasticsearch clients | Shipped default. Kafka consumption is decoupled from VLM latency | An invalid value fails startup. When the key is unset, the mode is derived from the deprecated `alert_agent.async_io.enabled` flag; set `pipeline_mode` explicitly instead. `alert_agent.chunk_size` is retired and ignored with a startup warning, as are the per-service switches `async_io.vst_enabled`, `async_io.elastic_enabled`, `async_io.dedup_enabled`, and `async_io.redis_enabled`. ### Scaling Across Processes The pipeline runs inside one Python process, and the interpreter lock allows one thread to execute Python code at a time. A single process therefore uses about one CPU core regardless of the thread count or the host size. `alert_agent.processes` runs the consume and dispatch stack in that many independent operating system processes, each with its own Kafka consumers, event loop, clients, and interpreter lock. #### When more processes help Additional processes add CPU capacity only. They help when a single process is consistently near one CPU core and the processed rate stays below the input rate, as described under [Monitoring throughput](#monitoring-throughput). A process below one core is limited by offered traffic or by a downstream service, and more processes add memory and consumer-group members without adding capacity. #### Configuration For the Alerts developer profile, edit the following file and redeploy: ```text deploy/docker/developer-profiles/dev-profile-alerts/vlm-as-verifier/configs/config.yml ``` The process count is startup topology. It is not resized while the service runs, and the developer profile does not expose it through `user-overrides.env`. A four-process configuration that keeps the profile's instance-wide load on the VLM and VIOS backends unchanged: ```yaml alert_agent: pipeline_mode: event_loop processes: 4 startup_timeout_seconds: 60 async_dispatch_max_in_flight: 4 async_io: max_vlm_concurrent: 1 max_vst_concurrent: 2 persistence: enabled: true ``` Every per-process cap is multiplied by the process count. The one-process profile sets these caps to 16, 4, and 8; dividing them by four keeps `processes × cap` equal to the shipped values, so the backends see the same load after enabling processes. When the VLM is the limiting service, capacity comes from the VLM side, for example more replicas behind the same endpoint. #### Rules 1. `processes` above `1` requires `pipeline_mode: event_loop`. The other modes hold their concurrency in threads, with no per-process cap on the load offered to the backends. 2. `processes` above `1` requires shared alert-configuration storage: `persistence.enabled: true` with Elasticsearch reachable. 3. Every input topic must have at least as many partitions as consumer-group members, counted per topic and not summed. The members are the pipeline processes of every Alerts Microservice instance that shares the consumer group, so with `replicas` instances: ```text replicas × processes ≤ partitions on every input topic ``` The developer profile creates eight partitions on each topic. One instance can run four processes without changing the Kafka layout; two instances with four processes each use all eight assignments. 4. Work is spread across processes by Kafka key. The shipped Behavior Analytics pipeline keys `mdx-incidents` by `sensorId`, so every message of one stream lands on one partition and one process, and every process receives work only when there are at least as many active streams as processes. Alerts submitted through the HTTP API are keyed by event id and spread across partitions regardless of stream. 5. Allocate about one CPU core per pipeline process, plus capacity for the parent and API processes. Each process holds its own Kafka consumers, HTTP connection pools to the VLM and VIOS backends, Elasticsearch clients, and memory, so connection, file-descriptor, and memory use grow with the process count. 6. Redeploy after changing the process count. Startup validates the count against the mode, the persistence setting, and the live partition count of every input topic. A combination the deployment cannot honor fails startup with a message that names the topic and its partition count. The service never runs fewer processes than configured. Startup also logs the resolved instance-wide caps, `processes × cap` for each backend. #### Startup, readiness, and shutdown * `startup_timeout_seconds` covers topic metadata, prompt seeding, VLM warmup, and every process joining its consumer group. The default is 60 seconds and the minimum is 30. With one process, the startup watchdog stops the instance when the budget runs out. With more than one process, it stops the instance 5 seconds after the budget, so size a startup probe against 65 seconds at the default. * `/health` and `/ready` return HTTP 503 until every configured process holds a decided Kafka assignment, and again during a rebalance until the assignment is decided. Do not point a liveness probe at them. * The shipped Docker and Helm definitions allow 60 seconds for shutdown. Keep that grace period so in-flight work can drain. * A pipeline process that exits takes the instance down, and the orchestrator restarts it. Processes are not respawned individually. #### Rollback Set `processes: 1` and redeploy without overlapping the old and new instances. No data migration is needed. ### Scaling Across Instances Multiple Alerts Microservice instances share `event_bridge.kafka_source.group_id`, and the broker distributes partitions across them. Consumer-group members are instances and processes together, so the partition rule above applies to the product `replicas × processes`. Instances share no state. In-process deduplication relies on every message of one sensor reaching the same member, which holds when the producer keys by `sensorId`, as the shipped Behavior Analytics pipeline does. Confirmed-verdict protection is stored in Elasticsearch and holds across members. ### Monitoring throughput The service keeps up when the processed rate matches the input rate. Compare the two over the same window: ```text input rate(alert_bridge_events_after_dedup_total[5m]) processed rate(alert_bridge_events_total[5m]) ``` When processed stays below input, `alert_bridge_kafka_lag_duration_seconds` grows, because the in-flight bound applies backpressure to the Kafka consumer and unread messages wait in the topic. The stage that is limiting shows as growing wait time: | Stage | Signal | Capacity Comes From | | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | | CPU | Process CPU near one core, `alert_bridge_dispatch_wait_seconds` growing while VLM and VIOS waits stay flat | More pipeline processes | | VLM | `alert_bridge_capacity_wait_seconds{service="vlm"}` growing, `alert_bridge_event_loop_vlm_in_flight` pinned at the cap, `alert_bridge_vlm_duration_seconds` rising | VLM backend capacity | | VIOS | `alert_bridge_capacity_wait_seconds{service="vst"}` growing, `alert_bridge_vst_duration_seconds` rising | VIOS capacity | With `processes` above `1`, also watch the following series: | Metric | Interpretation | | --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `alert_bridge_pipeline_processes_configured` | Requested pipeline-process count. | | `alert_bridge_pipeline_processes_alive` | Processes currently running. Should equal the configured count. | | `alert_bridge_pipeline_processes_ready` | Processes with a decided Kafka assignment. | | `alert_bridge_assigned_partitions` | Partitions currently assigned to the instance. A value below the process count means some processes hold nothing. | | `alert_bridge_pipeline_process_exits_total{reason}` | Process exits. `reason` is `shutdown` for an exit that a stop requested, or `unexpected`. | | `alert_bridge_rebalance_drains_total{outcome}` | Drains of revoked partitions. `outcome` is `nothing_owed` when the partitions had no work in flight, `drained` when the work finished before the deadline, or `timed_out`. | | `alert_bridge_records_read_after_revoke_total` | Records observed after partition ownership changed. Investigate sustained growth. | Metric names, labels, and scrape configuration are described under [Metrics](#metrics). ### Profile defaults The following caps are set in the shipped profile to match the backends it deploys with. They are per process, so the instance-wide value is `processes × cap`, and they are divided by the process count when `processes` is raised, as in the example above. They are not tuned to raise throughput: a cap above what the backend serves concurrently queues requests inside the backend and raises per-call latency without adding throughput. | Setting | Shipped Value | Meaning | | ------------------------------------------ | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `alert_agent.async_io.max_vlm_concurrent` | `4` | VLM requests in flight per process. Sized to what the VLM backend serves concurrently. | | `alert_agent.async_io.max_vst_concurrent` | `8` | VIOS requests in flight per process. | | `alert_agent.async_dispatch_max_in_flight` | `16` | Post-deduplication messages accepted but not finished, per process. Offsets are committed when a record is read, so this bounds memory and the number of read-but-unfinished messages lost if a process exits. | | `alert_agent.startup_timeout_seconds` | `60` | Budget for the whole of startup. Minimum 30. | `alert_agent.num_workers` applies to `sync` mode only and is kept for the rollback path. The shipped `event_loop` mode does not use it. ## Metrics [Prometheus](https://prometheus.io/) is a common open-source monitoring stack: a Prometheus server scrapes HTTP metrics endpoints on a schedule, stores the samples as labeled time series, and exposes the PromQL query language for charts and alerts. See the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/) for terminology and query examples. With metrics enabled, the Alerts Microservice serves Prometheus text format on `/metrics`; the port and enable flag are in the Configuration subsection. Metrics record pipeline latency and event throughput at the moment of processing. Series are held in memory as Prometheus histograms and counters; the `/metrics` endpoint returns cumulative bucket counts, sums, and counts from which Prometheus computes rates and approximate quantiles at query time. No per-event raw samples are stored. Configure Prometheus to scrape that URL, then use the Prometheus or Grafana UI to run PromQL. Deployments that include the blueprint observability bundle can follow [Observability](/system-components/middleware/observability) for UI URLs. ### Configuration Prometheus exposure is controlled by environment variables: | Option | Default | Meaning | | ---------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- | | `PROMETHEUS_METRICS_ENABLED` | `false` | Set to `true` to enable Prometheus metrics. When false, metric objects are not imported and the scrape port is not opened. | | `PROMETHEUS_PORT` | `9081` | Port for the Prometheus scrape server. Scrape `http://:/metrics`. | Metric shape and coverage are affected by these `config.yml` keys: | Key | Default | Effect | | -------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `alert_agent.metrics.per_sensor_labels` | `false` | Emits the opt-in `sensorId` histogram and counter variants listed below. | | `alert_agent.event_filters.end_time_delta_filter.enabled` | `false` | When enabled, dropped events increment `alert_bridge_events_dropped_total{reason="end_time_delta"}`. | | `vlm_rate_limit_enabled` | `false` | When enabled, VLM rate-limit drops increment `alert_bridge_events_dropped_total{reason="rate_limit"}`. | | `alert_agent.event_filters.protect_confirmed_verdicts.enabled` | `false` | When enabled, confirmed-verdict short-circuits increment `alert_bridge_events_skipped_confirmed_total` instead of `alert_bridge_events_total`. | | `alert_agent.include_latency_info` | `false` | Not required for Prometheus metrics. It only controls whether latency fields are written into output documents for Elasticsearch-based analysis. | > **Note** > > Dedup and filter tuning moved from `event_bridge.redis_source` to `alert_agent.event_filters` when Redis was removed. The old section is still read when the new one is absent, and the service logs a warning naming it, so an existing config keeps working — but new deployments should use `alert_agent.event_filters`. When per-sensor labels are enabled, missing, non-string, empty, or over-128-character sensor IDs are folded into `sensorId="unknown"`. Valid string IDs, including IDs with punctuation such as `/`, `:`, `?`, or `.`, pass through unchanged after trimming whitespace. The recorder caps distinct `sensorId` label values at 128 per process; any additional unseen ID is recorded as `sensorId="unknown_overflow"`. ### Pipeline histograms These histograms expose `_bucket`, `_sum`, and `_count` series. | Metric | Measures | | ------------------------------------------------- | --------------------------------------------------------------- | | `alert_bridge_upstream_duration_seconds` | Time from event end to Kafka publish (upstream path). | | `alert_bridge_kafka_lag_duration_seconds` | Time from Kafka publish to microservice consume. | | `alert_bridge_worker_queue_wait_duration_seconds` | Time from consume to worker assignment. | | `alert_bridge_vst_duration_seconds` | VIOS (VST) video URL fetch duration. | | `alert_bridge_video_length_seconds` | Effective clip length requested from VIOS (VST). | | `alert_bridge_vlm_duration_seconds` | Each VLM attempt, including retries and failed API calls. | | `alert_bridge_worker_processing_seconds` | Time from worker assignment to sink-ready completion. | | `alert_bridge_e2e_duration_seconds` | Time from event end to sink-ready completion (end-to-end view). | ### Event counters | Metric | Labels | Meaning | | --------------------------------------------- | --------- | ----------------------------------------------------------------------- | | `alert_bridge_events_after_dedup_total` | (none) | Events entering per-event processing after filters. | | `alert_bridge_events_dropped_total` | `reason` | Batch-level drops before per-event processing. | | `alert_bridge_events_skipped_confirmed_total` | (none) | Events skipped because a confirmed verdict was already stored for them. | | `alert_bridge_events_total` | `verdict` | Events completed by the per-event recorder. | | `alert_bridge_verification_failures_total` | `reason` | Events that failed verification. | **Drop reasons** (`alert_bridge_events_dropped_total`): `end_time_delta`, `dedup`, `rate_limit`. Unexpected call-site values are defensively folded into `unknown`. **Verdicts** (`alert_bridge_events_total`): `confirmed`, `rejected`, `verification-failed`, `unknown`. **Verification failure reasons** (for breakdowns and operations that label by failure): `vst_timeout`, `vst_overloaded`, `vst_not_found`, `vst_unavailable`, `vst_client_error`, `vst_server_error`, `vst_unknown`, `url_validation`, `vlm_parse_failure`, `vlm_timeout`, `vlm_connection_error`, `vlm_server_error`, `vlm_invalid_payload`, `no_prompt`, `redis_unavailable`, `unknown`. `redis_unavailable` is a legacy label name, kept so existing dashboards and alerting rules keep matching. It is emitted only when the pre-pipeline confirmed-verdict check raises a Redis-named exception, which the Elasticsearch-backed verdict store does not do — expect it to stay at zero. Watch `alert_bridge_verdict_fail_open_total` for verdict-store trouble instead. ### Per-sensor variants When `alert_agent.metrics.per_sensor_labels` is `true`, the service also emits the per-sensor series below. Histogram names use `_by_sensor_seconds`; counters use `_by_sensor_total`. | Aggregate histogram | Per-sensor variant | | ------------------------------------------------- | --------------------------------------------------------------------- | | `alert_bridge_upstream_duration_seconds` | `alert_bridge_upstream_duration_by_sensor_seconds{sensorId}` | | `alert_bridge_kafka_lag_duration_seconds` | `alert_bridge_kafka_lag_duration_by_sensor_seconds{sensorId}` | | `alert_bridge_worker_queue_wait_duration_seconds` | `alert_bridge_worker_queue_wait_duration_by_sensor_seconds{sensorId}` | | `alert_bridge_vst_duration_seconds` | `alert_bridge_vst_duration_by_sensor_seconds{sensorId}` | | `alert_bridge_video_length_seconds` | `alert_bridge_video_length_by_sensor_seconds{sensorId}` | | `alert_bridge_vlm_duration_seconds` | `alert_bridge_vlm_duration_by_sensor_seconds{sensorId}` | | `alert_bridge_worker_processing_seconds` | `alert_bridge_worker_processing_by_sensor_seconds{sensorId}` | | `alert_bridge_e2e_duration_seconds` | `alert_bridge_e2e_duration_by_sensor_seconds{sensorId}` | | Aggregate counter | Per-sensor variant | | -------------------------------------------------- | --------------------------------------------------------------------- | | `alert_bridge_events_after_dedup_total` | `alert_bridge_events_after_dedup_by_sensor_total{sensorId}` | | `alert_bridge_events_dropped_total{reason}` | `alert_bridge_events_dropped_by_sensor_total{reason,sensorId}` | | `alert_bridge_events_skipped_confirmed_total` | `alert_bridge_events_skipped_confirmed_by_sensor_total{sensorId}` | | `alert_bridge_events_total{verdict}` | `alert_bridge_events_by_sensor_total{verdict,sensorId}` | | `alert_bridge_verification_failures_total{reason}` | `alert_bridge_verification_failures_by_sensor_total{reason,sensorId}` | ### Additional metrics **Async I/O and sink** | Metric | Labels | Meaning | | ------------------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------------------------- | | `alert_bridge_async_external_io_duration_seconds` | `operation`, `mode`, `result` | Duration of async-capable VIOS/VST, sink, and dedup-state operations, including sync fallbacks. | | `alert_bridge_async_external_io_fallback_total` | `operation`, `reason` | Async operations that fell back to a synchronous path. | | `alert_bridge_async_sink_in_flight` | (none) | Current number of in-flight async sink operations. | Known `mode` values are `sync`, `async`, `sync_fallback`, and `async_submit`. Known `result` values are `success`, `error`, `timeout`, and `cancelled`. Known fallback `reason` values are `timeout`, `error`, `submit_error`, and `executor_unavailable`. **Confirmed-verdict protection** (active when `alert_agent.event_filters.protect_confirmed_verdicts.enabled` is `true`) | Metric | Labels | Meaning | | ----------------------------------------------------------- | --------- | ------------------------------------------------------------------------------ | | `alert_bridge_verdict_fail_open_total` | `reason` | Verdict checks that could not confirm and so let the event proceed to the VLM. | | `alert_bridge_verdict_es_get_duration_seconds` | (none) | Latency of one confirmed-verdict marker read from Elasticsearch. | | `alert_bridge_verdict_retention_deleted_total` | (none) | Expired markers deleted by the retention job. | | `alert_bridge_verdict_retention_runs_total` | `outcome` | Retention job runs, labelled `success` or `failure`. | | `alert_bridge_verdict_retention_last_run_timestamp_seconds` | (none) | Unix timestamp of the last successful retention run. | The check is deliberately fail-open: an unreachable store re-verifies an event rather than blocking it, so the store's health shows up here and not as a verification failure. Known `reason` values are `es_down` (client unavailable, including while disabled by backoff), `error` (the read raised), and `write_error` (the marker write failed) — these three are the outage signal — plus `expired` and `malformed`, which describe the marker that was found rather than a store problem. **Realtime and API** (where enabled in the deployment) | Metric | Labels | Meaning | | ---------------------------------------------- | --------- | ---------------------------------------------------------------------------------- | | `alert_bridge_pipeline_processes_configured` | (none) | Pipeline processes requested by `alert_agent.processes`. | | `alert_bridge_pipeline_processes_alive` | (none) | Pipeline processes currently running. | | `alert_bridge_pipeline_processes_ready` | (none) | Pipeline processes holding a decided Kafka assignment. | | `alert_bridge_assigned_partitions` | (none) | Source partitions currently assigned to the instance, summed across its processes. | | `alert_bridge_pipeline_process_exits_total` | `reason` | Pipeline process exits, labelled `shutdown` or `unexpected`. | | `alert_bridge_rebalance_drains_total` | `outcome` | Drains of revoked partitions, labelled `nothing_owed`, `drained`, or `timed_out`. | | `alert_bridge_records_read_after_revoke_total` | (none) | Records read for a partition after this member lost it. | | `alert_bridge_realtime_rules_active` | (none) | Current in-memory realtime rules registered with the service. | | `alert_bridge_realtime_rules_count` | (none) | Current durable realtime rules with `status=ACTIVE` in Elasticsearch. | | `alert_bridge_realtime_rules_created_total` | (none) | Realtime rules created and committed to the live registry. | | `alert_bridge_realtime_rules_persisted_total` | (none) | Realtime rules that reached durable `status=ACTIVE` storage in Elasticsearch. | | `alert_bridge_realtime_rules_deleted_total` | (none) | Realtime rules deleted from the live registry. | | `alert_bridge_realtime_rules_failed_total` | `stage` | Realtime rule creation or caption-task failures by stage. | | `alert_bridge_replay_invocations_total` | `outcome` | Replay loops that actually ran, labelled `success`, `partial`, or `failed`. | | `alert_bridge_replay_rule_failures_total` | (none) | Individual rules that failed during replay. | | `alert_bridge_rtvi_call_duration_seconds` | `method` | Duration of RTVI VLM calls made by realtime rule management. | | `alert_bridge_rtvi_call_failures_total` | `method` | Failed RTVI VLM calls made by realtime rule management. | | `alert_bridge_incident_query_duration_seconds` | (none) | Duration of Elasticsearch incident queries. | | `alert_bridge_incident_query_failures_total` | (none) | Failed Elasticsearch incident queries. | Known realtime failure `stage` values are `validation`, `es_persist`, `start_stream`, `missing_stream_id`, `generate_captions`, `es_update`, `caption_task_http`, and `caption_task_crash`. Known RTVI `method` values are `start_stream`, `generate_captions`, `stop_stream`, and `stop_captions`. Replay short-circuits such as persistence-disabled or replay-already-in-flight do not increment `alert_bridge_replay_invocations_total`. ### Realtime incident consolidation metrics These metrics are emitted only when `GET /api/v1/realtime/incidents` is called with `consolidate=true`. They describe the read-time grouping of confirmed RT-VLM chunks into events and are separate from the ingestion-time deduplication metrics above (`alert_bridge_events_after_dedup_total` and `alert_bridge_events_dropped_total{reason="dedup"}`), which count events entering alert verification. See [Realtime incident consolidation](/system-components/analytics-microservices/alerts-microservice/alert-verification-api#realtime-incident-consolidation) for the API contract. | Metric | Labels | Meaning | | --------------------------------------- | -------- | -------------------------------------------------------------------------- | | `alert_bridge_dedup_chunks_in_total` | (none) | Raw confirmed RT-VLM chunk incidents fed into the consolidator. | | `alert_bridge_dedup_events_out_total` | (none) | Consolidated events produced from those chunks. | | `alert_bridge_dedup_split_reason_total` | `reason` | Why an event ended and a new one began. | | `alert_bridge_dedup_duration_seconds` | (none) | Time spent grouping chunks into events, excluding the Elasticsearch query. | Known split `reason` values are `gap` (the inter-alert gap was exceeded) and `outer` (the maximum event duration was exceeded). A third value, `malformed`, is defined for a chunk whose `timestamp` or `end` cannot be parsed, but chunks without a parseable `timestamp` are dropped before grouping and a missing `end` falls back to `timestamp`, so it is not expected to occur in practice. The ratio of `alert_bridge_dedup_chunks_in_total` to `alert_bridge_dedup_events_out_total` shows how many chunks each event absorbs on average. The metrics carry no per-sensor labels. ### Sample PromQL queries The examples use `sum by (le) (...)` so quantiles aggregate correctly across multiple scrape targets or multiprocess workers. When `alert_agent.metrics.per_sensor_labels` is `true`, add `sensorId` to the grouping (for example `sum by (sensorId, le) (...)`) for per-camera views. Quickly confirm that `alert_bridge` series are present (default port shown; use your `PROMETHEUS_PORT`): ```bash curl -s http://localhost:9081/metrics | grep alert_bridge ``` End-to-end latency (P50 / P90 / P99): ```text histogram_quantile(0.50, sum by (le) (rate(alert_bridge_e2e_duration_seconds_bucket[5m]))) histogram_quantile(0.90, sum by (le) (rate(alert_bridge_e2e_duration_seconds_bucket[5m]))) histogram_quantile(0.99, sum by (le) (rate(alert_bridge_e2e_duration_seconds_bucket[5m]))) ``` VLM stage: quantiles and average latency over a range: ```text histogram_quantile(0.50, sum by (le) (rate(alert_bridge_vlm_duration_seconds_bucket[5m]))) histogram_quantile(0.90, sum by (le) (rate(alert_bridge_vlm_duration_seconds_bucket[5m]))) histogram_quantile(0.99, sum by (le) (rate(alert_bridge_vlm_duration_seconds_bucket[5m]))) sum(rate(alert_bridge_vlm_duration_seconds_sum[5m])) / sum(rate(alert_bridge_vlm_duration_seconds_count[5m])) ``` Kafka consumer lag (P90): ```text histogram_quantile(0.90, sum by (le) (rate(alert_bridge_kafka_lag_duration_seconds_bucket[5m]))) ``` Worker throughput (events completed per second, all outcomes): ```text sum(rate(alert_bridge_worker_processing_seconds_count[1m])) ``` Drop rate by reason: ```text sum by (reason) (rate(alert_bridge_events_dropped_total[5m])) ``` Events per second by verdict: ```text sum by (verdict) (rate(alert_bridge_events_total[1m])) ``` ## OpenTelemetry Tracing With tracing enabled, the Alerts Microservice starts one [OpenTelemetry](https://opentelemetry.io/) trace per alert and exports the sampled ones over OTLP/HTTP to the configured endpoint; it does not open a port or store spans itself. The shipped sampling ratio is `0.1`, so about one alert in ten is exported. Where the metrics above aggregate across events, a trace covers a single alert. Tracing is off by default and is enabled independently of Prometheus metrics. Turning it off, or leaving it off, does not change alert processing or the `alert_bridge_*` Prometheus metrics. Spans need a collector and a backend to reach. The Docker Compose deployment ships an OpenTelemetry Collector and a Jaeger instance behind their own Compose profiles, neither of which is in a shipped profile list. The microservice sends OTLP/HTTP to the collector on port `4318`. The collector forwards traces to both backends: * Jaeger, with its UI at `http://:16686` when the `jaeger` profile is enabled. * Phoenix, through the trace ingestion and UI service already deployed with the alert profiles. The collector's published ports are bound to `127.0.0.1` on the host; services reach the collector over the Compose network. > **Warning** > > The OTLP receivers on ports `4317` and `4318` are unauthenticated. Any client that can reach them can inject forged telemetry into Jaeger and Phoenix. Keep them on the loopback binding unless the host is on a trusted network, or the ports are protected by a firewall, an ACL, or an authenticated gateway. `OTEL_COLLECTOR_GRPC_HOST_PORT` and `OTEL_COLLECTOR_HTTP_HOST_PORT` change these bindings. The Prometheus scrape port `8889` is read-only and is also bound to `127.0.0.1`. Set `OTEL_COLLECTOR_METRICS_HOST_PORT` to `0.0.0.0:8889` when a Prometheus server outside the host needs to scrape it. Workflow endpoints are listed in [Alert Verification](/vision-agent/agent-workflows/alert-verification#service-endpoints) and [Real-Time Alerts](/vision-agent/agent-workflows/real-time-alerts#service-endpoints). ### Enabling tracing In `deploy/docker/developer-profiles/dev-profile-alerts/user-overrides.env`, the file the workflow pages create from the `overrides.env` template: 1. Set `ALERT_ENABLE_OTEL_MONITORING=true`. 2. Append `otel-collector,jaeger` to both `COMPOSE_PROFILES_CV` and `COMPOSE_PROFILES_VLM`. 3. Run the `docker compose ... up --detach --pull always --force-recreate --build` command from the workflow page again. Add the two profiles to the list definitions rather than to `COMPOSE_PROFILES`, which is derived from them. Then open the Jaeger UI and select the `vss-alert-ms` service. ### Span hierarchy Each alert produces one root span, `Alert Verification`, covering the event from ingestion to sink completion. Child spans divide that interval. ![One alert as a span tree in the Jaeger UI](/vss/_fern-img/dfd517d3fe230ab3d85ef2861be7007598225d2d08ecfd752db0675023378b88.webp) | Span | Covers | | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `Alert Verification` | Root span for one alert, from ingestion to sink-ready completion. | | `Kafka Consume Lag` | Kafka publish to microservice consume. | | `Worker Queue Wait` | Consume to worker assignment. | | `Dispatch Wait` | Task dispatch to task start. | | `VST Video URL Resolution (overlay)` | VIOS (VST) video URL fetch with overlay. | | `VST Video URL Resolution (no overlay)` | The retry fallback fetch. Both VST spans can appear for one event: the no-overlay call is a fallback, not a replacement. | | `VST Capacity Wait`, `VLM Capacity Wait` | Time spent waiting for a per-service concurrency slot before the call is made, one span per acquisition. A long wait here means the microservice is at its own configured limit rather than waiting on the downstream service. | | `VLM Request` | One VLM attempt. The retry loop emits one span per attempt, so a retried event shows several. | | `Elasticsearch get`, `Elasticsearch update`, `Elasticsearch write` | Elasticsearch reads and writes made while processing the event. | `Dispatch Wait`, the two `Capacity Wait` spans, and `VLM Request` are emitted by the `event_loop` pipeline, which is the shipped default. The synchronous path does not emit them. Requests to the REST API produce their own server spans through FastAPI instrumentation, and outbound HTTP calls to VIOS (VST) and the VLM backend add client spans beneath whichever span is active. Each span carries its attributes, such as `category`, `sensorId`, `pipeline_mode`, and `verdict` on the root span. Spans rebuilt from recorded timestamps carry `timing.source=reconstructed`. ![Span attributes in the Jaeger UI](/vss/_fern-img/f3576c93d05a74260ac865636cca71c646c3ff91f3f0f968197d8fc18d54302c.webp) ### Trace context Trace context is read from Kafka records. When an upstream producer stamps a W3C `traceparent` header, the root span for that record continues the caller's trace. One trace then spans the producer and the Alerts Microservice. Records without the header start their own trace. Each record uses its own header, so a batch carrying several callers is not collapsed onto one of them. REST requests carrying a `traceparent` header are joined the same way. The same mechanism can join the Alerts Microservice with the Real-Time VLM microservice, which also exports OpenTelemetry traces. Neither path is wired today: the records it produces carry no `traceparent` header, and its HTTP server does not read the header that the Alerts Microservice sends on REST calls. Until one of them is added, the two services appear as separate traces in the same backend. The current propagation boundaries are: * Inbound: Kafka record headers and REST request headers. * Outbound: HTTP calls to VIOS (VST) and the VLM backend carry the active trace context. The consolidation Kafka sink stamps `traceparent` on the records it produces; other produce sites do not. * Not included: `vss_agents` does not stamp `traceparent` on the records it produces, and Elasticsearch requests do not carry trace context. ### OpenTelemetry metrics With tracing enabled, the microservice also exports three OpenTelemetry metrics on the same OTLP/HTTP connection. The collector exposes them in Prometheus format on port `8889`, bound to `127.0.0.1` by default. No shipped profile scrapes the collector; the `dev-profile-alerts` profile does not deploy Prometheus. The `alert_bridge_*` metrics and the `/metrics` endpoint described above are unchanged. A Prometheus server you deploy can scrape the collector, the `/metrics` endpoint, or both. | Metric | Type | Attributes | Measures | | ------------------------------ | ------------- | -------------------------- | -------------------------------------------------------------------------------- | | `alert.verification.duration` | Histogram (s) | `pipeline_mode`, `verdict` | End-to-end verification time per event, from event end to sink-ready completion. | | `alert.vlm.attempt.count` | Counter | `success`, `attempt` | VLM requests, counted per attempt rather than per event. | | `alert.capacity.wait.duration` | Histogram (s) | `service` | Time spent waiting for a VST or VLM concurrency slot. | ### Log correlation Log lines written while a trace is active end with `trace_id= span_id=`. Use the `trace_id` with the Jaeger lookup by trace ID to open the trace for that log line. The IDs are appended for every alert, but the trace is present in Jaeger only when it was sampled. Log lines written outside a trace are unchanged. ### Configuration Tracing is switched on with an environment variable. The values below are the ones the Docker Compose deployment sets. A container run outside it keeps the service name `vss-alert-ms` and falls back to the OpenTelemetry SDK default endpoint on `localhost`: | Option | Compose value | Meaning | | ----------------------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | | `ENABLE_OTEL_MONITORING` | `false` | Set to `true` to enable tracing and OpenTelemetry metrics. When false, the OpenTelemetry SDK is not initialized and no spans are created. | | `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://otel-collector:4318` | Base OTLP/HTTP endpoint. The exporter appends `/v1/traces` and `/v1/metrics`. | | `OTEL_SERVICE_NAME` | `vss-alert-ms` | Service name that spans are recorded under in the backend. | | `OTEL_TRACES_EXPORTER` | `otlp` | Set to `none` to keep tracing initialized but export no spans. | | `OTEL_METRICS_EXPORTER` | Same as `OTEL_TRACES_EXPORTER` | Set to `none` to export no OpenTelemetry metrics. Follows the traces setting unless set explicitly. | | `OTEL_METRIC_EXPORT_INTERVAL` | `60000` | Metric export interval, in milliseconds. | | `OTEL_METRIC_EXPORT_TIMEOUT` | `5000` | Metric export timeout, in milliseconds. | | `OTEL_RESOURCE_ATTRIBUTES` | (empty) | Additional resource attributes, as `key=value` pairs. | | `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT` | `4096` | SDK-level ceiling on attribute value length; unlimited when unset. Applies after the `max_content_chars` limit below. | In the Docker Compose deployment, each option is set through the matching `ALERT_`-prefixed variable in `user-overrides.env`. `ALERT_ENABLE_OTEL_MONITORING` sets `ENABLE_OTEL_MONITORING`, `ALERT_OTEL_EXPORTER_OTLP_ENDPOINT` sets `OTEL_EXPORTER_OTLP_ENDPOINT`, and so on. The container receives the unprefixed name. Tracing behavior is tuned with these `config.yml` keys: | Key | Default | Effect | | --------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `alert_agent.tracing.sampling_ratio` | `0.1` | Fraction of traces sampled, 0.0-1.0. Applied with ParentBased sampling, so a record whose caller was sampled keeps its whole trace. It affects export volume and backend storage. In the recorded test below, throughput at `0.1` and at `1.0` was within about one point, so lowering the ratio does not recover the tracing overhead. | | `alert_agent.tracing.include_content` | `false` | When true, prompt text, VLM responses, and video URLs are attached to spans. See the warning below. | | `alert_agent.tracing.max_content_chars` | `512` | Maximum characters of any content attribute, applied only when `include_content` is true. Truncated values end with `...[+N chars]`, and that marker counts against the limit. `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT` applies after it and adds no marker, so the effective ceiling is the lower of the two. | > **Warning** > > `include_content` records prompts, VLM responses, and video URLs in spans. Alert data may contain personally identifiable information, and video URLs may carry presigned access tokens. Do not enable it where the tracing backend is not equally protected. ### Performance In a recorded test, the microservice processed 39.24 events/s with tracing disabled and 35.31 events/s with tracing enabled at the shipped `sampling_ratio` of `0.1`, about 10% lower. The test ran one pipeline process against the simulator rig under `services/alert/test/functional/capability/`, with no GPU, a simulated VLM, an offered rate of 120 events/s, and three 45-second repeats per arm. It measured the cost of producing telemetry; the exporters were set to `none`, so network and collector cost were not included. Results on other hardware and workloads will differ. Choose the sampling ratio by the trace volume and backend storage you need. To remove the overhead, disable tracing. ## Configuration The Alerts Microservice is configured via a YAML-based configuration file. The following tables describe key configuration parameters organized by category. ### VIOS Configuration | Parameter | Default | Description | | ------------------------------------- | ---------------------------- | ---------------------------------------------------- | | `vst_config.base_url` | `http://localhost:30888` | Base URL for VIOS APIs. | | `vst_config.sensor_list_endpoint` | `/vst/api/v1/sensor/streams` | Endpoint for retrieving sensor/stream list. | | `vst_config.segment_anchor` | `end` | Segment anchor mode (`end` for end-anchored window). | | `vst_config.segment_duration_seconds` | `10` | Clip segment duration in seconds. | | `vst_config.add_overlay` | `false` | Enable bounding box overlay on video segments. | ### Kafka Configuration | Parameter | Default | Description | | ----------------------------- | ------------------------ | ------------------------------------------------- | | `kafka.bootstrap_servers` | `localhost:9092` | Kafka broker addresses. | | `kafka.group_id` | `kafka-incidents-dumper` | Default Kafka consumer group ID. | | `kafka.auto_offset_reset` | `latest` | Offset reset policy (earliest/latest). | | `kafka.enable_auto_commit` | `false` | Enable automatic offset commit. | | `kafka.max_poll_records` | `10` | Maximum records per poll. | | `kafka.max_poll_interval_ms` | `60000` | Maximum interval between polls (ms). | | `kafka.session_timeout_ms` | `10000` | Session timeout (ms). | | `kafka.heartbeat_interval_ms` | `3000` | Heartbeat interval (ms). | | `kafka.poll_timeout` | `100` | Kafka consumer poll wait timeout (ms). | | `kafka.message_type` | `Incident` | Protobuf message type (`Incident` or `Behavior`). | ### Event Bridge Configuration | Parameter | Default | Description | | ------------------------------------------- | ------------------------ | ---------------------------- | | `event_bridge.sourceType` | `kafka` | Source type (`kafka`). | | `event_bridge.sinkType` | `kafka` | Sink type (`kafka`). | | `event_bridge.kafka_source.group_id` | `alert-bridge-vlm-group` | Kafka source consumer group. | | `event_bridge.kafka_source.topics.incident` | `mdx-incidents` | Kafka topic for incidents. | | `event_bridge.kafka_source.topics.alert` | `mdx-alerts` | Kafka topic for alerts. | ### VLM Configuration > **Note** > > For detailed VLM API documentation, refer to the [Cosmos3 Nano Reasoner NIM](https://build.nvidia.com/nvidia/cosmos3-nano-reasoner). | Parameter | Default | Description | | ------------------------------ | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `vlm.base_url` | (required) | OpenAI-compatible VLM endpoint URL. | | `vlm.model` | `nvidia/cosmos3-nano-reasoner` | VLM model name. | | `vlm.max_tokens` | `4096` | Maximum tokens for VLM responses. | | `vlm.min_pixels` | `1568` | Minimum pixel budget for video frames. | | `vlm.max_pixels` | `345600` | Maximum pixel budget for video frames. | | `vlm.num_frames` | `5` | Number of video frames to sample. | | `vlm.enable_sampling` | `false` | Enable frame sampling mode. | | `vlm.sampling_fps` | `4` | Sampling FPS when enabled. | | `vlm.temperature` | `0.6` | VLM sampling temperature. | | `vlm.request_timeout` | `5` | Per-request timeout in seconds for VLM API calls. Increase for larger models or longer videos. | | `vlm.max_retries` | `1` | Number of VLM request retries on API or validation failure. | | `vlm.use_vlm_media_defaults` | `false` | When true, skips custom pixel and frame parameters and lets the VLM backend use its own defaults. | | `vlm.response_parser` | (unset) | Dotted Python path to a pluggable response parser class (for example, `parsers.my_classifier.MyClassifier`). When set, replaces the built-in verification parsing and writes the parser's output into `info["vlm_response"]` instead of `info["reasoning"]`. Default: unset (built-in verification parser is used). See [Pluggable Response Parser](#pluggable-response-parser). | | `vlm.warmup.poll_timeout` | `300` | Seconds to wait for VLM NIM readiness at startup before aborting. | | `vlm.warmup.poll_interval` | `10` | Seconds between NIM health polls during warmup. | | `vlm.warmup.inference_timeout` | `120` | Timeout in seconds for the warmup inference request. | | `vlm.warmup.num_requests` | `3` | Number of warmup inference rounds to send at startup. Each round retries up to 3 times on failure. Set to `1` for faster startup on pre-warmed models. | > **Note** > > The `min_pixels` and `max_pixels` parameters must be set in accordance with the VLM's maximum context window. > **Note** > > VLM warmup runs automatically at startup by default and sends 3 inference rounds to prime the GPU model cache. To disable warmup entirely, set the environment variable `VLM_WARMUP_ENABLED=false`. To adjust the number of rounds, set `vlm.warmup.num_requests` in the config. The `vlm.warmup.*` parameters are optional; defaults work for most deployments. ### Alerts Microservice Configuration | Parameter | Default | Description | | ------------------------------------------ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `alert_agent.always_on` | `false` | Enables `POST /api/v1/realtime/always-on`. Docker Compose substitutes `ALERT_AGENT_ALWAYS_ON` into this setting. When disabled, the endpoint returns HTTP `503` with reason `ALWAYS_ON_DISABLED`. | | `alert_agent.pipeline_mode` | `event_loop` | Dispatch model: `sync`, `thread_bridge`, or `event_loop`. See [Pipeline Modes](#pipeline-modes). | | `alert_agent.processes` | `1` | Independent pipeline processes. Values above `1` require `event_loop`, `persistence.enabled: true`, and enough Kafka partitions. See [Scaling Across Processes](#scaling-across-processes). | | `alert_agent.startup_timeout_seconds` | `60` | Budget for the whole of startup. Minimum 30. | | `alert_agent.num_workers` | `10` | Worker threads in `sync` mode. Not a throughput control in the other modes. | | `alert_agent.async_dispatch_workers` | `num_workers` | Dispatch pool threads in `thread_bridge` mode. | | `alert_agent.async_dispatch_max_in_flight` | `16` | Post-deduplication messages accepted but not finished, per process. | | `alert_agent.async_io.max_vlm_concurrent` | `4` | VLM requests in flight per process in `event_loop` mode. | | `alert_agent.async_io.max_vst_concurrent` | `8` | VIOS requests in flight per process in `event_loop` mode. | | `alert_agent.max_allowed_stream_size` | `2` | Maximum stream size in minutes. | | `alert_agent.default_stream_interval` | `1` | Default stream interval in minutes. | | `alert_agent.vst_pass_through_mode` | `false` | Skip VIOS lookup; use local media files directly. | | `alert_agent.enrichment.enabled` | `false` | Enable alert enrichment for generating detailed event descriptions after verification. | | `alert_agent.metrics.per_sensor_labels` | `false` | When `true`, emits the opt-in `sensorId` histogram and counter variants documented under [Metrics](#metrics). | When always-on alerts are enabled, `ALWAYS_ON_RULES_CONFIG` must point to a valid YAML file containing `always_on_rules`. There is no implicit file lookup or fallback. A missing or invalid file fails service startup; a configuration error encountered by the endpoint returns HTTP `503` with reason `CONFIG_ERROR`. Docker Compose handles the container path: it mounts the host file selected by `VLM_AS_VERIFIER_CONFIG_FILE_REALTIME`, performs environment substitution into `/app/runtime/realtime-config.yml`, and sets `ALWAYS_ON_RULES_CONFIG=/app/runtime/realtime-config.yml`. Set `VLM_AS_VERIFIER_CONFIG_FILE_REALTIME` in the profile override file when you need a different rules source; do not set the container-internal path on the host. #### Event filter settings The Alerts developer profile ships the following `alert_agent.event_filters` values for the upstream alert-verification pipeline. Durations are in seconds unless noted otherwise. | Parameter | Shipped value | Description | | ---------------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------------------ | | `dedup_ttl_seconds` | `5` | Retain an in-process deduplication key for this many seconds. | | `protect_confirmed_verdicts.enabled` | `true` | Skip re-verification when Elasticsearch contains a confirmed-verdict marker. | | `protect_confirmed_verdicts.ttl_seconds` | `600` | Lifetime of a confirmed-verdict marker. | | `protect_confirmed_verdicts.es_retry_backoff_seconds` | `30` | Cooldown before retrying Elasticsearch verdict checks after a store failure. | | `protect_confirmed_verdicts.retention_interval_seconds` | `3600` | Interval between expired-marker retention runs. | | `protect_confirmed_verdicts.retention_requests_per_second` | `50` requests/s | Maximum deletion request rate used by the retention job. | | `end_time_delta_filter.enabled` | `true` | Drop a repeated incident unless its end time has changed by the configured threshold. | | `end_time_delta_filter.threshold_seconds` | `3` | Minimum end-time change required to pass the filter. | | `end_time_delta_filter.ttl_seconds` | `3600` | Lifetime of end-time filter state. | | `end_time_in_dedup_key_categories` | `[]` | Alert categories whose deduplication key retains the incident end timestamp. Other categories omit it. | These are service-level verification filters, not tuning fields for `POST /api/v1/realtime`. Real-time VLM rules tune their prompt and chunk cadence in the request payload. Edit the profile `config.yml` and restart Alert Bridge after changing event filters. ### Elasticsearch Configuration | Parameter | Default | Description | | ------------------------------------------ | ------------------------- | ------------------------------------------- | | `elastic.enabled` | `true` | Enable Elasticsearch persistence. | | `elastic.hosts` | `[http://localhost:9200]` | Elasticsearch host URLs. | | `vlm_enhanced_sink.incident.elastic.index` | `mdx-vlm-incidents` | Elasticsearch index for verified incidents. | | `vlm_enhanced_sink.alert.elastic.index` | `mdx-vlm-alerts` | Elasticsearch index for verified alerts. | ### Prompt Configuration | Parameter | Default | Description | | ---------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `alert_type_config_file` | `alert_type_config.json` | Path to alert type configuration file. | | `prompt.prefer_payload_prompt` | `false` | Prefer prompts from alert payload over stored prompts. | | `prompt.override_prompts_on_start` | `true` | Override stored prompts with config file on startup. | | `prompt.default_system_prompt` | `You are a helpful assistant.` | System prompt used when an alert config has none of its own. A stored `system_prompt` always wins; set to `""` to send no system message. A present-but-empty value (`default_system_prompt:` with nothing after it) fails startup rather than being read as `""`; remove the key to keep the default. | > **Note** > > The shipped `default_system_prompt` only guarantees that a system role is > present — it carries no verification framing of its own (no yes/no contract, no > domain guidance). Deployments that rely on it for more than one alert type > should override it with something verifier-oriented, or set `system` per alert > type. Note also that the opt-out is global: with a default configured, an > individual alert type cannot ask for no system message; clearing its > `system_prompt` puts it back on the default. ### Logging Configuration | Parameter | Default | Description | | --------------------------- | ------------ | -------------------------------------------------------------- | | `logging.level` | `DEBUG` | Application log level (DEBUG, INFO, WARNING, ERROR, CRITICAL). | | `logging.format` | (see config) | Log message format string. | | `logging.third_party_level` | `WARNING` | Log level for third-party libraries. | ## Upgrading from 3.2.x Redis is no longer part of the service. Only the first row below has to be done before you upgrade — it is the one that stops the service from starting. The rest is cleanup you can do whenever. | 3.2.x setting | What 3.3 does with it | What to do | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `event_bridge.sourceType` or `sinkType` set to `redisStream` | Refuses to start: `Unsupported source type: redisStream (only 'kafka' is supported)` | Set both to `kafka`, and repoint whatever wrote to the input stream and whatever read the output stream at Kafka topics. | | Filter tuning under `event_bridge.redis_source` — `dedup_ttl_seconds`, `end_time_in_dedup_key_categories`, `protect_confirmed_verdicts`, `end_time_delta_filter` | Still read, with a warning naming the old section | Move these keys under `alert_agent.event_filters`. | | Connection keys under `event_bridge.redis_source` and `redis_sink` — `host`, `port`, `db`, `streams`, `consumer_group` | Ignored | Delete them, along with the `alert-bridge-redis` container, its volume, and the `depends_on` entry that referenced it. | | `VSS_BASE_URL` and any `vss_agent` section | Ignored | Delete. Clips are retrieved through VIOS/VST and the VLM is called directly. | | `vss_params` in `alert_request_defaults.yaml` | Ignored. The file still loads as long as `vlm_params` and `request_defaults` are present | Nothing — a file carried over from 3.2.x keeps working. | A `vss_params.vlm_params` block in a *request payload* is a separate thing and is still honoured, so older clients keep working. New clients should send `vlm_params` at the top level. ## Docs - [Alert Verification API](https://docs.nvidia.com/vss/dev/system-components/analytics-microservices/alerts-microservice/alert-verification-api.md):