This page is for NVIDIA Infra Controller.
This file contains a list of metrics exported by NVIDIA Infra Controller (NICo). The list is auto-generated from an integration test (test_integration). Metrics no test exercises are added with cargo xtask check-metric-docs --fix. NVLink partition monitor’s metrics are documented in the manual: NVLink Partitioning.
| Name | Type | Description |
| carbide_active_host_firmware_update_count | gauge | Number of host machines in the system currently working on updating their firmware. |
| carbide_alerts_suppressed_count | gauge | Whether external metrics based alerting is suppressed for a specific object |
| carbide_api_admission_admitted_total | counter | Number of API requests admitted for execution |
| carbide_api_admission_handler_execution_duration_seconds | histogram | Duration of admitted API request handler execution |
| carbide_api_admission_pending_requests | gauge | Number of API requests currently waiting for an execution slot |
| carbide_api_admission_pending_wait_duration_seconds | histogram | Duration API requests spent waiting for admission |
| carbide_api_admission_rejected_total | counter | Number of API requests rejected before handler execution |
| carbide_api_admission_work_in_flight | gauge | Number of API requests currently holding an execution slot |
| carbide_api_bmc_proxy_client_reload_failures_total | counter | Number of failed rebuilds of the nico-bmc-proxy-facing client from on-disk certificates. |
| carbide_api_db_queries_total | counter | Number of database queries that occurred inside a span |
| carbide_api_db_span_query_time_milliseconds | histogram | Total time the request spent inside a span on database transactions |
| carbide_api_grpc_server_duration_milliseconds | histogram | Processing time for a request on the carbide API server |
| carbide_api_host_admin_address_reconcile_failures_total | counter | Number of host admin address reconciliations that failed after a discovery committed |
| carbide_api_ready | gauge | Whether the NICo API is running |
| carbide_api_secrets_request_duration_milliseconds | histogram | Duration of Postgres secrets operations, in milliseconds. |
| carbide_api_secrets_requests_failed_total | counter | Number of failed Postgres secrets operations. |
| carbide_api_secrets_requests_succeeded_total | counter | Number of successful Postgres secrets operations. |
| carbide_api_secrets_requests_total | counter | Number of Postgres secrets operations attempted. |
| carbide_api_tls_cert_refreshes_total | counter | Number of TLS acceptor refreshes performed by the API listener |
| carbide_api_tls_connection_attempted_total | counter | Number of inbound TLS connection attempts |
| carbide_api_tls_connection_fail_total | counter | Number of failed inbound TLS connection attempts |
| carbide_api_tls_connection_success_total | counter | Number of successful TLS connections |
| carbide_api_tracing_spans_open | gauge | Number of open logging/tracing spans |
| carbide_api_vault_request_duration_milliseconds | histogram | Duration of outbound Vault requests, in milliseconds |
| carbide_api_vault_requests_attempted_total | counter | Number of attempted Vault requests |
| carbide_api_vault_requests_failed_total | counter | Number of failed Vault requests |
| carbide_api_vault_requests_succeeded_total | counter | Number of successful Vault requests |
| carbide_api_vault_token_time_until_refresh_seconds | gauge | The amount of time, in seconds, until the Vault token is required to be refreshed |
| carbide_api_version | gauge | Version (git sha, build date, etc) of this service |
| carbide_attestation_attester_sets_total | counter | Number of previously unseen SPDM-capable attester sets recorded for a hardware class |
| carbide_attestation_profile_changes_total | counter | Number of accepted attestation profile create, update, and delete operations, by operation. |
| carbide_attestation_scheduling_total | counter | Number of SPDM attestation scheduling attempts, by outcome and which lookup supplied the profile |
| carbide_attestation_total | counter | Number of device attestations performed, by device type and outcome. |
| carbide_auth_context_missing_total | counter | Number of Forge authorization requests missing authentication context, by authorizer |
| carbide_auth_denied_total | counter | Number of Forge calls denied by the authorizer |
| carbide_auth_permissive_overrides_total | counter | Number of policy denials overridden by authorization permissive mode, by principal class |
| carbide_authn_client_cert_rejected_total | counter | Number of client certificates rejected during authentication |
| carbide_authn_connection_attributes_missing_total | counter | Number of requests authentication could not inspect because connection attributes were missing |
| carbide_available_ips_count | gauge | Number of available IPs per network segment |
| carbide_bmc_credential_rotation_results_total | counter | Number of persisted BMC credential rotation results, by result |
| carbide_bmc_proxy_admission_refused_total | counter | Number of requests the proxy refused without sending them, for want of a slot at their BMC or because their class’s breaker there was open, by request class and reason (queue_full, timeout, too_many_bmcs, breaker_open, shutting_down) |
| carbide_bmc_proxy_admission_wait_milliseconds | histogram | Time requests that got a slot at their BMC waited for it, by request class; only classes that take slots are observed, and requests refused or abandoned while waiting are not |
| carbide_bmc_proxy_authorization_denied_total | counter | Number of BMC proxy requests denied by authorization layer and HTTP method |
| carbide_bmc_proxy_authorization_errors_total | counter | Number of BMC proxy authorization errors caused by missing authentication context, by authorization layer and HTTP method |
| carbide_bmc_proxy_breaker_opened_total | counter | Number of times a request class’s circuit breaker at a BMC opened: the BMC failed too many of the class’s recent exchanges, by request class |
| carbide_bmc_proxy_redirects_total | counter | Number of BMC redirect responses observed by configured mode, response status, target classification, and proxy disposition |
| carbide_bmc_proxy_slo_missed_total | counter | Number of windows of requests in which a class with a latency target missed it at a BMC, each cutting the slots classes without a target may hold there, by request class |
| carbide_bmc_proxy_switch_probe_denied_total | counter | Number of times nico-api denied the proxy’s FindSwitchIds probe for the switch at a proxied address, after which the proxy resolves every address as a BMC’s until it probes again |
| carbide_bmc_proxy_switch_tls_certificate_out_of_validity_total | counter | Number of times a load or check of the tls.switch files found a certificate in them expired or not yet valid, by the file holding it |
| carbide_bmc_proxy_switch_tls_reload_failures_total | counter | Number of failed reloads of the tls.switch client identity and CA bundle the proxy uses for switches’ NVOS interfaces; the previously loaded ones stay in use |
| carbide_bmc_proxy_tls_connection_attempted_total | counter | Number of inbound TLS connection attempts |
| carbide_bmc_proxy_tls_connection_fail_total | counter | Number of failed inbound connections, by failure reason |
| carbide_bmc_proxy_tls_connection_success_total | counter | Number of successful TLS connections |
| carbide_bmc_proxy_tls_reload_failures_total | counter | Number of failed inbound TLS identity and trust-root reloads |
| carbide_bmc_proxy_upstream_auth_retries_total | counter | Number of forwarded requests replayed once with freshly resolved BMC credentials after the BMC rejected the proxy’s cached credential, by HTTP method |
| carbide_bmc_proxy_upstream_certificate_rejected_total | counter | Number of attempts the proxy did not send because the target’s certificate failed verification against tls.switch |
| carbide_bmc_proxy_upstream_request_duration_milliseconds | histogram | Duration of requests the proxy forwarded to BMCs, by HTTP method and upstream status class; the _count series, split by status, gives the request and outcome rates. |
| carbide_bmc_session_cleanup_failures_total | counter | Number of BMC session cleanup failures, by operation. |
| carbide_bmc_session_lockout_breaker_transitions_total | counter | Number of BMC session lockout-avoidance breaker transitions. |
| carbide_bms_dsx_commands_dropped_total | counter | Number of BMS DSX Exchange commands dropped because the worker stopped, by command kind. |
| carbide_certs_renewals_total | counter | Number of client certificate renewal attempts, by outcome |
| carbide_client_tcp_connect_attempts_total | counter | Number of outbound TCP connect attempts across all HTTP connectors |
| carbide_client_tcp_connect_errors_total | counter | Number of failed outbound TCP connect attempts across all HTTP connectors |
| carbide_client_tcp_connect_successes_total | counter | Number of successful outbound TCP connects across all HTTP connectors |
| carbide_concurrent_machine_updates_available | gauge | Number of machines in the system that can be updated concurrently. |
| carbide_config_drift_total | counter | Number of config-file seeded definitions that have drifted from their declaration, by resource_kind (resource_pool, network_definition) and drift_kind (changed, dropped). |
| carbide_database_readiness_check_failures_total | counter | Number of startup and periodic PostgreSQL readiness check failures for /ready |
| carbide_database_transaction_rollback_failures_total | counter | Number of database transaction rollback failures, by trigger. |
| carbide_db_pool_idle_conns | gauge | Number of idle connections in the carbide database pool |
| carbide_db_pool_total_conns | gauge | Number of (active + idle) connections in the carbide database pool |
| carbide_dhcp_certificate_expiration_time | gauge | The certificate expiration time (epoch seconds) |
| carbide_dhcp_certificate_expiration_time_seconds | gauge | The certificate expiration time (epoch seconds) |
| carbide_dhcp_dropped_requests_total | counter | Number of DHCPv4 packets dropped without a reply, by drop reason. |
| carbide_dhcp_replies_sent_total | counter | Number of DHCPv4 replies sent, by reply message type. |
| carbide_dhcp_requests_total | counter | Number of DHCPv4 packets received and decoded, by DHCP message type. |
| carbide_dhcp_socket_setup_failures_total | counter | Number of DHCP socket setup failures, by operation and next action. |
| carbide_dhcp_timestamp_file_failures_total | counter | Number of DHCP timestamp file failures, by operation |
| carbide_dhcp_v6_listener_unavailable_total | counter | Number of DHCPv6 listeners made unavailable by socket setup failure, by reason. |
| carbide_dhcp_v6_replies_sent_total | counter | Number of DHCPv6 replies sent, by response message type. |
| carbide_dhcp_v6_requests_dropped_total | counter | Number of DHCPv6 requests dropped or refused, by reason. |
| carbide_dhcp_v6_requests_total | counter | Number of DHCPv6 requests received, by DHCP message type. |
| carbide_dns_negative_cache_hit_count_total | counter | Number of negative DNS cache hits, by response code |
| carbide_dns_negative_cache_miss_count_total | counter | Number of negative DNS cache misses, by response code |
| carbide_dns_queries_total | counter | Number of DNS queries received, by query type |
| carbide_dns_request_duration_milliseconds | histogram | Time to process a DNS query, by query type and response code |
| carbide_dns_responses_total | counter | Number of DNS responses sent, by response code |
| carbide_dpa_monitor_iteration_latency_milliseconds | histogram | Time consumed for one monitor iteration |
| carbide_dpa_monitor_operations_latency_milliseconds | histogram | Time consumed for one operation |
| carbide_dpu_agent_report_total | counter | Number of DPU-agent report-loop iterations, by loop and outcome |
| carbide_dpu_agent_service_restart_attempts_total | counter | Number of DPU-agent service restart attempts, by service and result. |
| carbide_dpu_agent_version_count | gauge | Number of DPU agents which have reported a certain version. |
| carbide_dpu_bmc_service_credential_rotation_results_total | counter | Number of persisted DPU BMC service credential rotation results, by result |
| carbide_dpu_client_certificate_expiration_time | gauge | The expiration time (epoch seconds) for the client certificate associated with a given DPU. |
| carbide_dpu_client_certificate_expiration_time_seconds | gauge | The expiration time (epoch seconds) for the client certificate associated with a given DPU. |
| carbide_dpu_firmware_version_count | gauge | Number of DPUs which have reported a certain firmware version. |
| carbide_dpu_remediation_executor_failures_total | counter | Number of DPU remediation executor failures, by failure stage. |
| carbide_dpu_uefi_password_setup_skips_total | counter | Number of DPU UEFI password setup operations skipped, by reason. |
| carbide_dpus_healthy_count | gauge | Number of DPUs in the system that have reported healthy in the last report. Healthy does not imply up - the report from the DPU might be outdated. |
| carbide_dpus_up_count | gauge | Number of DPUs in the system that are up. Up means we have received a health report less than 5 minutes ago. |
| carbide_dsx_event_bus_publish_count_total | counter | Number of MQTT publish attempts |
| carbide_dsx_exchange_consumer_alerts_detected_total | counter | Number of leak alerts detected |
| carbide_dsx_exchange_consumer_dedup_skipped_total | counter | Number of messages skipped due to deduplication |
| carbide_dsx_exchange_consumer_health_report_persist_failures_total | counter | Number of rack health report persist failures against the Carbide API |
| carbide_dsx_exchange_consumer_message_age_seconds | histogram | Age of consumed BMS value messages at processing time (consumer lag), in seconds |
| carbide_dsx_exchange_consumer_messages_dropped_total | counter | Number of messages dropped due to queue overflow |
| carbide_dsx_exchange_consumer_messages_processed_total | counter | Number of messages successfully processed |
| carbide_dsx_exchange_consumer_messages_received_total | counter | Number of MQTT messages received |
| carbide_endpoint_exploration_duration_milliseconds | histogram | The time it took to explore an endpoint |
| carbide_endpoint_exploration_expected_machines_missing_overall_count | gauge | Number of machines expected but not identified |
| carbide_endpoint_exploration_expected_power_shelves_missing_overall_count | gauge | Number of power shelves expected but not identified |
| carbide_endpoint_exploration_identified_managed_hosts_overall_count | gauge | Number of managed hosts identified by expectation |
| carbide_endpoint_exploration_machines_explored_overall_count | gauge | Number of machines explored, by expectation and machine type |
| carbide_endpoint_exploration_step_latency_milliseconds | histogram | The time it took to perform one endpoint exploration step |
| carbide_endpoint_exploration_success_count | gauge | Number of successful endpoint explorations |
| carbide_endpoint_explorations_count | gauge | Number of attempted endpoint explorations |
| carbide_exhausted_reprovision_retry_count | gauge | Number of host machines in the system whose host firmware upgrade retry budget is exhausted. |
| carbide_extension_service_credential_cleanup_failures_total | counter | Number of extension-service credential cleanup failures, by operation. |
| carbide_extension_services_count | gauge | Number of carbide_extension_services in the system |
| carbide_extension_services_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_extension_services in the system |
| carbide_extension_services_iteration_latency_milliseconds | histogram | Elapsed time of each completed state processor iteration for objects of type carbide_extension_services |
| carbide_extension_services_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_extension_services |
| carbide_extension_services_object_tasks_running | gauge | Number of object handling tasks currently running of type carbide_extension_services |
| carbide_extension_services_total | gauge | Number of carbide_extension_services in the system |
| carbide_external_call_duration_milliseconds | histogram | Duration of outbound calls by backend, operation, and outcome; the _count series, split by outcome, gives the request and error rates. |
| carbide_firmware_artifact_unavailable_total | counter | Number of firmware artifacts unavailable before download, by reason. |
| carbide_firmware_download_duration_seconds | histogram | Duration of background firmware artifact downloads, by outcome; an ok attempt spans fetch, checksum verification, and publish, and the _count series, split by outcome, is the download and failure rate. |
| carbide_firmware_update_failures_total | counter | Number of firmware update failures, by update target and cause |
| carbide_firmware_updates_total | counter | Number of firmware updates started and completed, by update target and phase; only the host target emits both phases |
| carbide_fmds_config_updates_total | counter | Number of FMDS gRPC config-update ingests, by outcome |
| carbide_fmds_phone_home_total | counter | Number of FMDS tenant phone-home operations, by outcome |
| carbide_gpus_in_use_count | gauge | Number of GPUs actively used by tenants in instances in the NICo deployment |
| carbide_gpus_total_count | gauge | Number of GPUs in the NICo deployment |
| carbide_gpus_usable_count | gauge | Number of remaining GPUs in the NICo deployment available for immediate instance creation |
| carbide_hardware_health_component_inventory_info | gauge | Authoritative NICo component inventory for the current rack-ingestion session |
| carbide_hardware_health_hw_metric_correctable_io_read_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_correctable_io_read_errors_count_total | counter | Number of correctable read errors over the lifetime of the drive, by drive_id, storage_id, and system_id |
| carbide_hardware_health_hw_metric_correctable_io_write_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_correctable_io_write_errors_count_total | counter | Number of correctable write errors over the lifetime of the drive, by drive_id, storage_id, and system_id |
| carbide_hardware_health_hw_metric_current_correctable_ecc_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_current_correctable_ecc_errors_count_total | counter | Number of correctable memory ECC errors since the last system reset or ClearCurrentPeriod action, by memory_id and system_id |
| carbide_hardware_health_hw_metric_current_indeterminate_correctable_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_current_indeterminate_correctable_errors_count_total | counter | Number of indeterminate correctable memory errors since the last system reset or ClearCurrentPeriod action, by memory_id and system_id; the same error may appear for multiple memory resources |
| carbide_hardware_health_hw_metric_current_indeterminate_uncorrectable_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_current_indeterminate_uncorrectable_errors_count_total | counter | Number of indeterminate uncorrectable memory errors since the last system reset or ClearCurrentPeriod action, by memory_id and system_id; the same error may appear for multiple memory resources |
| carbide_hardware_health_hw_metric_current_uncorrectable_ecc_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_current_uncorrectable_ecc_errors_count_total | counter | Number of uncorrectable memory ECC errors since the last system reset or ClearCurrentPeriod action, by memory_id and system_id |
| carbide_hardware_health_hw_metric_lifetime_correctable_ecc_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_lifetime_correctable_ecc_errors_count_total | counter | Number of correctable ECC errors over the lifetime of the memory, by memory_id and system_id |
| carbide_hardware_health_hw_metric_lifetime_indeterminate_correctable_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_lifetime_indeterminate_correctable_errors_count_total | counter | Number of indeterminate correctable errors over the lifetime of the memory, by memory_id and system_id; the same error may appear for multiple memory resources |
| carbide_hardware_health_hw_metric_lifetime_indeterminate_uncorrectable_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_lifetime_indeterminate_uncorrectable_errors_count_total | counter | Number of indeterminate uncorrectable errors over the lifetime of the memory, by memory_id and system_id; the same error may appear for multiple memory resources |
| carbide_hardware_health_hw_metric_lifetime_uncorrectable_ecc_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_lifetime_uncorrectable_ecc_errors_count_total | counter | Number of uncorrectable ECC errors over the lifetime of the memory, by memory_id and system_id |
| carbide_hardware_health_hw_metric_nvidia_accumulated_gpu_context_utilization_seconds | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_nvidia_accumulated_gpu_context_utilization_seconds_total | counter | Number of seconds of GPU context utilization accumulated with the driver loaded, by processor_id and system_id; resets on SBR, fundamental reset, or ClearAccumulatedUtilizationCounters and wraps after approximately 49 days of activity |
| carbide_hardware_health_hw_metric_nvidia_accumulated_sm_utilization_seconds | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_nvidia_accumulated_sm_utilization_seconds_total | counter | Number of seconds of SM utilization accumulated with the driver loaded, by processor_id and system_id; resets on SBR, fundamental reset, or ClearAccumulatedUtilizationCounters and wraps after approximately 49 days of activity |
| carbide_hardware_health_hw_metric_nvidia_correctable_row_remapping_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_nvidia_correctable_row_remapping_count_total | counter | Number of memory row remappings due to correctable errors reported by the NVIDIA device, by memory_id and system_id |
| carbide_hardware_health_hw_metric_nvidia_global_software_violation_throttle_seconds | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_nvidia_global_software_violation_throttle_seconds_total | counter | Number of seconds the NVIDIA processor has throttled due to global software violation since reset, by processor_id and system_id |
| carbide_hardware_health_hw_metric_nvidia_hardware_violation_throttle_seconds | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_nvidia_hardware_violation_throttle_seconds_total | counter | Number of seconds the NVIDIA processor has throttled due to hardware violation since reset, by processor_id and system_id |
| carbide_hardware_health_hw_metric_nvidia_pcie_rx_bytes | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_nvidia_pcie_rx_bytes_total | counter | Number of bytes received by the NVIDIA processor PCIe receiver, by processor_id and system_id |
| carbide_hardware_health_hw_metric_nvidia_pcie_tx_bytes | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_nvidia_pcie_tx_bytes_total | counter | Number of bytes sent by the NVIDIA processor PCIe transmitter, by processor_id and system_id |
| carbide_hardware_health_hw_metric_nvidia_power_brake_assertion_seconds | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_nvidia_power_brake_assertion_seconds_total | counter | Number of seconds the NVIDIA processor power brake has been asserted since reset, by processor_id and system_id |
| carbide_hardware_health_hw_metric_nvidia_uncorrectable_row_remapping_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_nvidia_uncorrectable_row_remapping_count_total | counter | Number of memory row remappings due to uncorrectable errors reported by the NVIDIA device, by memory_id and system_id |
| carbide_hardware_health_hw_metric_pcie_bad_dllp_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_pcie_bad_dllp_count_total | counter | Number of bad PCIe data link layer packets issued on the PCIe link by the receiver |
| carbide_hardware_health_hw_metric_pcie_bad_tlp_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_pcie_bad_tlp_count_total | counter | Number of bad PCIe transaction layer packets issued on the PCIe link by the receiver |
| carbide_hardware_health_hw_metric_pcie_correctable_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_pcie_correctable_errors_count_total | counter | Number of PCIe correctable errors reported by the device |
| carbide_hardware_health_hw_metric_pcie_fatal_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_pcie_fatal_errors_count_total | counter | Number of PCIe fatal errors reported by the device |
| carbide_hardware_health_hw_metric_pcie_flow_control_timeout_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_pcie_flow_control_timeout_count_total | counter | Number of PCIe flow control credit timeouts reported by the device |
| carbide_hardware_health_hw_metric_pcie_l0_to_recovery_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_pcie_l0_to_recovery_count_total | counter | Number of PCIe link transitions from L0 to recovery for the device |
| carbide_hardware_health_hw_metric_pcie_nak_received_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_pcie_nak_received_count_total | counter | Number of PCIe negative acknowledgements received by the device |
| carbide_hardware_health_hw_metric_pcie_nak_sent_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_pcie_nak_sent_count_total | counter | Number of PCIe negative acknowledgements sent by the device |
| carbide_hardware_health_hw_metric_pcie_non_fatal_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_pcie_non_fatal_errors_count_total | counter | Number of PCIe non-fatal errors reported by the device |
| carbide_hardware_health_hw_metric_pcie_replay_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_pcie_replay_count_total | counter | Number of PCIe replays issued by the device |
| carbide_hardware_health_hw_metric_pcie_replay_rollover_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_pcie_replay_rollover_count_total | counter | Number of PCIe replay rollovers issued by the device |
| carbide_hardware_health_hw_metric_pcie_unsupported_request_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_pcie_unsupported_request_count_total | counter | Number of unsupported PCIe requests received by the device |
| carbide_hardware_health_hw_metric_power_limit_throttle_seconds | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_power_limit_throttle_seconds_total | counter | Number of seconds the processor has throttled due to a power limit since reset |
| carbide_hardware_health_hw_metric_read_io_kibibytes | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_read_io_kibibytes_total | counter | Number of kibibytes read from the drive since the last reset or wrap, by drive_id, storage_id, and system_id |
| carbide_hardware_health_hw_metric_thermal_limit_throttle_seconds | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_thermal_limit_throttle_seconds_total | counter | Number of seconds the processor has throttled due to a thermal limit since reset |
| carbide_hardware_health_hw_metric_uncorrectable_io_read_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_uncorrectable_io_read_errors_count_total | counter | Number of uncorrectable read errors over the lifetime of the drive, by drive_id, storage_id, and system_id |
| carbide_hardware_health_hw_metric_uncorrectable_io_write_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_uncorrectable_io_write_errors_count_total | counter | Number of uncorrectable write errors over the lifetime of the drive, by drive_id, storage_id, and system_id |
| carbide_hardware_health_hw_metric_write_io_kibibytes | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_hw_metric_write_io_kibibytes_total | counter | Number of kibibytes written to the drive since the last reset or wrap, by drive_id, storage_id, and system_id |
| carbide_hardware_health_inventory_last_success_time_seconds | gauge | Unix timestamp of the last successful NICo inventory reconciliation |
| carbide_hardware_health_inventory_refresh_failures_total | counter | Number of authoritative hardware inventory refreshes that failed. |
| carbide_hardware_health_nvue_gnmi_interface_in_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_in_errors_count_total | counter | Number of inbound packets with errors reported by the interface |
| carbide_hardware_health_nvue_gnmi_interface_in_octets_bytes | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_in_octets_bytes_total | counter | Number of octets received by the interface |
| carbide_hardware_health_nvue_gnmi_interface_in_packets_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_in_packets_count_total | counter | Number of packets received by the interface |
| carbide_hardware_health_nvue_gnmi_interface_link_downed_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_link_downed_count_total | counter | Number of link down events reported by the InfiniBand port |
| carbide_hardware_health_nvue_gnmi_interface_link_error_recovery_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_link_error_recovery_count_total | counter | Number of link error recovery events reported by the InfiniBand port |
| carbide_hardware_health_nvue_gnmi_interface_local_link_integrity_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_local_link_integrity_errors_count_total | counter | Number of local link integrity errors reported by the InfiniBand port |
| carbide_hardware_health_nvue_gnmi_interface_out_discards_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_out_discards_count_total | counter | Number of outbound packets discarded by the interface without detected errors |
| carbide_hardware_health_nvue_gnmi_interface_out_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_out_errors_count_total | counter | Number of outbound packets with errors reported by the interface |
| carbide_hardware_health_nvue_gnmi_interface_out_octets_bytes | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_out_octets_bytes_total | counter | Number of octets transmitted by the interface |
| carbide_hardware_health_nvue_gnmi_interface_out_packets_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_out_packets_count_total | counter | Number of packets transmitted by the interface |
| carbide_hardware_health_nvue_gnmi_interface_port_buffer_overrun_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_port_buffer_overrun_errors_count_total | counter | Number of excessive buffer overrun errors reported by the InfiniBand port |
| carbide_hardware_health_nvue_gnmi_interface_port_xmit_wait_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_port_xmit_wait_count_total | counter | Number of ticks during which the InfiniBand port had data to transmit but sent none for the entire tick |
| carbide_hardware_health_nvue_gnmi_interface_qp1_dropped_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_qp1_dropped_count_total | counter | Number of QP1 packets dropped by the InfiniBand port |
| carbide_hardware_health_nvue_gnmi_interface_rcv_constraint_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_rcv_constraint_errors_count_total | counter | Number of receive constraint errors reported by the InfiniBand port |
| carbide_hardware_health_nvue_gnmi_interface_rcv_remote_physical_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_rcv_remote_physical_errors_count_total | counter | Number of receive remote physical errors reported by the InfiniBand port |
| carbide_hardware_health_nvue_gnmi_interface_rcv_switch_relay_errors_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_rcv_switch_relay_errors_count_total | counter | Number of receive switch relay errors reported by the InfiniBand port |
| carbide_hardware_health_nvue_gnmi_interface_vl15_dropped_count | gauge | Metrics forwarded through sink pipeline |
| carbide_hardware_health_nvue_gnmi_interface_vl15_dropped_count_total | counter | Number of VL15 packets dropped by the InfiniBand port |
| carbide_hardware_health_rack_nvlink_domain_info | gauge | Authoritative NICo rack-to-NVLink-domain assignments for current rack-ingestion sessions |
| carbide_hardware_health_rack_session_start_time_seconds | gauge | NICo rack creation time in Unix seconds, labeled by its ingestion session |
| carbide_health_bmc_connection_circuit_closed_total | counter | Number of times a BMC connection circuit closed because the BMC answered again. |
| carbide_health_bmc_connection_circuit_opened_total | counter | Number of times a BMC connection circuit opened after a connect-level failure. |
| carbide_health_otlp_egress_records_total | counter | Number of OTLP log records or metric data points acknowledged by the receiver, excluding explicit partial-success rejections, by target and signal. |
| carbide_health_otlp_export_failures_total | counter | Number of OTLP export batches dropped after a send failure, by signal and gRPC status code. |
| carbide_health_otlp_ingress_records_total | counter | Number of log records or metric samples accepted into OTLP queues, including replacements and insertions that evict an older entry, by target and signal. |
| carbide_health_otlp_queue_depth | gauge | Number of entries waiting in an OTLP queue, by target and signal. |
| carbide_health_otlp_queue_dropped_total | counter | Number of OTLP queue entries dropped because a per-target queue reached capacity, by target and signal. |
| carbide_health_redfish_sse_event_record_resolution_failures_total | counter | Number of Redfish SSE event records dropped after a referenced record could not be resolved, by failure reason. |
| carbide_health_report_submissions_total | counter | Number of health report submissions to the NICo API, by report target and outcome. |
| carbide_host_reprovision_retries_total | counter | Number of times a failed host firmware upgrade was retried during host reprovisioning |
| carbide_hosts_alerts_suppressed_count | gauge | Whether external metrics based alerting is suppressed for a specific object |
| carbide_hosts_by_sku_count | gauge | Number of hosts by SKU and device type (‘unknown’ for hosts without SKU) |
| carbide_hosts_health_overrides_count | gauge | Number of health overrides configured in the site |
| carbide_hosts_health_status_count | gauge | Number of managed hosts in the system that have reported either a healthy or not healthy status - based on the presence of health probe alerts |
| carbide_hosts_in_use_count | gauge | Number of hosts actively used by tenants as instances in the NICo deployment |
| carbide_hosts_unhealthy_by_classification_count | gauge | Number of objects marked with a certain classification due to being unhealthy |
| carbide_hosts_unhealthy_by_probe_id_count | gauge | Number of objects which reported a certain Health Probe Alert |
| carbide_hosts_usable_count | gauge | Number of remaining hosts in the NICo deployment available for immediate instance creation |
| carbide_hosts_with_bios_password_set | gauge | Number of hosts in the system that have their BIOS password set. |
| carbide_hosts_with_scout_heartbeat_timeout | gauge | Scout heartbeat timeout status for hosts |
| carbide_ib_monitor_fabrics_count | gauge | Number of monitored InfiniBand fabrics |
| carbide_ib_monitor_iteration_latency_milliseconds | histogram | The time it took to perform one IB fabric monitor iteration |
| carbide_ib_monitor_machine_ib_status_updates_count | gauge | Number of Machines whose InfiniBand status observation was updated |
| carbide_ib_monitor_machines_with_missing_pkeys_count | gauge | Number of machines where at least one port is not assigned to the expected pkey on UFM |
| carbide_ib_monitor_machines_with_unexpected_pkeys_count | gauge | Number of machines where at least one port is assigned to an unexpected pkey on UFM |
| carbide_ib_monitor_machines_with_unknown_pkeys_count | gauge | Number of machines where at least one port is assigned to a pkey value not associated with any partition ID |
| carbide_ib_monitor_partial_failures_total | counter | Number of IB fabric monitor partial failures, by failure stage. |
| carbide_ib_monitor_ufm_changes_applied_total | counter | Number of changes performed at UFM |
| carbide_ib_partition_pkey_missing_total | counter | Number of IB partitions missing a pkey, by controller state |
| carbide_ib_partitions_count | gauge | Number of carbide_ib_partitions in the system |
| carbide_ib_partitions_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_ib_partitions in the system |
| carbide_ib_partitions_iteration_latency_milliseconds | histogram | Elapsed time of each completed state processor iteration for objects of type carbide_ib_partitions |
| carbide_ib_partitions_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_ib_partitions |
| carbide_ib_partitions_object_tasks_running | gauge | Number of object handling tasks currently running of type carbide_ib_partitions |
| carbide_ib_partitions_total | gauge | Number of carbide_ib_partitions in the system |
| carbide_ipmi_commands_total | counter | Number of IPMI command executions, by command and outcome. |
| carbide_kms_token_maintenance_failures_total | counter | Number of Transit KMS token maintenance failures, by maintenance stage |
| carbide_log_events_total | counter | Number of log events emitted, by level and component. The always-on log-volume and error-rate signal for every binary. |
| carbide_machine_dpu_info | gauge | Association between a host machine and one of its DPUs. Exposed only on the opt-in per-object endpoint. |
| carbide_machine_identity_sign_proxy_failures_total | counter | Number of machine identity sign proxy transport, local response handling, and upstream server failures, by failure stage. |
| carbide_machine_identity_stored_secret_decryption_failures_total | counter | Number of stored machine identity secret decryption failures, by secret kind. |
| carbide_machine_identity_token_exchange_failures_total | counter | Number of machine identity token exchange failures, by failure stage |
| carbide_machine_instance_info | gauge | Association between a host machine and its tenant instance. Exposed only on the opt-in per-object endpoint. |
| carbide_machine_reboot_duration_seconds | histogram | Time taken for machine/host to reboot in seconds |
| carbide_machine_updates_started_count | gauge | Number of machines in the system in the process of updating |
| carbide_machine_validation_completed | gauge | Number of successfully completed machine validation runs |
| carbide_machine_validation_failed | gauge | Number of failed machine validation runs |
| carbide_machine_validation_heartbeat_failures_total | counter | Number of machine validation heartbeat failures, by stage and reason. |
| carbide_machine_validation_in_progress | gauge | Number of machine validation runs in progress |
| carbide_machine_validation_oldest_active_age_seconds | gauge | Age in seconds of the oldest active machine validation run |
| carbide_machine_validation_outcomes_total | counter | Number of machine validation runs that completed as passed or failed, by outcome and failure cause; runs skipped by a disabled validation config are not counted |
| carbide_machine_validation_result_persistence_attempts_total | counter | Number of machine validation result persistence attempts, by outcome. |
| carbide_machine_validation_stale_runs_count | gauge | Number of active machine validation runs considered stale |
| carbide_machine_validation_tests | gauge | The details of machine validation tests |
| carbide_machines_count | gauge | Number of carbide_machines in the system |
| carbide_machines_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_machines in the system |
| carbide_machines_handler_latency_in_state_milliseconds | histogram | The amount of time it took to invoke the state handler for objects of type carbide_machines in a certain state |
| carbide_machines_in_maintenance_count | gauge | Number of machines in the system in maintenance |
| carbide_machines_iteration_latency_milliseconds | histogram | Elapsed time of each completed state processor iteration for objects of type carbide_machines |
| carbide_machines_object_tasks_completed_total | counter | Number of object handling tasks completed for objects of type carbide_machines |
| carbide_machines_object_tasks_dispatched_total | counter | Number of object handling tasks dequeued and dispatched for processing for objects of type carbide_machines |
| carbide_machines_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_machines |
| carbide_machines_object_tasks_errored_total | counter | Number of object handling tasks completed with an error for objects of type carbide_machines |
| carbide_machines_object_tasks_requeued_total | counter | Number of object handling tasks requeued for objects of type carbide_machines |
| carbide_machines_object_tasks_running | gauge | Number of object handling tasks currently running of type carbide_machines |
| carbide_machines_per_state | gauge | Number of carbide_machines in the system with a given state |
| carbide_machines_per_state_above_sla | gauge | Number of carbide_machines currently in a state longer than the SLA allows |
| carbide_machines_state_entered_total | counter | Number of times objects of type carbide_machines have entered a certain state |
| carbide_machines_state_exited_total | counter | Number of times objects of type carbide_machines have exited a certain state |
| carbide_machines_time_in_state_seconds | histogram | The amount of time objects of type carbide_machines have spent in a certain state |
| carbide_machines_total | gauge | Number of carbide_machines in the system |
| carbide_machines_with_state_handling_errors_per_state | gauge | Number of carbide_machines whose most recent state handler run returned an error, by state, substate, and error; the error=any series is the total across all errors for that state and substate, so exclude it when summing over error |
| carbide_managed_loop_iterations_total | counter | Number of managed loop iterations, by manager and outcome; the measured boot metrics collector’s iterations are counted by its latency histogram instead |
| carbide_measured_boot_bundles_count | gauge | Number of measured boot bundles. |
| carbide_measured_boot_bundles_per_pcr_value_count | gauge | Number of measured boot bundles containing a given PCR value at a given PCR index. |
| carbide_measured_boot_bundles_total | gauge | Number of measured boot bundles. |
| carbide_measured_boot_collector_iteration_latency_milliseconds | histogram | Number of milliseconds a full measured boot metrics collector iteration took, by outcome |
| carbide_measured_boot_machines_count | gauge | Number of discovered candidate machines, including machines without measurements. |
| carbide_measured_boot_machines_per_bundle_count | gauge | Number of machines per measured boot bundle. |
| carbide_measured_boot_machines_per_bundle_state_count | gauge | Number of machines per measured boot bundle state. |
| carbide_measured_boot_machines_per_bundle_state_total | gauge | Number of machines per measured boot bundle state. |
| carbide_measured_boot_machines_per_bundle_total | gauge | Number of machines per measured boot bundle. |
| carbide_measured_boot_machines_per_machine_state_count | gauge | Number of machines per measured boot machine state. |
| carbide_measured_boot_machines_per_machine_state_total | gauge | Number of machines per measured boot machine state. |
| carbide_measured_boot_machines_per_pcr_value_total | gauge | Number of machines with a given PCR value at a given PCR index. |
| carbide_measured_boot_machines_per_profile_count | gauge | Number of machines per measured boot system profile. |
| carbide_measured_boot_machines_per_profile_total | gauge | Number of machines per measured boot system profile. |
| carbide_measured_boot_machines_total | gauge | Number of machines reporting measurements. |
| carbide_measured_boot_profiles_count | gauge | Number of measured boot profiles. |
| carbide_measured_boot_profiles_total | gauge | Number of measured boot profiles. |
| carbide_measured_boot_verification_failures_total | counter | Number of measured boot verification failures, across quote verification and attestation handling, by cause |
| carbide_metrics_scrape_failures_total | counter | Number of /metrics scrape failures, by outcome. |
| carbide_mqtt_connected | gauge | Number of active broker connections held by the MQTT client (1 while connected, 0 otherwise) |
| carbide_mqtt_dispatch_dropped_total | counter | Number of received MQTT messages dropped because the handler concurrency semaphore could not be acquired |
| carbide_mqtt_dropped_bytes_total | counter | Number of bytes of received MQTT messages dropped because the client’s processing queue was full |
| carbide_mqtt_event_loop_errors_total | counter | Number of connection errors encountered by the MQTT client’s event loop |
| carbide_mqtt_messages_dropped_total | counter | Number of received MQTT messages dropped because the client’s processing queue was full |
| carbide_mqtt_messages_failed_total | counter | Number of received MQTT messages whose handler failed or that had no handler registered |
| carbide_mqtt_messages_processed_total | counter | Number of received MQTT messages successfully processed by a registered handler |
| carbide_mqtt_messages_published_total | counter | Number of MQTT messages successfully queued for publishing to the broker |
| carbide_mqtt_processed_bytes_total | counter | Number of bytes of received MQTT messages successfully processed by a registered handler |
| carbide_mqtt_publish_failures_total | counter | Number of failed MQTT message publish attempts |
| carbide_mqtt_published_bytes_total | counter | Number of bytes of MQTT messages successfully queued for publishing to the broker |
| carbide_mqtt_queue_pending_bytes | gauge | Number of bytes of received MQTT messages currently waiting in the client’s processing queue |
| carbide_mqtt_queue_pending_messages | gauge | Number of received MQTT messages currently waiting in the client’s processing queue |
| carbide_mqtt_reconnects_total | counter | Number of times an MQTT client re-established its broker connection after the initial connect |
| carbide_mqtt_unmatched_topics_total | counter | Number of received MQTT messages whose topic matched no registered handler pattern |
| carbide_network_segments_count | gauge | Number of carbide_network_segments in the system |
| carbide_network_segments_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_network_segments in the system |
| carbide_network_segments_handler_latency_in_state_milliseconds | histogram | The amount of time it took to invoke the state handler for objects of type carbide_network_segments in a certain state |
| carbide_network_segments_iteration_latency_milliseconds | histogram | Elapsed time of each completed state processor iteration for objects of type carbide_network_segments |
| carbide_network_segments_object_tasks_completed_total | counter | Number of object handling tasks completed for objects of type carbide_network_segments |
| carbide_network_segments_object_tasks_dispatched_total | counter | Number of object handling tasks dequeued and dispatched for processing for objects of type carbide_network_segments |
| carbide_network_segments_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_network_segments |
| carbide_network_segments_object_tasks_requeued_total | counter | Number of object handling tasks requeued for objects of type carbide_network_segments |
| carbide_network_segments_object_tasks_running | gauge | Number of object handling tasks currently running of type carbide_network_segments |
| carbide_network_segments_per_state | gauge | Number of carbide_network_segments in the system with a given state |
| carbide_network_segments_per_state_above_sla | gauge | Number of carbide_network_segments currently in a state longer than the SLA allows |
| carbide_network_segments_state_entered_total | counter | Number of times objects of type carbide_network_segments have entered a certain state |
| carbide_network_segments_state_exited_total | counter | Number of times objects of type carbide_network_segments have exited a certain state |
| carbide_network_segments_time_in_state_seconds | histogram | The amount of time objects of type carbide_network_segments have spent in a certain state |
| carbide_network_segments_total | gauge | Number of carbide_network_segments in the system |
| carbide_network_segments_with_state_handling_errors_per_state | gauge | Number of carbide_network_segments whose most recent state handler run returned an error, by state, substate, and error; the error=any series is the total across all errors for that state and substate, so exclude it when summing over error |
| carbide_nvlink_partition_monitor_groups_total | counter | Number of machine groups (chassis or rack) the partition monitor finished, by outcome |
| carbide_nvlink_partition_monitor_iteration_latency_milliseconds | histogram | Time consumed for one monitor iteration |
| carbide_nvlink_partition_monitor_iteration_overruns_total | counter | Number of partition monitor iterations that ran longer than monitor_run_interval |
| carbide_nvlink_partition_monitor_machine_status_updates_count | gauge | Number of machines whose NVLink status observation was updated |
| carbide_nvlink_partition_monitor_nmx_c_unreachable_chassis_count | gauge | Number of machine groups (chassis or rack) where NMX-C was unreachable during partition monitor iteration |
| carbide_nvlink_partition_monitor_nmxc_changes_applied_total | counter | Number of changes requested to NMX-C |
| carbide_nvlink_partition_monitor_nmxc_op_latency_milliseconds | histogram | Time consumed for one NMX-C operation |
| carbide_nvlink_partition_monitor_num_logical_partitions | gauge | Number of monitored logical partitions |
| carbide_nvlink_partition_monitor_num_physical_partitions | gauge | Number of monitored physical partitions |
| carbide_nvlink_partition_monitor_nvlink_info_mismatches | gauge | Number of NVLink GPU partition ID mismatches between DB and NMX-C |
| carbide_nvlink_partition_monitor_stage_latency_milliseconds | histogram | Time consumed by one stage of a partition monitor iteration, by stage |
| carbide_nvlink_partition_monitor_stale_partitions_deleted | gauge | Number of stale partitions deleted from DB (not found in NMX-C) |
| carbide_nvlink_partition_monitor_unhealthy_alert_write_failures_total | counter | Number of NvlinkUnhealthy health-report writes that failed and were deferred to the next pass |
| carbide_nvlink_switch_cert_monitor_iteration_latency_milliseconds | histogram | Time consumed for one NMX-C switch certificate monitor iteration |
| carbide_object_info | gauge | Stable identifying traits for a state-controller object. Exposed only on the opt-in per-object endpoint. |
| carbide_object_manual_intervention_required | gauge | Indicates that an object currently requires manual operator intervention. Exposed only on the opt-in per-object endpoint. |
| carbide_object_state_entered_timestamp_seconds | gauge | Unix timestamp at which an object entered its current state. Exposed only on the opt-in per-object endpoint. |
| carbide_object_state_sla_seconds | gauge | Resolved SLA in seconds for an objects current state. Exposed only on the opt-in per-object endpoint. |
| carbide_pending_dpu_nic_firmware_update_count | gauge | Number of machines in the system that need a DPU/NIC firmware update |
| carbide_pending_host_firmware_update_count | gauge | Number of host machines in the system that need a firmware update. |
| carbide_power_shelves_count | gauge | Number of carbide_power_shelves in the system |
| carbide_power_shelves_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_power_shelves in the system |
| carbide_power_shelves_handler_latency_in_state_milliseconds | histogram | The amount of time it took to invoke the state handler for objects of type carbide_power_shelves in a certain state |
| carbide_power_shelves_health_overrides_count | gauge | Number of health overrides configured in the site |
| carbide_power_shelves_health_status_count | gauge | Number of power shelves in the system that have reported either a healthy or not healthy status - based on the presence of health probe alerts |
| carbide_power_shelves_iteration_latency_milliseconds | histogram | Elapsed time of each completed state processor iteration for objects of type carbide_power_shelves |
| carbide_power_shelves_object_tasks_completed_total | counter | Number of object handling tasks completed for objects of type carbide_power_shelves |
| carbide_power_shelves_object_tasks_dispatched_total | counter | Number of object handling tasks dequeued and dispatched for processing for objects of type carbide_power_shelves |
| carbide_power_shelves_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_power_shelves |
| carbide_power_shelves_object_tasks_requeued_total | counter | Number of object handling tasks requeued for objects of type carbide_power_shelves |
| carbide_power_shelves_object_tasks_running | gauge | Number of object handling tasks currently running of type carbide_power_shelves |
| carbide_power_shelves_per_state | gauge | Number of carbide_power_shelves in the system with a given state |
| carbide_power_shelves_per_state_above_sla | gauge | Number of carbide_power_shelves currently in a state longer than the SLA allows |
| carbide_power_shelves_state_entered_total | counter | Number of times objects of type carbide_power_shelves have entered a certain state |
| carbide_power_shelves_state_exited_total | counter | Number of times objects of type carbide_power_shelves have exited a certain state |
| carbide_power_shelves_time_in_state_seconds | histogram | The amount of time objects of type carbide_power_shelves have spent in a certain state |
| carbide_power_shelves_total | gauge | Number of carbide_power_shelves in the system |
| carbide_power_shelves_with_state_handling_errors_per_state | gauge | Number of carbide_power_shelves whose most recent state handler run returned an error, by state, substate, and error; the error=any series is the total across all errors for that state and substate, so exclude it when summing over error |
| carbide_preingestion_bfb_copy_duration_seconds | histogram | Duration of preingestion BFB copies to a DPU rshim, by outcome; the _count series, split by outcome, is the copy and failure rate. |
| carbide_preingestion_count | gauge | Number of known machines currently being evaluated prior to ingestion |
| carbide_preingestion_firmware_upgrade_tasks_total | counter | Number of preingestion firmware upgrade Redfish tasks reaching a terminal state, by firmware component, final task state, and outcome. |
| carbide_preingestion_firmware_upload_total | counter | Number of preingestion firmware uploads to a BMC, by upload method and outcome. |
| carbide_preingestion_power_control_total | counter | Number of preingestion Redfish power operations (host power control, BMC and chassis resets), by operation and outcome. |
| carbide_preingestion_total | gauge | Number of known machines currently being evaluated prior to ingestion |
| carbide_preingestion_waiting_download | gauge | Number of machines that are waiting for firmware downloads on other machines to complete before doing their own |
| carbide_preingestion_waiting_installation | gauge | Number of machines which have had firmware uploaded to them and are currently in the process of installing that firmware |
| carbide_pxe_boot_outcomes_total | counter | Number of PXE boot-path outcomes served, by endpoint and reason. |
| carbide_rack_maintenance_access_token_cleanup_failures_total | counter | Number of rack maintenance access token cleanup failures |
| carbide_racks_count | gauge | Number of carbide_racks in the system |
| carbide_racks_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_racks in the system |
| carbide_racks_handler_latency_in_state_milliseconds | histogram | The amount of time it took to invoke the state handler for objects of type carbide_racks in a certain state |
| carbide_racks_health_overrides_count | gauge | Number of health overrides configured in the site |
| carbide_racks_health_status_count | gauge | Number of racks in the system that have reported either a healthy or not healthy status - based on the presence of health probe alerts |
| carbide_racks_iteration_latency_milliseconds | histogram | Elapsed time of each completed state processor iteration for objects of type carbide_racks |
| carbide_racks_object_tasks_completed_total | counter | Number of object handling tasks completed for objects of type carbide_racks |
| carbide_racks_object_tasks_dispatched_total | counter | Number of object handling tasks dequeued and dispatched for processing for objects of type carbide_racks |
| carbide_racks_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_racks |
| carbide_racks_object_tasks_requeued_total | counter | Number of object handling tasks requeued for objects of type carbide_racks |
| carbide_racks_object_tasks_running | gauge | Number of object handling tasks currently running of type carbide_racks |
| carbide_racks_per_state | gauge | Number of carbide_racks in the system with a given state |
| carbide_racks_per_state_above_sla | gauge | Number of carbide_racks currently in a state longer than the SLA allows |
| carbide_racks_total | gauge | Number of carbide_racks in the system |
| carbide_racks_with_state_handling_errors_per_state | gauge | Number of carbide_racks whose most recent state handler run returned an error, by state, substate, and error; the error=any series is the total across all errors for that state and substate, so exclude it when summing over error |
| carbide_reboot_attempts_in_booting_with_discovery_image | histogram | Reboot attempts per machine in BootingWithDiscoveryImage, recorded when a machine is rebooted again after no response from the host. |
| carbide_redfish_action_result_persistence_failures_total | counter | Number of detached Redfish action results that failed to persist |
| carbide_reserved_ips_count | gauge | Number of reserved IPs per network segment |
| carbide_resource_pool_cleanup_pending | gauge | Number of remote allocations awaiting confirmed cancellation or release |
| carbide_resource_pool_lifecycle_failures_total | counter | Number of resource pool lifecycle failures, by operation, failure, failure policy, allocation mode, and value type. |
| carbide_resourcepool_free_count | gauge | Number of values in the resource pool currently available for allocation |
| carbide_resourcepool_stats_available | gauge | Whether the last resource pool statistics poll returned known counts, regardless of age |
| carbide_resourcepool_stats_fresh | gauge | Whether the last successful resource pool statistics poll is less than 120 seconds old; zero before any success. Used/free count series are absent before success, after a failed poll, or when this gauge is zero |
| carbide_resourcepool_stats_last_success_timestamp_seconds | gauge | Unix timestamp of the last successful resource pool statistics poll, retained after failure and absent before any success |
| carbide_resourcepool_stats_poll_failures_total | counter | Number of failed resource pool observations: one per unavailable or unsupported pool, or per failed database snapshot. Labels: backend (integrated for the local snapshot, grpc for a remote pool), failure (database or timeout for snapshots, unknown for unsupported statistics, unavailable for other pool failures) |
| carbide_resourcepool_used_count | gauge | Number of currently allocated values in the resource pool |
| carbide_rms_switch_certificate_unrecognized_job_states_total | counter | Number of unrecognized RMS switch certificate job states. |
| carbide_running_dpu_updates_count | gauge | Number of machines in the system that are currently running a DPU/NIC firmware update |
| carbide_scout_actions_total | counter | Number of scout control-loop actions handled, by action and outcome. |
| carbide_scout_firmware_download_attempt_failures_total | counter | Number of failed Scout firmware download attempts, by download kind and next action. |
| carbide_scout_mlx_failures_total | counter | Number of Scout MLX observation, read, mutation, and recovery failures, by operation and failure stage. |
| carbide_scout_pci_evaluations_total | counter | Number of comparisons between scout PCI address ordering and stored boot interface selections, by result. |
| carbide_scout_storage_device_cleanup_duration_seconds | histogram | Duration of per-device scout storage cleanup operations, by device type and outcome. |
| carbide_scout_stream_connections_total | counter | Number of scout stream connection attempts, by outcome. |
| carbide_scout_stream_reconnects_total | counter | Number of scout stream reconnect cycles after a stream closed or errored. |
| carbide_scout_stream_responses_dropped_total | counter | Number of scout stream responses dropped after the outbound request stream closed. |
| carbide_site_exploration_expected_machines_sku_count | gauge | Number of expected machines by SKU ID and device type |
| carbide_site_exploration_identified_managed_hosts_count | gauge | Number of Host+DPU pairs identified in the last SiteExplorer run |
| carbide_site_explorer_bmc_password_rotations_total | counter | Number of BMC root password rotations onto the site-wide credential, by outcome |
| carbide_site_explorer_bmc_reset_attempts_total | counter | Number of Site Explorer BMC reset attempts, by method and status. |
| carbide_site_explorer_bmc_reset_count | gauge | Number of successful BMC resets in the last SiteExplorer run |
| carbide_site_explorer_bmc_reset_timestamp_persistence_failures_total | counter | Number of Site Explorer BMC reset timestamp persistence failures, by method. |
| carbide_site_explorer_boot_interface_selections_total | counter | Number of successful Site Explorer boot interface selections for hosts with more than one explored DPU and effective DPU policy Manage, by selection mechanism. |
| carbide_site_explorer_create_machines | gauge | Whether site-explorer machine creation is enabled (1) or disabled (0) |
| carbide_site_explorer_create_machines_latency_milliseconds | histogram | The time it took to perform create_machines inside site-explorer |
| carbide_site_explorer_created_machines_count | gauge | Number of machine pairs created by Site Explorer after identification in the last SiteExplorer run |
| carbide_site_explorer_created_power_shelves_count | gauge | Number of power shelves created by Site Explorer after identification in the last SiteExplorer run |
| carbide_site_explorer_dpf_explorations_deferred_total | counter | Number of periodic endpoint candidates skipped, re-exploration requests rejected, or refresh requests deferred during DPF provisioning, by request type. |
| carbide_site_explorer_dpu_migration_signals_count | gauge | Number of DPU NIC-mode migration signals by signal type — mode-mismatch found, set_nic_mode issued, reset requested, and zero-DPU registered for a host whose DPU policy is nic. |
| carbide_site_explorer_enabled | gauge | Whether site-explorer is enabled (1) or paused (0) |
| carbide_site_explorer_host_admin_address_reconcile_failures_total | counter | Number of host admin address reconciliations that failed after Site Explorer committed or revisited a managed host |
| carbide_site_explorer_iteration_latency_milliseconds | histogram | The time it took to perform one site explorer iteration |
| carbide_site_explorer_last_run_status | gauge | The status of the latest Site Explorer run |
| carbide_site_explorer_machine_slot_tray_enrichment_failures_total | counter | Number of Site Explorer machine slot and tray enrichment failures, by failure stage. |
| carbide_site_explorer_managed_host_creation_latency_milliseconds | histogram | Time one Site Explorer managed host creation attempt took, from admission to the end of the after-commit repair, by outcome; the _count series, split by outcome, is the per-host creation and failure rate. |
| carbide_site_explorer_managed_host_creation_stage_latency_milliseconds | histogram | Time one stage of a Site Explorer managed host creation attempt took, by stage: the admission permit wait, the creation transaction, and the after-commit admin address repair. |
| carbide_site_explorer_phase_latency_milliseconds | histogram | The time it took to perform one site explorer iteration phase |
| carbide_site_explorer_rack_host_fill_count | gauge | Racks with expected hosts before the last create_machines phase, by kind: expected, all_hosts_ingested (every expected host has a machine), and completable_this_run (every expected host still without a machine was identified for creation in that run) |
| carbide_site_explorer_update_explored_endpoints_count | gauge | Counts from the last update_explored_endpoints phase by kind |
| carbide_site_prefix_admission_total | counter | Number of completed tenant SitePrefix admission attempts, by result. |
| carbide_site_prefix_retirements_total | counter | Number of tenant SitePrefix retirements, by previous lifecycle state. |
| carbide_spdm_attestation_count | gauge | Number of carbide_spdm_attestation in the system |
| carbide_spdm_attestation_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_spdm_attestation in the system |
| carbide_spdm_attestation_iteration_latency_milliseconds | histogram | Elapsed time of each completed state processor iteration for objects of type carbide_spdm_attestation |
| carbide_spdm_attestation_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_spdm_attestation |
| carbide_spdm_attestation_object_tasks_running | gauge | Number of object handling tasks currently running of type carbide_spdm_attestation |
| carbide_spdm_attestation_total | gauge | Number of carbide_spdm_attestation in the system |
| carbide_spdm_evidence_collection_unexpected_task_states_total | counter | Number of unexpected SPDM evidence collection task states, by task state and next action. |
| carbide_ssh_console_bmc_recovery_attempts_total | counter | Number of BMC connection recovery iterations started during this service lifetime, excluding each client’s initial connection |
| carbide_ssh_console_stream_lines_dropped_total | counter | Number of console lines omitted from client-specific streams, by reason |
| carbide_state_handler_wakeup_failures_total | counter | Number of times a machine’s state handler could not be woken after an observed or desired state change |
| carbide_static_address_assignments_total | counter | Number of static address assignment attempts, by outcome. |
| carbide_static_address_preallocations_total | counter | Number of static address preallocation outcomes recorded, by outcome; successful outcomes are counted only after commit. |
| carbide_static_address_removals_total | counter | Number of static address removal attempts, by outcome. |
| carbide_static_credential_watcher_failures_total | counter | Number of static credential watcher failures, by operation. |
| carbide_switch_slot_tray_enrichment_failures_total | counter | Number of switch slot and tray enrichment failures, by failure stage. |
| carbide_switches_count | gauge | Number of carbide_switches in the system |
| carbide_switches_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_switches in the system |
| carbide_switches_handler_latency_in_state_milliseconds | histogram | The amount of time it took to invoke the state handler for objects of type carbide_switches in a certain state |
| carbide_switches_health_overrides_count | gauge | Number of health overrides configured in the site |
| carbide_switches_health_status_count | gauge | Number of switches in the system that have reported either a healthy or not healthy status - based on the presence of health probe alerts |
| carbide_switches_iteration_latency_milliseconds | histogram | Elapsed time of each completed state processor iteration for objects of type carbide_switches |
| carbide_switches_object_tasks_completed_total | counter | Number of object handling tasks completed for objects of type carbide_switches |
| carbide_switches_object_tasks_dispatched_total | counter | Number of object handling tasks dequeued and dispatched for processing for objects of type carbide_switches |
| carbide_switches_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_switches |
| carbide_switches_object_tasks_requeued_total | counter | Number of object handling tasks requeued for objects of type carbide_switches |
| carbide_switches_object_tasks_running | gauge | Number of object handling tasks currently running of type carbide_switches |
| carbide_switches_per_state | gauge | Number of carbide_switches in the system with a given state |
| carbide_switches_per_state_above_sla | gauge | Number of carbide_switches currently in a state longer than the SLA allows |
| carbide_switches_state_entered_total | counter | Number of times objects of type carbide_switches have entered a certain state |
| carbide_switches_state_exited_total | counter | Number of times objects of type carbide_switches have exited a certain state |
| carbide_switches_time_in_state_seconds | histogram | The amount of time objects of type carbide_switches have spent in a certain state |
| carbide_switches_total | gauge | Number of carbide_switches in the system |
| carbide_switches_with_state_handling_errors_per_state | gauge | Number of carbide_switches whose most recent state handler run returned an error, by state, substate, and error; the error=any series is the total across all errors for that state and substate, so exclude it when summing over error |
| carbide_tokio_runtime_workers | gauge | Number of worker threads in the Tokio runtime |
| carbide_tokio_worker_busy_seconds_total | counter | Number of seconds each Tokio runtime worker thread has spent running tasks |
| carbide_total_ips_count | gauge | Number of IPs per network segment |
| carbide_unavailable_dpu_nic_firmware_update_count | gauge | Number of machines in the system that need a DPU/NIC firmware update but are unavailable for update |
| carbide_vpc_prefixes_count | gauge | Number of carbide_vpc_prefixes in the system |
| carbide_vpc_prefixes_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_vpc_prefixes in the system |
| carbide_vpc_prefixes_handler_latency_in_state_milliseconds | histogram | The amount of time it took to invoke the state handler for objects of type carbide_vpc_prefixes in a certain state |
| carbide_vpc_prefixes_iteration_latency_milliseconds | histogram | Elapsed time of each completed state processor iteration for objects of type carbide_vpc_prefixes |
| carbide_vpc_prefixes_object_tasks_completed_total | counter | Number of object handling tasks completed for objects of type carbide_vpc_prefixes |
| carbide_vpc_prefixes_object_tasks_dispatched_total | counter | Number of object handling tasks dequeued and dispatched for processing for objects of type carbide_vpc_prefixes |
| carbide_vpc_prefixes_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_vpc_prefixes |
| carbide_vpc_prefixes_object_tasks_requeued_total | counter | Number of object handling tasks requeued for objects of type carbide_vpc_prefixes |
| carbide_vpc_prefixes_object_tasks_running | gauge | Number of object handling tasks currently running of type carbide_vpc_prefixes |
| carbide_vpc_prefixes_per_state | gauge | Number of carbide_vpc_prefixes in the system with a given state |
| carbide_vpc_prefixes_per_state_above_sla | gauge | Number of carbide_vpc_prefixes currently in a state longer than the SLA allows |
| carbide_vpc_prefixes_state_entered_total | counter | Number of times objects of type carbide_vpc_prefixes have entered a certain state |
| carbide_vpc_prefixes_state_exited_total | counter | Number of times objects of type carbide_vpc_prefixes have exited a certain state |
| carbide_vpc_prefixes_time_in_state_seconds | histogram | The amount of time objects of type carbide_vpc_prefixes have spent in a certain state |
| carbide_vpc_prefixes_total | gauge | Number of carbide_vpc_prefixes in the system |
| carbide_vpc_prefixes_with_state_handling_errors_per_state | gauge | Number of carbide_vpc_prefixes whose most recent state handler run returned an error, by state, substate, and error; the error=any series is the total across all errors for that state and substate, so exclude it when summing over error |
| carbide_work_lock_failures_total | counter | Number of work-lock lifecycle failures, by operation and failure kind. |
| site_explorer_create_power_shelves_latency_seconds | histogram | Duration of power shelf creation |
| site_explorer_create_switches_latency_seconds | histogram | Duration of switch creation |