NVIDIA Infra Controller (NICo) Core Metrics
NVIDIA Infra Controller (NICo) Core Metrics
This file contains a list of metrics exported by NVIDIA Infra Controller (NICo). The list is auto-generated from an integration test (test_integration). Metrics no test exercises are added with cargo xtask check-metric-docs --fix. NVLink partition monitor’s metrics are documented in the manual: NVLink Partitioning.
| Name | Type | Description |
| carbide_active_host_firmware_update_count | gauge | Number of host machines in the system currently working on updating their firmware. |
| carbide_api_db_queries_total | counter | Number of database queries that occurred inside a span |
| carbide_api_db_span_query_time_milliseconds | histogram | Total time the request spent inside a span on database transactions |
| carbide_api_grpc_server_duration_milliseconds | histogram | Processing time for a request on the carbide API server |
| carbide_api_ready | gauge | Whether the NICo API is running |
| carbide_api_secrets_request_duration_milliseconds | histogram | Duration of Postgres secrets operations, in milliseconds. |
| carbide_api_secrets_requests_failed_total | counter | Number of failed Postgres secrets operations. |
| carbide_api_secrets_requests_succeeded_total | counter | Number of successful Postgres secrets operations. |
| carbide_api_secrets_requests_total | counter | Number of Postgres secrets operations attempted. |
| carbide_api_tls_cert_refreshes_total | counter | Number of TLS acceptor refreshes performed by the API listener |
| carbide_api_tls_connection_attempted_total | counter | Number of inbound TLS connection attempts |
| carbide_api_tls_connection_fail_total | counter | Number of failed inbound TLS connection attempts |
| carbide_api_tls_connection_success_total | counter | Number of successful TLS connections |
| carbide_api_tracing_spans_open | gauge | Number of open logging/tracing spans |
| carbide_api_vault_request_duration_milliseconds | histogram | Duration of outbound Vault requests, in milliseconds |
| carbide_api_vault_requests_attempted_total | counter | Number of attempted Vault requests |
| carbide_api_vault_requests_failed_total | counter | Number of failed Vault requests |
| carbide_api_vault_requests_succeeded_total | counter | Number of successful Vault requests |
| carbide_api_vault_token_time_until_refresh_seconds | gauge | The amount of time, in seconds, until the Vault token is required to be refreshed |
| carbide_api_version | gauge | Version (git sha, build date, etc) of this service |
| carbide_attestation_total | counter | Number of device attestations performed, by device type and outcome. |
| carbide_auth_context_missing_total | counter | Number of Forge authorization requests missing authentication context, by authorizer |
| carbide_auth_denied_total | counter | Number of Forge calls denied by the authorizer |
| carbide_auth_permissive_overrides_total | counter | Number of policy denials overridden by authorization permissive mode, by principal class |
| carbide_authn_client_cert_rejected_total | counter | Number of client certificates rejected during authentication |
| carbide_authn_connection_attributes_missing_total | counter | Number of requests authentication could not inspect because connection attributes were missing |
| carbide_available_ips_count | gauge | Number of available IPs per network segment |
| carbide_bmc_proxy_authorization_denied_total | counter | Number of BMC proxy requests denied by authorization layer and HTTP method |
| carbide_bmc_proxy_authorization_errors_total | counter | Number of BMC proxy authorization errors caused by missing authentication context, by authorization layer and HTTP method |
| carbide_bmc_proxy_tls_connection_attempted_total | counter | Number of inbound TLS connection attempts |
| carbide_bmc_proxy_tls_connection_fail_total | counter | Number of failed inbound connections, by failure reason |
| carbide_bmc_proxy_tls_connection_success_total | counter | Number of successful TLS connections |
| carbide_bmc_proxy_upstream_request_duration_milliseconds | histogram | Duration of requests the proxy forwarded to BMCs, by HTTP method and upstream status class; the _count series, split by status, gives the request and outcome rates. |
| carbide_bmc_session_lockout_breaker_transitions_total | counter | Number of BMC session lockout-avoidance breaker transitions. |
| carbide_certs_renewals_total | counter | Number of client certificate renewal attempts, by outcome |
| carbide_client_tcp_connect_attempts_total | counter | Number of outbound TCP connect attempts across all HTTP connectors |
| carbide_client_tcp_connect_errors_total | counter | Number of failed outbound TCP connect attempts across all HTTP connectors |
| carbide_client_tcp_connect_successes_total | counter | Number of successful outbound TCP connects across all HTTP connectors |
| carbide_concurrent_machine_updates_available | gauge | Number of machines in the system that can be updated concurrently. |
| carbide_db_pool_idle_conns | gauge | Number of idle connections in the carbide database pool |
| carbide_db_pool_total_conns | gauge | Number of (active + idle) connections in the carbide database pool |
| carbide_dhcp_dropped_requests_total | counter | Number of DHCP packets dropped without a reply, by drop reason. |
| carbide_dhcp_replies_sent_total | counter | Number of DHCP replies sent, by reply message type. |
| carbide_dhcp_requests_total | counter | Number of DHCP packets received and decoded, by DHCP message type. |
| carbide_dhcp_timestamp_file_failures_total | counter | Number of DHCP timestamp file failures, by operation |
| carbide_dhcp_v6_replies_sent_total | counter | Number of DHCPv6 replies sent, by response message type. |
| carbide_dns_negative_cache_hit_count_total | counter | Number of negative DNS cache hits, by response code |
| carbide_dns_negative_cache_miss_count_total | counter | Number of negative DNS cache misses, by response code |
| carbide_dns_queries_total | counter | Number of DNS queries received, by query type |
| carbide_dns_request_duration_milliseconds | histogram | Time to process a DNS query, by query type and response code |
| carbide_dns_responses_total | counter | Number of DNS responses sent, by response code |
| carbide_dpa_monitor_iteration_latency_milliseconds | histogram | Time consumed for one monitor iteration |
| carbide_dpa_monitor_operations_latency_milliseconds | histogram | Time consumed for one operation |
| carbide_dpu_agent_report_total | counter | Number of DPU-agent report-loop iterations, by loop and outcome |
| carbide_dpu_agent_version_count | gauge | Number of DPU agents which have reported a certain version. |
| carbide_dpu_firmware_version_count | gauge | Number of DPUs which have reported a certain firmware version. |
| carbide_dpu_remediation_executor_failures_total | counter | Number of DPU remediation executor failures, by failure stage. |
| carbide_dpus_healthy_count | gauge | Number of DPUs in the system that have reported healthy in the last report. Healthy does not imply up - the report from the DPU might be outdated. |
| carbide_dpus_up_count | gauge | Number of DPUs in the system that are up. Up means we have received a health report less than 5 minutes ago. |
| carbide_dropped_v6_requests_total | counter | Number of dropped DHCPv6 requests, by reason. |
| carbide_dsx_event_bus_publish_count_total | counter | Number of MQTT publish attempts |
| carbide_dsx_exchange_consumer_alerts_detected_total | counter | Number of leak alerts detected |
| carbide_dsx_exchange_consumer_dedup_skipped_total | counter | Number of messages skipped due to deduplication |
| carbide_dsx_exchange_consumer_health_report_persist_failures_total | counter | Number of rack health report persist failures against the Carbide API |
| carbide_dsx_exchange_consumer_message_age_seconds | histogram | Age of consumed BMS value messages at processing time (consumer lag), in seconds |
| carbide_dsx_exchange_consumer_messages_dropped_total | counter | Number of messages dropped due to queue overflow |
| carbide_dsx_exchange_consumer_messages_processed_total | counter | Number of messages successfully processed |
| carbide_dsx_exchange_consumer_messages_received_total | counter | Number of MQTT messages received |
| carbide_endpoint_exploration_duration_milliseconds | histogram | The time it took to explore an endpoint |
| carbide_endpoint_exploration_expected_machines_missing_overall_count | gauge | Number of machines expected but not identified |
| carbide_endpoint_exploration_expected_power_shelves_missing_overall_count | gauge | Number of power shelves expected but not identified |
| carbide_endpoint_exploration_identified_managed_hosts_overall_count | gauge | Number of managed hosts identified by expectation |
| carbide_endpoint_exploration_machines_explored_overall_count | gauge | Number of machines explored, by expectation and machine type |
| carbide_endpoint_exploration_step_latency_milliseconds | histogram | The time it took to perform one endpoint exploration step |
| carbide_endpoint_exploration_success_count | gauge | Number of successful endpoint explorations |
| carbide_endpoint_explorations_count | gauge | Number of attempted endpoint explorations |
| carbide_exhausted_reprovision_retry_count | gauge | Number of host machines in the system whose host firmware upgrade retry budget is exhausted. |
| carbide_extension_service_credential_cleanup_failures_total | counter | Number of extension-service credential cleanup failures, by operation. |
| carbide_external_call_duration_milliseconds | histogram | Duration of outbound calls by backend, operation, and outcome; the _count series, split by outcome, gives the request and error rates. |
| carbide_firmware_download_duration_seconds | histogram | Duration of background firmware artifact downloads, by outcome; an ok attempt spans fetch, checksum verification, and publish, and the _count series, split by outcome, is the download and failure rate. |
| carbide_firmware_update_failures_total | counter | Number of firmware update failures, by update target and cause |
| carbide_firmware_updates_total | counter | Number of firmware updates started and completed, by update target and phase; only the host target emits both phases |
| carbide_fmds_config_updates_total | counter | Number of FMDS gRPC config-update ingests, by outcome |
| carbide_fmds_phone_home_total | counter | Number of FMDS tenant phone-home operations, by outcome |
| carbide_gpus_in_use_count | gauge | Number of GPUs actively used by tenants in instances in the NICo deployment |
| carbide_gpus_total_count | gauge | Number of GPUs in the NICo deployment |
| carbide_gpus_usable_count | gauge | Number of remaining GPUs in the NICo deployment available for immediate instance creation |
| carbide_health_otlp_export_failures_total | counter | Number of OTLP export batches dropped after a send failure, by signal and gRPC status code. |
| carbide_health_report_submissions_total | counter | Number of health report submissions to the NICo API, by report target and outcome. |
| carbide_host_reprovision_retries_total | counter | Number of times a failed host firmware upgrade was retried during host reprovisioning |
| carbide_hosts_by_sku_count | gauge | Number of hosts by SKU and device type (‘unknown’ for hosts without SKU) |
| carbide_hosts_health_overrides_count | gauge | Number of health overrides configured in the site |
| carbide_hosts_health_status_count | gauge | Number of managed hosts in the system that have reported either a healthy or not healthy status - based on the presence of health probe alerts |
| carbide_hosts_in_use_count | gauge | Number of hosts actively used by tenants as instances in the NICo deployment |
| carbide_hosts_unhealthy_by_classification_count | gauge | Number of objects marked with a certain classification due to being unhealthy |
| carbide_hosts_unhealthy_by_probe_id_count | gauge | Number of objects which reported a certain Health Probe Alert |
| carbide_hosts_usable_count | gauge | Number of remaining hosts in the NICo deployment available for immediate instance creation |
| carbide_hosts_with_bios_password_set | gauge | Number of hosts in the system that have their BIOS password set. |
| carbide_ib_monitor_fabrics_count | gauge | Number of monitored InfiniBand fabrics |
| carbide_ib_monitor_iteration_latency_milliseconds | histogram | The time it took to perform one IB fabric monitor iteration |
| carbide_ib_monitor_machine_ib_status_updates_count | gauge | Number of Machines whose InfiniBand status observation was updated |
| carbide_ib_monitor_machines_with_missing_pkeys_count | gauge | Number of machines where at least one port is not assigned to the expected pkey on UFM |
| carbide_ib_monitor_machines_with_unexpected_pkeys_count | gauge | Number of machines where at least one port is assigned to an unexpected pkey on UFM |
| carbide_ib_monitor_machines_with_unknown_pkeys_count | gauge | Number of machines where at least one port is assigned to a pkey value not associated with any partition ID |
| carbide_ib_monitor_ufm_changes_applied_total | counter | Number of changes performed at UFM |
| carbide_ib_partitions_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_ib_partitions in the system |
| carbide_ib_partitions_iteration_latency_milliseconds | histogram | The elapsed time in the last state processor iteration to handle objects of type carbide_ib_partitions |
| carbide_ib_partitions_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_ib_partitions |
| carbide_ib_partitions_total | gauge | Number of carbide_ib_partitions in the system |
| carbide_ipmi_commands_total | counter | Number of IPMI command executions, by command and outcome. |
| carbide_log_events_total | counter | Number of log events emitted, by level and component. The always-on log-volume and error-rate signal for every binary. |
| carbide_machine_dpu_info | gauge | Association between a host machine and one of its DPUs. Exposed only on the opt-in per-object endpoint. |
| carbide_machine_identity_token_exchange_failures_total | counter | Number of machine identity token exchange failures, by failure stage |
| carbide_machine_instance_info | gauge | Association between a host machine and its tenant instance. Exposed only on the opt-in per-object endpoint. |
| carbide_machine_reboot_duration_seconds | histogram | Time taken for machine/host to reboot in seconds |
| carbide_machine_updates_started_count | gauge | Number of machines in the system in the process of updating |
| carbide_machine_validation_completed | gauge | Number of successfully completed machine validation runs |
| carbide_machine_validation_failed | gauge | Number of failed machine validation runs |
| carbide_machine_validation_in_progress | gauge | Number of machine validation runs in progress |
| carbide_machine_validation_oldest_active_age_seconds | gauge | Age in seconds of the oldest active machine validation run |
| carbide_machine_validation_outcomes_total | counter | Number of machine validation runs that completed as passed or failed, by outcome and failure cause; runs skipped by a disabled validation config are not counted |
| carbide_machine_validation_stale_runs_count | gauge | Number of active machine validation runs considered stale |
| carbide_machine_validation_tests | gauge | The details of machine validation tests |
| carbide_machines_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_machines in the system |
| carbide_machines_handler_latency_in_state_milliseconds | histogram | The amount of time it took to invoke the state handler for objects of type carbide_machines in a certain state |
| carbide_machines_in_maintenance_count | gauge | Number of machines in the system in maintenance |
| carbide_machines_iteration_latency_milliseconds | histogram | The elapsed time in the last state processor iteration to handle objects of type carbide_machines |
| carbide_machines_object_tasks_completed_total | counter | Number of object handling tasks completed for objects of type carbide_machines |
| carbide_machines_object_tasks_dispatched_total | counter | Number of object handling tasks dequeued and dispatched for processing for objects of type carbide_machines |
| carbide_machines_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_machines |
| carbide_machines_object_tasks_errored_total | counter | Number of object handling tasks completed with an error for objects of type carbide_machines |
| carbide_machines_object_tasks_requeued_total | counter | Number of object handling tasks requeued for objects of type carbide_machines |
| carbide_machines_per_state | gauge | Number of carbide_machines in the system with a given state |
| carbide_machines_per_state_above_sla | gauge | Number of carbide_machines currently in a state longer than the SLA allows |
| carbide_machines_state_entered_total | counter | Number of times objects of type carbide_machines have entered a certain state |
| carbide_machines_state_exited_total | counter | Number of times objects of type carbide_machines have exited a certain state |
| carbide_machines_time_in_state_seconds | histogram | The amount of time objects of type carbide_machines have spent in a certain state |
| carbide_machines_total | gauge | Number of carbide_machines in the system |
| carbide_machines_with_state_handling_errors_per_state | gauge | Number of state-handling errors for carbide_machines in a given state |
| carbide_managed_loop_iterations_total | counter | Number of managed loop iterations, by manager and outcome; the measured boot metrics collector’s iterations are counted by its latency histogram instead |
| carbide_measured_boot_bundles_total | gauge | Number of measured boot bundles. |
| carbide_measured_boot_collector_iteration_latency_milliseconds | histogram | Number of milliseconds a full measured boot metrics collector iteration took, by outcome |
| carbide_measured_boot_machines_per_bundle_state_total | gauge | Number of machines per measured boot bundle state. |
| carbide_measured_boot_machines_per_machine_state_total | gauge | Number of machines per measured boot machine state. |
| carbide_measured_boot_machines_total | gauge | Number of machines reporting measurements. |
| carbide_measured_boot_profiles_total | gauge | Number of measured boot profiles. |
| carbide_measured_boot_verification_failures_total | counter | Number of measured boot verification failures, across quote verification and attestation handling, by cause |
| carbide_metrics_scrape_failures_total | counter | Number of /metrics scrape failures, by outcome. |
| carbide_mqtt_dispatch_dropped_total | counter | Number of received MQTT messages dropped because the handler concurrency semaphore could not be acquired |
| carbide_mqtt_reconnects_total | counter | Number of times an MQTT client re-established its broker connection after the initial connect |
| carbide_network_segments_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_network_segments in the system |
| carbide_network_segments_handler_latency_in_state_milliseconds | histogram | The amount of time it took to invoke the state handler for objects of type carbide_network_segments in a certain state |
| carbide_network_segments_iteration_latency_milliseconds | histogram | The elapsed time in the last state processor iteration to handle objects of type carbide_network_segments |
| carbide_network_segments_object_tasks_completed_total | counter | Number of object handling tasks completed for objects of type carbide_network_segments |
| carbide_network_segments_object_tasks_dispatched_total | counter | Number of object handling tasks dequeued and dispatched for processing for objects of type carbide_network_segments |
| carbide_network_segments_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_network_segments |
| carbide_network_segments_object_tasks_requeued_total | counter | Number of object handling tasks requeued for objects of type carbide_network_segments |
| carbide_network_segments_per_state | gauge | Number of carbide_network_segments in the system with a given state |
| carbide_network_segments_per_state_above_sla | gauge | Number of carbide_network_segments currently in a state longer than the SLA allows |
| carbide_network_segments_state_entered_total | counter | Number of times objects of type carbide_network_segments have entered a certain state |
| carbide_network_segments_state_exited_total | counter | Number of times objects of type carbide_network_segments have exited a certain state |
| carbide_network_segments_time_in_state_seconds | histogram | The amount of time objects of type carbide_network_segments have spent in a certain state |
| carbide_network_segments_total | gauge | Number of carbide_network_segments in the system |
| carbide_network_segments_with_state_handling_errors_per_state | gauge | Number of state-handling errors for carbide_network_segments in a given state |
| carbide_nvlink_partition_monitor_iteration_latency_milliseconds | histogram | Time consumed for one monitor iteration |
| carbide_nvlink_partition_monitor_machine_status_updates_count | gauge | Number of machines whose NVLink status observation was updated |
| carbide_nvlink_partition_monitor_nmxc_changes_applied_total | counter | Number of changes requested to NMX-C |
| carbide_nvlink_partition_monitor_nmxc_op_latency_milliseconds | histogram | Time consumed for one NMX-C operation |
| carbide_nvlink_partition_monitor_num_logical_partitions | gauge | Number of monitored logical partitions |
| carbide_nvlink_partition_monitor_num_physical_partitions | gauge | Number of monitored physical partitions |
| carbide_nvlink_partition_monitor_nvlink_info_mismatches | gauge | Number of NVLink GPU partition ID mismatches between DB and NMX-C |
| carbide_nvlink_partition_monitor_stale_partitions_deleted | gauge | Number of stale partitions deleted from DB (not found in NMX-C) |
| carbide_nvlink_switch_cert_monitor_iteration_latency_milliseconds | histogram | Time consumed for one NMX-C switch certificate monitor iteration |
| carbide_object_info | gauge | Stable identifying traits for a state-controller object. Exposed only on the opt-in per-object endpoint. |
| carbide_object_manual_intervention_required | gauge | Indicates that an object currently requires manual operator intervention. Exposed only on the opt-in per-object endpoint. |
| carbide_object_state_entered_timestamp_seconds | gauge | Unix timestamp at which an object entered its current state. Exposed only on the opt-in per-object endpoint. |
| carbide_object_state_sla_seconds | gauge | Resolved SLA in seconds for an objects current state. Exposed only on the opt-in per-object endpoint. |
| carbide_pending_dpu_nic_firmware_update_count | gauge | Number of machines in the system that need a DPU/NIC firmware update |
| carbide_pending_host_firmware_update_count | gauge | Number of host machines in the system that need a firmware update. |
| carbide_power_shelves_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_power_shelves in the system |
| carbide_power_shelves_health_overrides_count | gauge | Number of health overrides configured in the site |
| carbide_power_shelves_health_status_count | gauge | Number of power shelves in the system that have reported either a healthy or not healthy status - based on the presence of health probe alerts |
| carbide_power_shelves_iteration_latency_milliseconds | histogram | The elapsed time in the last state processor iteration to handle objects of type carbide_power_shelves |
| carbide_power_shelves_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_power_shelves |
| carbide_power_shelves_total | gauge | Number of carbide_power_shelves in the system |
| carbide_preingestion_bfb_copy_duration_seconds | histogram | Duration of preingestion BFB copies to a DPU rshim, by outcome; the _count series, split by outcome, is the copy and failure rate. |
| carbide_preingestion_firmware_upgrade_tasks_total | counter | Number of preingestion firmware upgrade Redfish tasks reaching a terminal state, by firmware component, final task state, and outcome. |
| carbide_preingestion_firmware_upload_total | counter | Number of preingestion firmware uploads to a BMC, by upload method and outcome. |
| carbide_preingestion_power_control_total | counter | Number of preingestion Redfish power operations (host power control, BMC and chassis resets), by operation and outcome. |
| carbide_preingestion_total | gauge | Number of known machines currently being evaluated prior to ingestion |
| carbide_preingestion_waiting_download | gauge | Number of machines that are waiting for firmware downloads on other machines to complete before doing their own |
| carbide_preingestion_waiting_installation | gauge | Number of machines which have had firmware uploaded to them and are currently in the process of installing that firmware |
| carbide_pxe_boot_outcomes_total | counter | Number of PXE boot-path outcomes served, by endpoint and reason. |
| carbide_racks_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_racks in the system |
| carbide_racks_health_overrides_count | gauge | Number of health overrides configured in the site |
| carbide_racks_health_status_count | gauge | Number of racks in the system that have reported either a healthy or not healthy status - based on the presence of health probe alerts |
| carbide_racks_iteration_latency_milliseconds | histogram | The elapsed time in the last state processor iteration to handle objects of type carbide_racks |
| carbide_racks_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_racks |
| carbide_racks_total | gauge | Number of carbide_racks in the system |
| carbide_reboot_attempts_in_booting_with_discovery_image | histogram | Reboot attempts per machine in BootingWithDiscoveryImage, recorded when a machine is rebooted again after no response from the host. |
| carbide_redfish_action_result_persistence_failures_total | counter | Number of detached Redfish action results that failed to persist |
| carbide_reserved_ips_count | gauge | Number of reserved IPs per network segment |
| carbide_resourcepool_free_count | gauge | Number of values in the resource pool currently available for allocation |
| carbide_resourcepool_used_count | gauge | Number of currently allocated values in the resource pool |
| carbide_running_dpu_updates_count | gauge | Number of machines in the system that are currently running a DPU/NIC firmware update |
| carbide_scout_actions_total | counter | Number of scout control-loop actions handled, by action and outcome. |
| carbide_scout_storage_device_cleanup_duration_seconds | histogram | Duration of per-device scout storage cleanup operations, by device type and outcome. |
| carbide_scout_stream_connections_total | counter | Number of scout stream connection attempts, by outcome. |
| carbide_scout_stream_reconnects_total | counter | Number of scout stream reconnect cycles after a stream closed or errored. |
| carbide_scout_stream_responses_dropped_total | counter | Number of scout stream responses dropped after the outbound request stream closed. |
| carbide_site_exploration_expected_machines_sku_count | gauge | Number of expected machines by SKU ID and device type |
| carbide_site_exploration_identified_managed_hosts_count | gauge | Number of Host+DPU pairs identified in the last SiteExplorer run |
| carbide_site_explorer_bmc_password_rotations_total | counter | Number of BMC root password rotations onto the site-wide credential, by outcome |
| carbide_site_explorer_bmc_reset_attempts_total | counter | Number of Site Explorer BMC reset attempts, by method and status. |
| carbide_site_explorer_bmc_reset_count | gauge | Number of successful BMC resets in the last SiteExplorer run |
| carbide_site_explorer_bmc_reset_timestamp_persistence_failures_total | counter | Number of Site Explorer BMC reset timestamp persistence failures, by method. |
| carbide_site_explorer_create_machines | gauge | Whether site-explorer machine creation is enabled (1) or disabled (0) |
| carbide_site_explorer_create_machines_latency_milliseconds | histogram | The time it took to perform create_machines inside site-explorer |
| carbide_site_explorer_created_machines_count | gauge | Number of machine pairs created by Site Explorer after identification |
| carbide_site_explorer_created_power_shelves_count | gauge | Number of power shelves created by Site Explorer after identification |
| carbide_site_explorer_dpu_migration_signals_count | gauge | Number of DPU NIC-mode migration signals by signal type — mode-mismatch found, set_nic_mode issued, reset requested, and zero-DPU registered for a NicMode host. |
| carbide_site_explorer_enabled | gauge | Whether site-explorer is enabled (1) or paused (0) |
| carbide_site_explorer_iteration_latency_milliseconds | histogram | The time it took to perform one site explorer iteration |
| carbide_site_explorer_last_run_status | gauge | The status of the latest Site Explorer run |
| carbide_site_explorer_phase_latency_milliseconds | histogram | The time it took to perform one site explorer iteration phase |
| carbide_site_explorer_update_explored_endpoints_count | gauge | Counts from the last update_explored_endpoints phase by kind |
| carbide_state_handler_wakeup_failures_total | counter | Number of times a machine’s state handler could not be woken after an agent-reported event |
| carbide_switches_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_switches in the system |
| carbide_switches_health_overrides_count | gauge | Number of health overrides configured in the site |
| carbide_switches_health_status_count | gauge | Number of switches in the system that have reported either a healthy or not healthy status - based on the presence of health probe alerts |
| carbide_switches_iteration_latency_milliseconds | histogram | The elapsed time in the last state processor iteration to handle objects of type carbide_switches |
| carbide_switches_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_switches |
| carbide_switches_total | gauge | Number of carbide_switches in the system |
| carbide_total_ips_count | gauge | Number of IPs per network segment |
| carbide_unavailable_dpu_nic_firmware_update_count | gauge | Number of machines in the system that need a DPU/NIC firmware update but are unavailable for update |
| carbide_vpc_prefixes_enqueuer_iteration_latency_milliseconds | histogram | The overall time it took to enqueue state handling tasks for all carbide_vpc_prefixes in the system |
| carbide_vpc_prefixes_handler_latency_in_state_milliseconds | histogram | The amount of time it took to invoke the state handler for objects of type carbide_vpc_prefixes in a certain state |
| carbide_vpc_prefixes_iteration_latency_milliseconds | histogram | The elapsed time in the last state processor iteration to handle objects of type carbide_vpc_prefixes |
| carbide_vpc_prefixes_object_tasks_completed_total | counter | Number of object handling tasks completed for objects of type carbide_vpc_prefixes |
| carbide_vpc_prefixes_object_tasks_dispatched_total | counter | Number of object handling tasks dequeued and dispatched for processing for objects of type carbide_vpc_prefixes |
| carbide_vpc_prefixes_object_tasks_enqueued_total | counter | Number of object handling tasks freshly enqueued for objects of type carbide_vpc_prefixes |
| carbide_vpc_prefixes_object_tasks_requeued_total | counter | Number of object handling tasks requeued for objects of type carbide_vpc_prefixes |
| carbide_vpc_prefixes_per_state | gauge | Number of carbide_vpc_prefixes in the system with a given state |
| carbide_vpc_prefixes_per_state_above_sla | gauge | Number of carbide_vpc_prefixes currently in a state longer than the SLA allows |
| carbide_vpc_prefixes_state_entered_total | counter | Number of times objects of type carbide_vpc_prefixes have entered a certain state |
| carbide_vpc_prefixes_state_exited_total | counter | Number of times objects of type carbide_vpc_prefixes have exited a certain state |
| carbide_vpc_prefixes_time_in_state_seconds | histogram | The amount of time objects of type carbide_vpc_prefixes have spent in a certain state |
| carbide_vpc_prefixes_total | gauge | Number of carbide_vpc_prefixes in the system |
| carbide_vpc_prefixes_with_state_handling_errors_per_state | gauge | Number of state-handling errors for carbide_vpc_prefixes in a given state |
| carbide_work_lock_failures_total | counter | Number of work-lock lifecycle failures, by operation and failure kind. |
| site_explorer_create_power_shelves_latency_seconds | histogram | Duration of power shelf creation |
| site_explorer_create_switches_latency_seconds | histogram | Duration of switch creation |