Core Metrics | NVIDIA Infra Controller

This file contains a list of metrics exported by NVIDIA Infra Controller (NICo). The list is auto-generated from an integration test (test_integration). Metrics for workflows which are not exercised by the test are missing. NVLink partition monitor’s metrics are documented in the manual: NVLink Partitioning.

Name	Type	Description
carbide_active_host_firmware_update_count	gauge	The number of host machines in the system currently working on updating their firmware.
carbide_api_db_queries_total	counter	The amount of database queries that occurred inside a span
carbide_api_db_span_query_time_milliseconds	histogram	Total time the request spent inside a span on database transactions
carbide_api_grpc_server_duration_milliseconds	histogram	Processing time for a request on the carbide API server
carbide_api_ready	gauge	Whether the NICo API is running
carbide_api_tls_cert_refreshes_total	counter	Number of TLS acceptor refreshes performed by the API listener
carbide_api_tls_connection_attempted_total	counter	Number of attempted TLS connections
carbide_api_tls_connection_success_total	counter	Number of successful TLS connections
carbide_api_tracing_spans_open	gauge	Number of open logging/tracing spans
carbide_api_vault_request_duration_milliseconds	histogram	the duration of outbound vault requests, in milliseconds
carbide_api_vault_requests_attempted_total	counter	The amount of tls connections that were attempted
carbide_api_vault_requests_failed_total	counter	The amount of tcp connections that were failures
carbide_api_vault_requests_succeeded_total	counter	The amount of tls connections that were successful
carbide_api_vault_token_time_until_refresh_seconds	gauge	The amount of time, in seconds, until the vault token is required to be refreshed
carbide_api_version	gauge	Version (git sha, build date, etc) of this service
carbide_authn_client_cert_rejected_total	counter	Number of client certificates rejected during authentication
carbide_available_ips_count	gauge	The total number of available ips in the site
carbide_client_tcp_connect_attempts_total	counter	Number of outbound TCP connect attempts across all HTTP connectors
carbide_client_tcp_connect_errors_total	counter	Number of failed outbound TCP connect attempts across all HTTP connectors
carbide_client_tcp_connect_successes_total	counter	Number of successful outbound TCP connects across all HTTP connectors
carbide_concurrent_machine_updates_available	gauge	The number of machines in the system that we will update concurrently.
carbide_db_pool_idle_conns	gauge	The amount of idle connections in the carbide database pool
carbide_db_pool_total_conns	gauge	The amount of total (active + idle) connections in the carbide database pool
carbide_dpu_agent_version_count	gauge	The amount of DPU agents which have reported a certain version.
carbide_dpu_firmware_version_count	gauge	The amount of DPUs which have reported a certain firmware version.
carbide_dpus_healthy_count	gauge	The total number of DPUs in the system that have reported healthy in the last report. Healthy does not imply up - the report from the DPU might be outdated.
carbide_dpus_up_count	gauge	The total number of DPUs in the system that are up. Up means we have received a health report less than 5 minutes ago.
carbide_endpoint_exploration_duration_milliseconds	histogram	The time it took to explore an endpoint
carbide_endpoint_exploration_expected_machines_missing_overall_count	gauge	The total number of machines that were expected but not identified
carbide_endpoint_exploration_expected_power_shelves_missing_overall_count	gauge	The total number of power shelves that were expected but not identified
carbide_endpoint_exploration_identified_managed_hosts_overall_count	gauge	The total number of managed hosts identified by expectation
carbide_endpoint_exploration_machines_explored_overall_count	gauge	The total number of machines explored by machine type
carbide_endpoint_exploration_step_latency_milliseconds	histogram	The time it took to perform one endpoint exploration step
carbide_endpoint_exploration_success_count	gauge	The amount of endpoint explorations that have been successful
carbide_endpoint_explorations_count	gauge	The amount of endpoint explorations that have been attempted
carbide_exhausted_reprovision_retry_count	gauge	Number of host machines in the system whose host firmware upgrade retry budget is exhausted.
carbide_external_call_duration_milliseconds	histogram	Duration of outbound calls by backend, operation, and outcome; the _count series, split by outcome, gives the request and error rates.
carbide_firmware_update_failures_total	counter	Number of firmware update failures, by update target and cause
carbide_firmware_updates_total	counter	Number of firmware updates started and completed, by update target and phase; only the host target emits both phases
carbide_gpus_in_use_count	gauge	The total number of GPUs that are actively used by tenants in instances in the NICo deployment
carbide_gpus_total_count	gauge	The total number of GPUs available in the NICo deployment
carbide_gpus_usable_count	gauge	The remaining number of GPUs in the NICo deployment which are available for immediate instance creation
carbide_hosts_by_sku_count	gauge	The amount of hosts by SKU and device type (‘unknown’ for hosts without SKU)
carbide_hosts_health_overrides_count	gauge	The amount of health overrides that are configured in the site
carbide_hosts_health_status_count	gauge	The total number of Managed Hosts in the system that have reported either a healthy or not healthy status - based on the presence of health probe alerts
carbide_hosts_in_use_count	gauge	The total number of hosts that are actively used by tenants as instances in the NICo deployment
carbide_hosts_unhealthy_by_classification_count	gauge	The amount of objects which are marked with a certain classification due to being unhealthy
carbide_hosts_unhealthy_by_probe_id_count	gauge	The amount of objects which reported a certain Health Probe Alert
carbide_hosts_usable_count	gauge	The remaining number of hosts in the NICo deployment which are available for immediate instance creation
carbide_hosts_with_bios_password_set	gauge	The total number of Hosts in the system that have their BIOS password set.
carbide_ib_monitor_fabrics_count	gauge	The amount of InfiniBand fabrics that are monitored
carbide_ib_monitor_machine_ib_status_updates_count	gauge	The amount of Machines where the infiniband_status_observation got updated
carbide_ib_monitor_machines_with_missing_pkeys_count	gauge	The amount of machines where at least one port is not assigned to the expected pkey on UFM
carbide_ib_monitor_machines_with_unexpected_pkeys_count	gauge	The amount of machines where at least one port is assigned to an unexpected pkey on UFM
carbide_ib_monitor_machines_with_unknown_pkeys_count	gauge	The amount of machines where at least one port is assigned to a pkey value that is not associated with any partition ID
carbide_ib_partitions_enqueuer_iteration_latency_milliseconds	histogram	The overall time it took to enqueue state handling tasks for all carbide_ib_partitions in the system
carbide_ib_partitions_iteration_latency_milliseconds	histogram	The elapsed time in the last state processor iteration to handle objects of type carbide_ib_partitions
carbide_ib_partitions_object_tasks_enqueued_total	counter	The amount of types that object handling tasks that have been freshly enqueued for objects of type carbide_ib_partitions
carbide_ib_partitions_total	gauge	The total number of carbide_ib_partitions in the system
carbide_log_events_total	counter	Number of log events emitted, by level and component. The always-on log-volume and error-rate signal for every binary.
carbide_machine_reboot_duration_seconds	histogram	Time taken for machine/host to reboot in seconds
carbide_machine_updates_started_count	gauge	The number of machines in the system that are in the process of updating.
carbide_machine_validation_completed	gauge	Count of machine validation that have completed successfully
carbide_machine_validation_failed	gauge	Count of machine validation that have failed
carbide_machine_validation_in_progress	gauge	Count of machine validation that are in progress
carbide_machine_validation_oldest_active_age_seconds	gauge	Age in seconds of the oldest active machine validation run
carbide_machine_validation_outcomes_total	counter	Number of machine validation runs that completed as passed or failed, by outcome and failure cause; runs skipped by a disabled validation config are not counted
carbide_machine_validation_stale_runs_count	gauge	Count of active machine validation runs considered stale
carbide_machine_validation_tests	gauge	The details of machine validation tests
carbide_machines_enqueuer_iteration_latency_milliseconds	histogram	The overall time it took to enqueue state handling tasks for all carbide_machines in the system
carbide_machines_handler_latency_in_state_milliseconds	histogram	The amount of time it took to invoke the state handler for objects of type carbide_machines in a certain state
carbide_machines_in_maintenance_count	gauge	The total number of machines in the system that are in maintenance.
carbide_machines_iteration_latency_milliseconds	histogram	The elapsed time in the last state processor iteration to handle objects of type carbide_machines
carbide_machines_object_tasks_completed_total	counter	The amount of object handling tasks that have been completed for objects of type carbide_machines
carbide_machines_object_tasks_dispatched_total	counter	The amount of types that object handling tasks that have been dequeued and dispatched for processing for objects of type carbide_machines
carbide_machines_object_tasks_enqueued_total	counter	The amount of types that object handling tasks that have been freshly enqueued for objects of type carbide_machines
carbide_machines_object_tasks_errored_total	counter	Number of object handling tasks that have completed with an error for objects of type carbide_machines
carbide_machines_object_tasks_requeued_total	counter	The amount of object handling tasks that have been requeued for objects of type carbide_machines
carbide_machines_per_state	gauge	The number of carbide_machines in the system with a given state
carbide_machines_per_state_above_sla	gauge	The number of carbide_machines in the system which had been longer in a state than allowed per SLA
carbide_machines_state_entered_total	counter	The amount of types that objects of type carbide_machines have entered a certain state
carbide_machines_state_exited_total	counter	The amount of types that objects of type carbide_machines have exited a certain state
carbide_machines_time_in_state_seconds	histogram	The amount of time objects of type carbide_machines have spent in a certain state
carbide_machines_total	gauge	The total number of carbide_machines in the system
carbide_machines_with_state_handling_errors_per_state	gauge	The number of carbide_machines in the system with a given state that failed state handling
carbide_measured_boot_bundles_total	gauge	The total number of measured boot bundles.
carbide_measured_boot_collector_iteration_latency_milliseconds	histogram	Number of milliseconds a full measured boot metrics collector iteration took, by outcome
carbide_measured_boot_machines_per_bundle_state_total	gauge	The total number of machines per a given measured boot bundle state.
carbide_measured_boot_machines_per_machine_state_total	gauge	The total number of machines per a given measured boot machine state.
carbide_measured_boot_machines_total	gauge	The total number of machines reporting measurements.
carbide_measured_boot_profiles_total	gauge	The total number of measured boot profiles.
carbide_measured_boot_verification_failures_total	counter	Number of measured boot verification failures, across quote verification and attestation handling, by cause
carbide_network_segments_enqueuer_iteration_latency_milliseconds	histogram	The overall time it took to enqueue state handling tasks for all carbide_network_segments in the system
carbide_network_segments_handler_latency_in_state_milliseconds	histogram	The amount of time it took to invoke the state handler for objects of type carbide_network_segments in a certain state
carbide_network_segments_iteration_latency_milliseconds	histogram	The elapsed time in the last state processor iteration to handle objects of type carbide_network_segments
carbide_network_segments_object_tasks_completed_total	counter	The amount of object handling tasks that have been completed for objects of type carbide_network_segments
carbide_network_segments_object_tasks_dispatched_total	counter	The amount of types that object handling tasks that have been dequeued and dispatched for processing for objects of type carbide_network_segments
carbide_network_segments_object_tasks_enqueued_total	counter	The amount of types that object handling tasks that have been freshly enqueued for objects of type carbide_network_segments
carbide_network_segments_object_tasks_requeued_total	counter	The amount of object handling tasks that have been requeued for objects of type carbide_network_segments
carbide_network_segments_per_state	gauge	The number of carbide_network_segments in the system with a given state
carbide_network_segments_per_state_above_sla	gauge	The number of carbide_network_segments in the system which had been longer in a state than allowed per SLA
carbide_network_segments_state_entered_total	counter	The amount of types that objects of type carbide_network_segments have entered a certain state
carbide_network_segments_state_exited_total	counter	The amount of types that objects of type carbide_network_segments have exited a certain state
carbide_network_segments_time_in_state_seconds	histogram	The amount of time objects of type carbide_network_segments have spent in a certain state
carbide_network_segments_total	gauge	The total number of carbide_network_segments in the system
carbide_network_segments_with_state_handling_errors_per_state	gauge	The number of carbide_network_segments in the system with a given state that failed state handling
carbide_nvlink_partition_monitor_machine_status_updates_count	gauge	Number of machines nvlink_status_observation got updated
carbide_nvlink_partition_monitor_nmxc_changes_applied_total	counter	Number of changes requested to NMX-C
carbide_nvlink_partition_monitor_num_logical_partitions	gauge	Number of logical partitions that were monitored
carbide_nvlink_partition_monitor_num_physical_partitions	gauge	Number of physical partitions that were monitored
carbide_nvlink_partition_monitor_nvlink_info_mismatches	gauge	Number of NVLink GPU partition ID mismatches between DB and NMX-C
carbide_nvlink_partition_monitor_stale_partitions_deleted	gauge	Number of stale partitions deleted from DB (not found in NMX-C)
carbide_pending_dpu_nic_firmware_update_count	gauge	The number of machines in the system that need a firmware update.
carbide_pending_host_firmware_update_count	gauge	The number of host machines in the system that need a firmware update.
carbide_power_shelves_enqueuer_iteration_latency_milliseconds	histogram	The overall time it took to enqueue state handling tasks for all carbide_power_shelves in the system
carbide_power_shelves_health_overrides_count	gauge	The amount of health overrides that are configured in the site
carbide_power_shelves_health_status_count	gauge	The total number of Power Shelves in the system that have reported either a healthy or not healthy status - based on the presence of health probe alerts
carbide_power_shelves_iteration_latency_milliseconds	histogram	The elapsed time in the last state processor iteration to handle objects of type carbide_power_shelves
carbide_power_shelves_object_tasks_enqueued_total	counter	The amount of types that object handling tasks that have been freshly enqueued for objects of type carbide_power_shelves
carbide_power_shelves_total	gauge	The total number of carbide_power_shelves in the system
carbide_preingestion_total	gauge	The amount of known machines currently being evaluated prior to ingestion
carbide_preingestion_waiting_download	gauge	The amount of machines that are waiting for firmware downloads on other machines to complete before doing their own
carbide_preingestion_waiting_installation	gauge	The amount of machines which have had firmware uploaded to them and are currently in the process of installing that firmware
carbide_racks_enqueuer_iteration_latency_milliseconds	histogram	The overall time it took to enqueue state handling tasks for all carbide_racks in the system
carbide_racks_health_overrides_count	gauge	The amount of health overrides that are configured in the site
carbide_racks_health_status_count	gauge	The total number of Racks in the system that have reported either a healthy or not healthy status - based on the presence of health probe alerts
carbide_racks_iteration_latency_milliseconds	histogram	The elapsed time in the last state processor iteration to handle objects of type carbide_racks
carbide_racks_object_tasks_enqueued_total	counter	The amount of types that object handling tasks that have been freshly enqueued for objects of type carbide_racks
carbide_racks_total	gauge	The total number of carbide_racks in the system
carbide_reboot_attempts_in_booting_with_discovery_image	histogram	The amount of machines rebooted again in BootingWithDiscoveryImage since there is no response after a certain time from host.
carbide_reserved_ips_count	gauge	The total number of reserved ips in the site
carbide_resourcepool_free_count	gauge	Count of values in the pool currently available for allocation
carbide_resourcepool_used_count	gauge	Count of values in the pool currently allocated
carbide_running_dpu_updates_count	gauge	The number of machines in the system that are running a firmware update.
carbide_site_exploration_expected_machines_sku_count	gauge	The total count of expected machines by SKU ID and device type
carbide_site_exploration_identified_managed_hosts_count	gauge	The amount of Host+DPU pairs that has been identified in the last SiteExplorer run
carbide_site_explorer_bmc_password_rotations_total	counter	Number of BMC root password rotations onto the site-wide credential, by outcome
carbide_site_explorer_bmc_reset_count	gauge	The amount of BMC resets initiated in the last SiteExplorer run
carbide_site_explorer_create_machines	gauge	Whether site-explorer machine creation is enabled (1) or disabled (0)
carbide_site_explorer_create_machines_latency_milliseconds	histogram	The time it took to perform create_machines inside site-explorer
carbide_site_explorer_created_machines_count	gauge	The amount of Machine pairs that had been created by Site Explorer after being identified
carbide_site_explorer_created_power_shelves_count	gauge	The amount of Power Shelves that had been created by Site Explorer after being identified
carbide_site_explorer_dpu_migration_signals_count	gauge	Count of DPU NIC-mode migration signals by kind — mode-mismatch found, set_nic_mode issued, reset requested, and zero-DPU registered for a NicMode host.
carbide_site_explorer_enabled	gauge	Whether site-explorer is enabled (1) or paused (0)
carbide_site_explorer_iteration_latency_milliseconds	histogram	The time it took to perform one site explorer iteration
carbide_site_explorer_last_run_status	gauge	The status of the latest Site Explorer run
carbide_site_explorer_phase_latency_milliseconds	histogram	The time it took to perform one site explorer iteration phase
carbide_site_explorer_update_explored_endpoints_count	gauge	Counts from the last update_explored_endpoints phase by kind
carbide_switches_enqueuer_iteration_latency_milliseconds	histogram	The overall time it took to enqueue state handling tasks for all carbide_switches in the system
carbide_switches_health_overrides_count	gauge	The amount of health overrides that are configured in the site
carbide_switches_health_status_count	gauge	The total number of Switches in the system that have reported either a healthy or not healthy status - based on the presence of health probe alerts
carbide_switches_iteration_latency_milliseconds	histogram	The elapsed time in the last state processor iteration to handle objects of type carbide_switches
carbide_switches_object_tasks_enqueued_total	counter	The amount of types that object handling tasks that have been freshly enqueued for objects of type carbide_switches
carbide_switches_total	gauge	The total number of carbide_switches in the system
carbide_total_ips_count	gauge	The total number of ips in the site
carbide_unavailable_dpu_nic_firmware_update_count	gauge	The number of machines in the system that need a firmware update but are unavailable for update.
carbide_vpc_prefixes_enqueuer_iteration_latency_milliseconds	histogram	The overall time it took to enqueue state handling tasks for all carbide_vpc_prefixes in the system
carbide_vpc_prefixes_handler_latency_in_state_milliseconds	histogram	The amount of time it took to invoke the state handler for objects of type carbide_vpc_prefixes in a certain state
carbide_vpc_prefixes_iteration_latency_milliseconds	histogram	The elapsed time in the last state processor iteration to handle objects of type carbide_vpc_prefixes
carbide_vpc_prefixes_object_tasks_completed_total	counter	The amount of object handling tasks that have been completed for objects of type carbide_vpc_prefixes
carbide_vpc_prefixes_object_tasks_dispatched_total	counter	The amount of types that object handling tasks that have been dequeued and dispatched for processing for objects of type carbide_vpc_prefixes
carbide_vpc_prefixes_object_tasks_enqueued_total	counter	The amount of types that object handling tasks that have been freshly enqueued for objects of type carbide_vpc_prefixes
carbide_vpc_prefixes_object_tasks_requeued_total	counter	The amount of object handling tasks that have been requeued for objects of type carbide_vpc_prefixes
carbide_vpc_prefixes_per_state	gauge	The number of carbide_vpc_prefixes in the system with a given state
carbide_vpc_prefixes_per_state_above_sla	gauge	The number of carbide_vpc_prefixes in the system which had been longer in a state than allowed per SLA
carbide_vpc_prefixes_state_entered_total	counter	The amount of types that objects of type carbide_vpc_prefixes have entered a certain state
carbide_vpc_prefixes_state_exited_total	counter	The amount of types that objects of type carbide_vpc_prefixes have exited a certain state
carbide_vpc_prefixes_time_in_state_seconds	histogram	The amount of time objects of type carbide_vpc_prefixes have spent in a certain state
carbide_vpc_prefixes_total	gauge	The total number of carbide_vpc_prefixes in the system
carbide_vpc_prefixes_with_state_handling_errors_per_state	gauge	The number of carbide_vpc_prefixes in the system with a given state that failed state handling
site_explorer_create_power_shelves_latency_seconds	histogram	Duration of power shelf creation
site_explorer_create_switches_latency_seconds	histogram	Duration of switch creation