NVIDIA UFM Cable Validation Tool v1.7.1

CVT as a Service

The Cable Validation Tool (CVT) can be deployed and managed as a service, providing automated cable validation capabilities with persistent operation, automatic startup, and web-based management. This document covers the service architecture, configuration, deployment, and management of CVT in service mode.

The CVT service architecture consists of multiple components managed by supervisord.

CVT Container                <p></p>┌──────────────────────────────────────────┐<p></p>│              supervisord                 │<p></p>│  ┌─────────────────────────────────────┐ │<p></p>│  │          CVT Service                │ │<p></p>│  │  ┌─────────────────────────────────┐│ │<p></p>│  │  │     BringupCLI Engine           ││ │<p></p>│  │  │  • Topology Management          ││ │<p></p>│  │  │  • Agent Deployment             ││ │<p></p>│  │  │  • Validation Orchestration     ││ │<p></p>│  │  └─────────────────────────────────┘│ │<p></p>│  │  ┌─────────────────────────────────┐│ │<p></p>│  │  │     Web Service                 ││ │<p></p>│  │  │  • REST API                     ││ │<p></p>│  │  │  • Web UI                       ││ │<p></p>│  │  │  • Authentication               ││ │<p></p>│  │  └─────────────────────────────────┘│ │<p></p>│  └─────────────────────────────────────┘ │<p></p>│  ┌─────────────────────────────────────┐ │<p></p>│  │        Apache Web Server            │ │<p></p>│  │  • HTTPS/SSL Support                │ │<p></p>│  │  • Reverse Proxy                    │ │<p></p>│  │  • Static Content Serving           │ │<p></p>│  └─────────────────────────────────────┘ │<p></p>│  ┌─────────────────────────────────────┐ │<p></p>│  │       CV Controller                 │ │<p></p>│  │  • Service Management API           │ │<p></p>│  │  • System Administration            │ │<p></p>│  └─────────────────────────────────────┘ │<p></p>└──────────────────────────────────────────┘

CVT Service (cvt-service)

  • Purpose: Main CVT application running in daemon mode

  • Command: /usr/local/bin/cvt-service --daemon

  • Features:

    • Automatic topology loading

    • Validation orchestration

    • Agent management

    • Signal handling (SIGTERM, SIGINT)

Apache Web Server (apache)

  • Purpose: HTTPS frontend and reverse proxy

  • Command: /usr/local/bin/apache2_wrapper.sh

  • Features:

    • SSL/TLS termination

    • Web UI serving

    • API reverse proxy

    • Authentication handling

CV Controller (cv_controller)

  • Purpose: Service management and system administration

  • Command: /usr/local/bin/run_cv_controller.sh

  • Features:

    • Service start/stop operations

    • System status monitoring

    • Configuration management

Cron Service (cron)

  • Purpose: Scheduled tasks and maintenance

  • Command: cron -f

  • Features:

    • Log rotation

    • Periodic cleanup

    • Scheduled reports

Docker Deployment for CVT Service

The CVT service is deployed using a pre-built Docker image (bringup service image). This allows for a consistent and isolated runtime environment.

  1. Load the CVT Collector Service Image.

    If you have a local Docker image file (.tar.gz), load it using:

    Copy
    Copied!
                

    docker load -i /path/to/cables_bringup_{version}.tar.gz

  2. [Optional] Pull the Image from a Registry

    If the image is available in a Docker registry, you can pull it directly:

    Copy
    Copied!
                

    docker pull mellanox/cables_bringup:{version}

  3. Run the CVT Service Container: Use the following command to start the CVT service:

    Copy
    Copied!
                

    docker run -itd \ --name cables_bringup \ --network=host \ {override/add env variables} \ -v /path/to/data:/cable_bringup_root \ mellanox/cables_bringup:{version}

    Options:

    Option

    Description

    -itd

    Run container in interactive mode, allocate a pseudo-TTY, and run in detached mode

    --name cables_bringup

    Assigns a name to the container for easier management.

    --network=host

    Shares the host’s network stack for the container.

    {override/add env variables}

    Specify any environment variables needed by the service.

    -v /path/to/data:/cable_bringup_root

    Mounts a host directory into the container for persistent data.

    mellanox/cables_bringup:{version}

    The Docker image and version tag to run.

Environment Variables

The CVT service behavior is controlled through environment variables and configuration files:

Configuration File (/etc/cablevalidation/cvt_env.conf)

Run the following command to edit the file and restart cvt-service:

Copy
Copied!
            

supervisorctl restart cvt-service

For more information, refer to CVT Configuration.

Environment Variable Configuration

Run the following command to configure environment variables:

Copy
Copied!
            

docker run 

Web Service Configuration

Web service configuration can be done only in the docker run command.

Copy
Copied!
            

# Web service interface and ports export CVT_WEB_INTERFACE=127.0.0.1 export CVT_WEB_SERVICE_PORT=8251 export APACHE_HTTPS_PORT=443   # Controller service port export CV_CONTROLLER_PORT=8252

UFM Plugin Mode

Install the CVT UFM plugin using the UFM CLI or Web UI, like any other plugin.

NetQ Plugin Mode

TBD

Supervisord Configuration

The service is managed by supervisordwith the following configuration:

Directive

Description

command

Command to start the CVT service in daemon mode.

autostart

Automatically starts the service when Supervisor starts.

autorestart

Restarts the service automatically if it exits unexpectedly.

startretries

Number of retry attempts if the service fails to start.

startsecs

Number of seconds the service must stay running to be considered started.

stderr_logfile

Path to the log file for standard error output.

stdout_logfile

Path to the log file for standard output.

stdout_logfile_maxbytes / stderr_logfile_maxbytes

Maximum size of each log file before rotation.

stdout_logfile_backups / stderr_logfile_backups

Number of rotated log backups to keep.

user

The user under which the service runs.

killasgroup

Terminates all processes in the service’s process group on stop.

stopasgroup

Sends stop signals to the entire process group.

environment

Sets environment variables for the service (PYTHONUNBUFFERED=1 ensures unbuffered Python output).

CVT Service Management with Supervisor

The CVT service is managed through Supervisor, which allows you to start, stop, restart, and monitor services on your appliance. Below are the common commands:

Service

Description

Command

Start CVT Service

Starts the CVT service if it is not already running.

supervisorctl start cvt-service

Stop CVT Service

Stops the CVT service gracefully.

supervisorctl stop cvt-service

Restart CVT Service

Stops and then starts the CVT service. Useful for applying configuration changes.

supervisorctl restart cvt-service

Check Service Status

Displays the current status of the CVT service (e.g., RUNNING, STOPPED, FATAL).

supervisorctl status cvt-service

View Service Logs

Displays real-time logs from the CVT service. Useful for troubleshooting.

supervisorctl tail cvt-service

Start All Services

Starts all services managed by Supervisor.

supervisorctl start all

Reload Configuration

  • reread: Reloads Supervisor configuration files to detect new or removed programs.

  • update: Applies the configuration changes, starting or stopping services as needed.

supervisorctl reread && supervisorctl update

CVT Service Commands

Command

Description

supervisorctl start cvt-service

Starts the CVT service if it is not already running.

supervisorctl stop cvt-service

Stops the CVT service gracefully.

supervisorctl restart cvt-service

Restarts the CVT service. Useful for applying configuration changes.

supervisorctl status cvt-service

Displays the current status of the CVT service (RUNNING, STOPPED, FATAL).

supervisorctl tail cvt-service

Displays real-time logs from the CVT service for troubleshooting.

supervisorctl start all

Starts all services managed by Supervisor.

supervisorctl reread && supervisorctl update

Reloads configuration


Using CVT Controller API

The CVT Controller provides programmatic service management:

Command

Description

curl -X POST http://localhost:8252/start

Starts the CVT service via the controller.

curl -X POST http://localhost:8252/stop

Stops the CVT service via the controller.

curl -X GET http://localhost:8252/status

Retrieves the CVT service status via the controller. Note: This feature is not supported yet.

Legacy Mode Operation

For troubleshooting, development, or manual operations, you can stop the CVT service and run the traditional interactive CLI:

Stop Service and Run Legacy CLI

  1. Stop the CVT Service

    Copy
    Copied!
                

    supervisorctl stop cvt-service

  2. Run the Legacy Bringup CLI:

    Copy
    Copied!
                

    bringupcli

Legacy CLI Features

When running in legacy mode, you get:

  • Interactive Command Shell: Full command-line interface with tab completion

  • Manual Control: Step-by-step topology loading and validation control

  • Real-time Feedback: Immediate command output and status updates

  • Debugging: Easier debugging with direct command execution

Common Legacy Mode Workflow

  1. Stop the CVT service:

    Copy
    Copied!
                

    supervisorctl stop cvt-service

  2. Start interactive CLI:

    Copy
    Copied!
                

    bringupcli

  3. Perform manual operations:

    Copy
    Copied!
                

    Cable Bringup: load_topo /path/to/topology.topo Cable Bringup: deploy_all_agents Cable Bringup: start_validation Cable Bringup: show_switches Cable Bringup: exit

  4. Restart service when done:

    Copy
    Copied!
                

    upervisorctl start cvt-service

When to Use Legacy Mode

  • Initial Setup: First-time configuration and testing

  • Troubleshooting: Debugging connectivity or configuration issues

  • Manual Operations: One-time tasks that don't require automation

  • Development: Testing new topologies or validation scenarios

  • Training: Learning CVT commands and workflows

Switching Back to Service Mode

  1. Exit the legacy CLI:

    Copy
    Copied!
                

    Cable Bringup: exit

  2. Restart the CVT service:

    Copy
    Copied!
                

    supervisorctl start cvt-service

  3. Verify that the service is running:

    Copy
    Copied!
                

    supervisorctl status cvt-service

Note

While in legacy mode, automatic features (e.g., topology loading and validation) are disabled. The web UI remains accessible. Restarting the service restores full automatic functionality.

Command Line Options

The CVT service supports several command line options:

Command

Description

cvt-service --daemon

Runs the CVT service as a background daemon.

cvt-service --kill-other-sessions

Terminates any other running CVT sessions before starting a new one.

cvt-service --version

Displays the current version of the CVT service.

cvt-service

Starts the CVT service in interactive mode, useful for debugging.

Topology Loading

The CVT service can automatically load topology files on startup based on the STARTUP_TOPOLOGY configuration:

Load Last Topology

STARTUP_TOPOLOGY=last
  • Loads the most recently used topology from history

  • Supports all topology formats (.topo, .dot, .xlsx, .json)

Load Specific File

STARTUP_TOPOLOGY=/path/to/topology.topo
  • Loads a specific topology file on startup

  • Supports absolute and relative paths

  • Auto-detects file format based on extension

Supported Formats

  • .topo: Native topology format

  • .dot: Graphviz DOT format

  • .xlsx: Unified topology Excel format

  • .json: JSON topology format

Automatic Validation

Enable automatic validation start after topology loading:

AUTO_START_VALIDATION=true

Behavior:

  • Only starts if topology was successfully loaded

  • Uses validation settings from history

  • Logs all activities for monitoring

Signal Handling

The service handles system signals gracefully:

  • SIGTERM: Graceful shutdown with cleanup

  • SIGINT:

    • Daemon mode: Graceful shutdown

    • Interactive mode: Cancel current operation (double Ctrl-C to exit)

Monitoring and Logging

Log Files

All service logs are centrally managed:

/cable_bringup_root/log/<p></p>├── cvt-service.log              # Main service logs<p></p>├── cv_controller_service.log    # Controller logs<p></p>├── apache2/<p></p>│   ├── access.log              # Web access logs<p></p>│   └── error.log               # Apache error logs<p></p>└── supervisord.log             # Supervisor logs

Log Rotation

Logs are automatically rotated with the following settings:

  • Max size: 10MB per log file

  • Backups: 7 historical files

  • Format: Timestamped entries with log levels

Health Monitoring

Monitor service health using:

Command

Description

supervisorctl status

Checks the status of all services managed by Supervisor.

supervisorctl tail -f cvt-service

Monitors CVT service logs in real-time.

systemctl status apache2

Displays the current status of the Apache web server.

htop

Interactive tool to monitor system processes and resource usage.

iostat -x 1

Displays detailed I/O statistics for devices every 1 second.

Common Issues

Issue

Troubleshooting

Service Won't Start

  1. Check Supervisord Status:

    supervisorctl statussystemctl status supervisor
  2. Check Log Files:

    tail -f /cable_bringup_root/log/cvt-service.logtail -f /var/log/supervisor/supervisord.log
  3. Verify Configuration:

    supervisorctl rereadsupervisorctl update

Port Conflicts

  1. Check Port Usage:

    netstat -tlnp | grep -E ':(443|8251|8252)'lsof -i :443
  2. Update Port Configuration:

    # Edit environment variablesexport CVT_WEB_SERVICE_PORT=8253export CV_CONTROLLER_PORT=8254

Permission Issues

  1. Check File Permissions:

    ls -la /cable_bringup_root/ls -la /etc/cablevalidation/
  2. Fix Ownership:

    chown -R root:root /cable_bringup_root/chmod -R 755 /cable_bringup_root/

Memory/Performance Issues

  1. Monitor Resource Usage:

    htopfree -hdf -h
  2. Adjust Worker Limits:

    # In cvt_env.confCVT_MAX_WORKERS=15  # Reduce for limited memory

Topology Loading Failures

  1. Check File Paths:

    ls -la /path/to/topology/file
  2. Validate File Format:

    file topology.topohead -n 10 topology.topo
  3. Check History:

    # View topology loading history in logsgrep "Loading topology" /cable_bringup_root/log/cvt-service.log

Debug Mode

For detailed debugging, run CVT in interactive mode:

Command

Description

supervisorctl stop cvt-service

Stops the CVT service managed by Supervisor.

/usr/local/bin/cvt-service

Runs the CVT service in interactive mode.

PYTHONUNBUFFERED=1 /usr/local/bin/cvt-service

Runs the CVT service interactively with debug logging enabled.

Log Analysis

Analyze logs for common patterns:

Command

Description

grep -i error /cable_bringup_root/log/cvt-service.log

Searches the CVT service log for error messages.

grep -i "starting|daemon" /cable_bringup_root/log/cvt-service.log

Monitors the startup sequence of the CVT service.

grep -i "topology|load" /cable_bringup_root/log/cvt-service.log

Checks log entries related to topology loading.

grep -i "validation|start|stop" /cable_bringup_root/log/cvt-service.log

Monitors validation activities and their start/stop events.

Flow

Security

  1. Use HTTPS: Always use HTTPS for production deployments

  2. Strong Authentication: Configure strong passwords for web users

  3. SSH Keys: Use SSH key-based authentication for agent deployment

  4. Firewall: Restrict access to necessary ports only

  5. Regular Updates: Keep CVT and system packages updated

Performance

  1. Resource Allocation: Ensure adequate CPU and memory resources

  2. Worker Tuning: Adjust CVT_MAX_WORKERS based on system capacity

  3. Log Management: Implement log rotation and archival

  4. Monitoring: Set up comprehensive monitoring and alerting

Reliability

  1. Backup Configuration: Regularly backup configuration files and data

  2. Health Checks: Implement automated health monitoring

  3. Graceful Shutdown: Always use proper shutdown procedures

  4. Recovery Procedures: Document and test disaster recovery procedures

Maintenance

  1. Regular Monitoring: Check service status and logs regularly

  2. Capacity Planning: Monitor resource usage and plan for growth

  3. Update Procedures: Establish procedures for updates and patches

  4. Documentation: Keep configuration and operational documentation current

Development and Testing

  1. Staging Environment: Use a staging environment for testing changes

  2. Configuration Management: Use version control for configuration files

  3. Automated Testing: Implement automated testing for service functionality

  4. Rollback Procedures: Have rollback procedures ready for failed deployments

© Copyright 2025, NVIDIA. Last updated on Nov 12, 2025