NVIDIA DOCA SimX Service Guide
NOT MEANT FOR PUBLIC DEVZONE! DO NOT PUBLISH!
Contents:
This guide provides configuration instructions for DOCA SimX NIC and explains how to use SimX NIC over docker.
SimX is a software simulation of NVIDIA PCIe devices that use virtualization:
Virtual NVIDIA® ConnectX® device in VM
Seamless interaction with real driver and software stack
SimX replaces both the hardware and firmware components.
Docker with version 20.10 or later.
RHEL 7.5, Fedora 34, Ubuntu 20.04.
Network bridge to network management interface (must be configured in an JSON file under the
management_bridgefield).Python 3.11 or above
The next python packages need to be installed : pydantic, pydantic_extra_types, uvicorn & httpx
To perform the following, you must have a tar file distribution of SimX. This tar file can be downloaded from SimX's NGC page.
After downloading the tar file, please follow these steps to create a SimX docker image:
Extract the tar file:
tar-xvf <tar_file>Enter the
simx_dockerdirectory:cdsimx_dockerGenerate the container image using the helper script:
./simx_dockbuild_from_tar.sh --name <container-image-name>
A docker image is created. This should be used as the <IMAGE> argument when starting the SimX docker container.
Parameters File
The parameters file is a standard JSON text file that contains sections with key-value pairs for properties.
simx_nic_docker.jsonis the default parameters file. You may use it as a reference to build your own file(s).
External Connectivity to VM
If you have a DHCP server you can set management_bridge to a bridge that is connected to a trusted interface.
To SSH to the VM and to get internet connectivity, the management_interface_mac in the JSON file must be modified to a specific value. Do not leave it to be a random value.
The DHPC server will get the IP address from the configured MAC address.
Start SimX Docker Container
manage_simx_nic_docker.py run <CONTAINER_NAME> <IMAGE> <PARAMS_FILE>
Where:
CONTAINER_NAME– name of the docker container.IMAGE– name or ID of docker image.PARAMS_FILE– full path to JSON file of parameters.
All parameters are mandatory.
Stop SimX Docker Container
manage_simx_nic_docker.py stop <CONTAINER_NAME>
Where:
CONTAINER_NAME– name of running SimX docker container.
This command stops the running container.
Resume SimX Docker Container
manage_simx_nic_docker.py resume <CONTAINER_NAME>
Where:
CONTAINER_NAME– name of running SimX docker container.
This command starts a container that was previously stopped using the command. It sets the network configuration of the host and configures internal environment in docker container, then starts the VM(s) according to the configurations provided in the run command.
Remove SimX Docker Container
manage_simx_nic_docker.py remove <CONTAINER_NAME>
Where:
CONTAINER_NAME– name of running SimX docker container.
This command removes the container and all network configurations in the host.
Connect to VM Console Through SimX Container
To connect to the VM running on the SimX docker container, the VM must be in running state and have a serial connection.
telnet localhost <TELNET_PORT_NUMBER>
Where:
TELNET_PORT_NUMBER - telnet port to relevant VM
Get telnet ports
manage_simx_nic_docker.py console_ports_info <CONTAINER_NAME>
Where:
CONTAINER_NAME– name of running SimX docker container.
Restart VM in SimX Container
To restart a VM in the SimX docker container:
Try to gracefully shutdown the VM.
Wait a few seconds and make sure the VM is down.
If the VM is still running, then force VM shutdown.
Start VM:
manage_simx_nic_docker.py reboot <CONTAINER_NAME>
Where:
CONTAINER_NAME– name of running SimX docker container
Shutdown VM in SimX Container
To shutdown VM with ConnectX:
./manage_simx_nic_docker.py shutdown <container name>
To shutdown VM with Bluefield run the next:
First, run on DPU side:
./manage_simx_nic_docker.py shutdown <DPU container name>
Then, run on host side:
./manage_simx_nic_docker.py shutdown <Host container name>
It is recommended to run on host side only after DPU side is down (i.e., only after shutdown script on DPU returns with success).
Save SimX Docker Container
Save SimX docker container to the docker image. This command can be useful for reproduction.
manage_simx_nic_docker.py save <CONTAINER_NAME> <DOCKER_IMAGE>
Where:
CONTAINER_NAME– name of running SimX docker container.DOCKER_IMAGE- name of docker image on which to save the docker container.
Print Default SimX Configuration File
Print the default SimX configuration file (simx-qemu.cfg) from the SimX docker image.
manage_simx_nic_docker.py print_config <IMAGE>
Where:
IMAGE– name or ID of SimX docker image.
Print Information about SimX Docker Image
Print some information about the SimX docker image (e.g., version of SimX, number of internal images it contains, etc).
manage_simx_nic_docker.py info <IMAGE>
Where:
IMAGE– name or ID of SimX docker image.
Preparing VM Image for Docker
It is recommended to use QCOW2 format but other formats may also work.
Any Ubuntu VM must have the serial-getty@ttyS0.service enabled to use telnet access
Please contact NVIDIA Enterprise Support for further information.
Permission problem
In case that you get the next error:
error: Cannot access storage file '/vm/<VM_image>.img' (as uid:64055, gid:109): Permission denied
You will have to enter the docker container with the next command:
docker exec -ti <docker container name> bash
And add to the next file /etc/libvirt/qemu.conf:
user = "root"
group = "root"
Then restart libvirtd with the next command:
systemctl restart libvirtd
After this change start again the VM.
Create VM Image for Docker Based on BFB
To create a VM image for a docker based on a BFB file, use the git repository https://github.com/Mellanox/bfb2image. This script takes a BFB and creates a VM image from it.
Usage:
./bfb_to_raw_img.sh -bfb <BFB-file> -out <output-directory> -verbose <true-for-printing-logs> -os <OS included in the BFB>
Ubuntu/Centos VM Mount
If the Ubuntu/CentOS VM image does not have local directories mounted, you may mount a specific directory from the host:
On the host:
Go to
/etc/exportsand add a new line:<path-to-directory-to-be-mounted> <vm-ip>(rw)
Run:
sudosystemctl restart nfs-server
On the VM:
Edit
/etc/fstab/and add a new line:<
hostname>:<directory-path-to-mount-on-host> <where-to-mount-on-vm>Defaults are 0 0.
Reboot the VM
The VM must have a valid IP address with connectivity to mount the directory.
Connection to VM
The VM image must have tty service enabled before running with docker:
sudo systemctl enable serial-getty@ttyS0.service
sudo systemctl start serial-getty@ttyS0.service
Check or install docker on the server.
Version 20.10 and above is recommended.
If the host is running Ubuntu with standard configuration and SimX version is below 23.7-1064, then run the following commands:
sudo apt purge snapd
sudo systemctl restart apparmor
sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/usr.sbin.libvirtd
sudo reboot now