Usage ===== Basic Usage ----------- Start the server with a UNIX socket and the desired backend: .. code-block:: bash # Loopback backend (no external dependencies) ./build/rocm-ernic \ --socket /tmp/vfio-user-rocm-ernic.sock \ --backend loopback # In-process NVMe-oF controller the guest can connect to ./build/rocm-ernic \ --socket /tmp/vfio-user-rocm-ernic.sock \ --backend nvmeof:size=1G,bs=4096 --tap ernic0 # No backend (minimal stubs) ./build/rocm-ernic \ --socket /tmp/vfio-user-rocm-ernic.sock \ --backend none # Ethernet attached to a host TAP ./build/rocm-ernic \ --socket /tmp/vfio-user-rocm-ernic.sock \ --backend loopback --tap ernic0 Whoever connects to the socket drives the device as the guest's QEMU, so the server makes it mode ``0600``: only the user the server runs as, and root, can connect. Run QEMU as the same user as the server, or as root. To let a QEMU that runs as another user connect, name a group it is in with ``--socket-group`` (``-G``); the server gives the socket that group and makes it ``0660`` before it serves it. The server must run as root or be a member of the group: .. code-block:: bash sudo ./build/rocm-ernic \ --socket /run/vfio-user-rocm-ernic.sock \ --socket-group kvm --backend loopback Changing the socket's group by hand after the server starts works too, but the next server started on that path, after a restart or a hot reload, creates a new socket without it. The server refuses to start if the socket ends up with different permissions, which a default ACL on its directory can cause, or with an ACL of its own, which it inherits from a default ACL with more than the owner, group, and other entries. An entry naming another user would let that user connect without the mode showing it. Remove the default ACL (``setfacl -k DIR``), or put the socket in another directory. Put the socket in a directory that only the server's user and root can write to, such as ``/run/rocm-ernic``, which the service uses, or one of your own. Don't put it in a directory other users can write to, such as ``/tmp``, even though the server's default socket and the examples here are there. The sticky bit on ``/tmp`` stops other users removing or replacing the server's socket, but not creating their own at the same name first, while the server isn't running. A server that isn't root then can't remove it and fails to start, and a QEMU that connects to it gives that user the guest's memory, which vfio-user shares with whatever serves the device. In a directory others can write to that isn't sticky, they can also replace the server's socket. With ``--socket-group``, the group also has to be able to reach the socket through its directory, which ``/run/rocm-ernic`` allows (mode ``0755``) and ``$XDG_RUNTIME_DIR`` (``0700``) doesn't. A directory of your own can have that group and mode ``0710``, which lets the group reach the socket but not list or change the directory: .. code-block:: bash sudo install -d -o root -g kvm -m 0710 /run/vfio-user The Emulated Device ------------------- The server emulates an AMD Pensando ionic NIC (``1dd8:1002``) with subsystem ``1dd8:5400``, driven by the upstream Linux ``ionic`` and ``ionic_rdma`` modules. Once the ``1dd8:5400`` entry is published in ``pci.ids``, ``lspci`` names that subsystem ``ROCm Emulated RDMA NIC`` after ``update-pciids``; until then it displays ``Device [1dd8:5400]``. There is nothing to select. ``--tap IFNAME`` (short ``-T``) attaches the emulated Ethernet interface to an existing host TAP. ``IFNAME`` has at most 15 characters, the Linux limit; the server refuses a longer one rather than shortening it. Create the TAP up front, owned by the user running the server: .. code-block:: bash sudo ip tuntap add dev ernic0 mode tap user "$USER" The two paths are independent: ``--backend`` governs the RDMA data path, ``--tap`` the Ethernet one. See :doc:`ionic` for the full picture. ``nvmeof`` is the odd one out: rather than carrying guest RDMA traffic somewhere, it answers it, presenting an NVMe over Fabrics target the guest connects to with stock ``nvme connect -t rdma``. See :doc:`nvmeof`. Log Levels ---------- ``--log-level`` (short ``-L``) sets how much the server prints. It can also be set with the ``ERNIC_LOG_LEVEL`` environment variable; the command line wins when both are given. .. list-table:: :header-rows: 1 :widths: 15 85 * - Level - Output * - ``none`` - Nothing at all. * - ``error`` - ``ERROR:`` lines only. * - ``warn`` - **Default.** ``ERROR:`` and ``WARN:`` lines, plus a short startup/shutdown banner. * - ``info`` - Adds the per-operation ``INFO:`` lines (BAR writes, QP operations, ARP/ICMP packets, TCP mesh events). * - ``debug`` - Everything, including libvfio-user debug output. ``-v`` / ``--verbose`` remains as shorthand for ``--log-level debug``. .. code-block:: bash # Quiet by default -- only warnings and errors ./build/rocm-ernic --backend loopback # Full per-operation tracing ERNIC_LOG_LEVEL=info ./build/rocm-ernic --backend loopback Launching a VM -------------- Attach the emulated device to a QEMU virtual machine using the ``vfio-user-pci`` transport. QEMU 10.1 or later is required. .. code-block:: bash qemu-system-x86_64 \ -machine q35,accel=kvm \ -cpu EPYC \ -smp cpus=2 \ -object memory-backend-memfd,\ id=mem0,share=on,size=2048M \ -machine memory-backend=mem0 \ -nographic \ -drive if=virtio,format=qcow2,\ file=vm-image.qcow2 \ -netdev user,id=net0,\ hostfwd=tcp::2222-:22 \ -device virtio-net-pci,netdev=net0 \ -device '{"driver":"vfio-user-pci",\ "socket":{"path":"/tmp/vfio-user-rocm-ernic.sock",\ "type":"unix"}}' Inside the guest, load the upstream modules and verify: .. code-block:: bash sudo modprobe ionic sudo modprobe ionic_rdma sudo update-pciids lspci -nnv -d 1dd8:1002 | grep '\[1dd8:5400\]' ibv_devices Statistics Collection --------------------- The server can collect detailed statistics about doorbell rings, WQE processing, and completion queue entries. Statistics are written to a file approximately every second while the server is running. .. code-block:: bash ./build/rocm-ernic \ --socket /tmp/vfio-user-rocm-ernic.sock \ --backend loopback \ --stats-file /tmp/rocm_ernic_stats.txt # Monitor statistics in real-time watch -n 0.5 cat /tmp/rocm_ernic_stats.txt The statistics file includes: - Device-level statistics (commands, register reads/writes, UAR writes, interrupts) - Per-QP statistics: - Doorbell rings (send, receive, SRQ) - WQEs processed (total and by opcode type) - CQEs posted - Continuation callbacks scheduled Statistics are automatically written on server exit (``SIGINT`` / ``SIGTERM``) in addition to the periodic updates. Each update is written to a temporary file next to the stats file (its name with six random characters appended), which then replaces the stats file, so a reader always sees one whole update. The server therefore needs write permission on the stats file's directory. The file is mode ``0644`` whatever the umask. If an update can't be written, the server logs why and leaves the previous file in place. Development Workflow: Hot Reload -------------------------------- During development you often need to rebuild the server and test changes against a running VM. The hot-reload workflow lets you do this without tearing down and rebooting the VM, cutting the typical iteration cycle from 60+ seconds down to roughly 5--15 seconds (depending on build time). The recommended approach is the ``ernicctl`` service CLI, which manages server instances, VMs, QMP sockets, and driver distribution through a single tool. See :doc:`service` for full documentation. Quick example using ernicctl ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash # Start the service (launches all instances) sudo systemctl start rocm-ernic # Launch a VM for instance 1 sudo ernicctl vm-launch 1 # After making code changes, hot-reload instance 1 sudo ernicctl hot-reload 1 --update-driver Standalone hot-reload script ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``scripts/hot-reload.sh`` script is also available for standalone use outside the systemd service. It uses QEMU's QMP (QEMU Machine Protocol) to hot-unplug the ``vfio-user-pci`` device, restart the server, and hot-plug a fresh device -- all while the VM keeps running. **Host dependency:** ``socat`` is required for QMP communication. .. code-block:: bash sudo apt install socat .. code-block:: bash ./scripts/hot-reload.sh Useful options: - ``--no-build`` -- skip the rebuild step - ``--build-only`` -- rebuild without cycling the device - ``--update-driver`` -- rebuild and reload the guest kernel module - ``--backend TYPE`` -- choose a different backend (default: ``loopback``)