Usage

Usage#

Basic Usage#

Start the server with a UNIX socket and the desired backend:

# Loopback backend (no external dependencies)
./build/rocm-ernic \
  --socket /tmp/vfio-user-rocm-ernic.sock \
  --backend loopback

# In-process NVMe-oF controller the guest can connect to
./build/rocm-ernic \
  --socket /tmp/vfio-user-rocm-ernic.sock \
  --backend nvmeof:size=1G,bs=4096 --tap ernic0

# No backend (minimal stubs)
./build/rocm-ernic \
  --socket /tmp/vfio-user-rocm-ernic.sock \
  --backend none

# Ethernet attached to a host TAP
./build/rocm-ernic \
  --socket /tmp/vfio-user-rocm-ernic.sock \
  --backend loopback --tap ernic0

Whoever connects to the socket drives the device as the guest’s QEMU, so the server makes it mode 0600: only the user the server runs as, and root, can connect. Run QEMU as the same user as the server, or as root. To let a QEMU that runs as another user connect, name a group it is in with --socket-group (-G); the server gives the socket that group and makes it 0660 before it serves it. The server must run as root or be a member of the group:

sudo ./build/rocm-ernic \
  --socket /run/vfio-user-rocm-ernic.sock \
  --socket-group kvm --backend loopback

Changing the socket’s group by hand after the server starts works too, but the next server started on that path, after a restart or a hot reload, creates a new socket without it.

The server refuses to start if the socket ends up with different permissions, which a default ACL on its directory can cause, or with an ACL of its own, which it inherits from a default ACL with more than the owner, group, and other entries. An entry naming another user would let that user connect without the mode showing it. Remove the default ACL (setfacl -k DIR), or put the socket in another directory.

Put the socket in a directory that only the server’s user and root can write to, such as /run/rocm-ernic, which the service uses, or one of your own. Don’t put it in a directory other users can write to, such as /tmp, even though the server’s default socket and the examples here are there. The sticky bit on /tmp stops other users removing or replacing the server’s socket, but not creating their own at the same name first, while the server isn’t running. A server that isn’t root then can’t remove it and fails to start, and a QEMU that connects to it gives that user the guest’s memory, which vfio-user shares with whatever serves the device. In a directory others can write to that isn’t sticky, they can also replace the server’s socket.

With --socket-group, the group also has to be able to reach the socket through its directory, which /run/rocm-ernic allows (mode 0755) and $XDG_RUNTIME_DIR (0700) doesn’t. A directory of your own can have that group and mode 0710, which lets the group reach the socket but not list or change the directory:

sudo install -d -o root -g kvm -m 0710 /run/vfio-user

The Emulated Device#

The server emulates an AMD Pensando ionic NIC (1dd8:1002) with subsystem 1dd8:5400, driven by the upstream Linux ionic and ionic_rdma modules. Once the 1dd8:5400 entry is published in pci.ids, lspci names that subsystem ROCm Emulated RDMA NIC after update-pciids; until then it displays Device [1dd8:5400]. There is nothing to select.

--tap IFNAME (short -T) attaches the emulated Ethernet interface to an existing host TAP. IFNAME has at most 15 characters, the Linux limit; the server refuses a longer one rather than shortening it. Create the TAP up front, owned by the user running the server:

sudo ip tuntap add dev ernic0 mode tap user "$USER"

The two paths are independent: --backend governs the RDMA data path, --tap the Ethernet one. See The ionic Device for the full picture.

nvmeof is the odd one out: rather than carrying guest RDMA traffic somewhere, it answers it, presenting an NVMe over Fabrics target the guest connects to with stock nvme connect -t rdma. See NVMe-oF Controller.

Log Levels#

--log-level (short -L) sets how much the server prints. It can also be set with the ERNIC_LOG_LEVEL environment variable; the command line wins when both are given.

Level

Output

none

Nothing at all.

error

ERROR: lines only.

warn

Default. ERROR: and WARN: lines, plus a short startup/shutdown banner.

info

Adds the per-operation INFO: lines (BAR writes, QP operations, ARP/ICMP packets, TCP mesh events).

debug

Everything, including libvfio-user debug output.

-v / --verbose remains as shorthand for --log-level debug.

# Quiet by default -- only warnings and errors
./build/rocm-ernic --backend loopback

# Full per-operation tracing
ERNIC_LOG_LEVEL=info ./build/rocm-ernic --backend loopback

Launching a VM#

Attach the emulated device to a QEMU virtual machine using the vfio-user-pci transport. QEMU 10.1 or later is required.

qemu-system-x86_64 \
  -machine q35,accel=kvm \
  -cpu EPYC \
  -smp cpus=2 \
  -object memory-backend-memfd,\
id=mem0,share=on,size=2048M \
  -machine memory-backend=mem0 \
  -nographic \
  -drive if=virtio,format=qcow2,\
file=vm-image.qcow2 \
  -netdev user,id=net0,\
hostfwd=tcp::2222-:22 \
  -device virtio-net-pci,netdev=net0 \
  -device '{"driver":"vfio-user-pci",\
"socket":{"path":"/tmp/vfio-user-rocm-ernic.sock",\
"type":"unix"}}'

Inside the guest, load the upstream modules and verify:

sudo modprobe ionic
sudo modprobe ionic_rdma
sudo update-pciids
lspci -nnv -d 1dd8:1002 | grep '\[1dd8:5400\]'
ibv_devices

Statistics Collection#

The server can collect detailed statistics about doorbell rings, WQE processing, and completion queue entries. Statistics are written to a file approximately every second while the server is running.

./build/rocm-ernic \
  --socket /tmp/vfio-user-rocm-ernic.sock \
  --backend loopback \
  --stats-file /tmp/rocm_ernic_stats.txt

# Monitor statistics in real-time
watch -n 0.5 cat /tmp/rocm_ernic_stats.txt

The statistics file includes:

  • Device-level statistics (commands, register reads/writes, UAR writes, interrupts)

  • Per-QP statistics:

    • Doorbell rings (send, receive, SRQ)

    • WQEs processed (total and by opcode type)

    • CQEs posted

    • Continuation callbacks scheduled

Statistics are automatically written on server exit (SIGINT / SIGTERM) in addition to the periodic updates.

Each update is written to a temporary file next to the stats file (its name with six random characters appended), which then replaces the stats file, so a reader always sees one whole update. The server therefore needs write permission on the stats file’s directory. The file is mode 0644 whatever the umask. If an update can’t be written, the server logs why and leaves the previous file in place.

Development Workflow: Hot Reload#

During development you often need to rebuild the server and test changes against a running VM. The hot-reload workflow lets you do this without tearing down and rebooting the VM, cutting the typical iteration cycle from 60+ seconds down to roughly 5–15 seconds (depending on build time).

The recommended approach is the ernicctl service CLI, which manages server instances, VMs, QMP sockets, and driver distribution through a single tool. See systemd Service and ernicctl for full documentation.

Quick example using ernicctl#

# Start the service (launches all instances)
sudo systemctl start rocm-ernic

# Launch a VM for instance 1
sudo ernicctl vm-launch 1

# After making code changes, hot-reload instance 1
sudo ernicctl hot-reload 1 --update-driver

Standalone hot-reload script#

The scripts/hot-reload.sh script is also available for standalone use outside the systemd service. It uses QEMU’s QMP (QEMU Machine Protocol) to hot-unplug the vfio-user-pci device, restart the server, and hot-plug a fresh device – all while the VM keeps running.

Host dependency: socat is required for QMP communication.

sudo apt install socat

./scripts/hot-reload.sh

Useful options:

  • --no-build – skip the rebuild step

  • --build-only – rebuild without cycling the device

  • --update-driver – rebuild and reload the guest kernel module

  • --backend TYPE – choose a different backend (default: loopback)