The ionic Device

The ionic Device#

rocm-ernic presents a single PCIe device to the guest:

VID:DID / SSVID:SDID

Guest driver

1dd8:1002 / 1dd8:5400

Upstream Linux ionic.ko + ionic_rdma.ko, with the UC queue-pair patch in ansible/files/patches/ applied, and the upstream providers/ionic in rdma-core

The server emulates the register and queue protocol of the AMD Pensando ionic NIC, so the guest runs a driver that is already in mainline Linux rather than one that only exists here. The driver source is near-stock — one small RDMA patch — but the guest itself is not a stock cloud image. It needs a mainline kernel matching IONIC_KERNEL_REF and the headers to build against, which no Ubuntu release ships. Nothing in this repo bakes such an image either: the guest is the published ionic flavour of batesste-ci-images, fetched by scripts/fetch-guest-image.sh. See the warning below.

The Pensando vendor ID 0x1dd8 is kept, and the emulated PF now uses the upstream device ID 0x1002. The subsystem ID 0x5400 distinguishes this function from physical PFs with the same vendor/device pair. Once its entry is published in pci.ids, lspci will name it ROCm Emulated RDMA NIC after update-pciids. The underlying vendor/device pair remains intact for the upstream driver.

Starting the Server#

./build/rocm-ernic \
  --socket /tmp/vfio-user-rocm-ernic.sock \
  --backend loopback \
  --tap ernic0 \
  --log-level info

There is no flag to select the device. --tap (short -T) attaches the emulated Ethernet LIF to a host TAP interface.

All the RDMA backends (loopback, tcp, none) work the same way: the backend choice governs the RDMA data path, while --tap governs the Ethernet data path.

BAR Layout#

The BAR layout matches what ionic_dev_setup() in the upstream driver expects:

BAR

Size

Purpose

BAR 0

64 KB

32 KB device register window (device info, device command, interrupt control) plus the MSI-X table and PBA above it

BAR 2

4 MB

Doorbell pages; the offset within a page encodes the queue type, the page index the process ID

The MSI-X table has room for 128 vectors; the device advertises the profile’s queues.intr_vectors of them (32 under compat).

NIC Profiles#

A NIC profile is the shape of the emulated NIC: what it reports about itself (ASIC type and revision, firmware version), its link (speed, transceiver, largest MTU) and the queues and interrupt vectors it offers. Without --nic-profile the server runs the built-in compat profile, which is the device described above, byte for byte.

# layer a file over compat; later layers win, key by key
./build/rocm-ernic --tap ernic0 --nic-profile ./fast.ini

# show the merged profile and exit
./build/rocm-ernic --nic-profile ./fast.ini --print-profile

--nic-profile can be given more than once. Each one names either a built-in profile (a bare name: compat, small-vm, generic-100g, generic-400g) or a file (a name containing a / or ending in .ini). The layers apply in order on top of compat. A profile is an ini file:

# fast.ini
[identity]
fw_version = 1.8.0-E-32

[link]
speed_mbps = 400000
xcvr = fiber

Validation is strict. An unknown section or key, a duplicate key, a value of the wrong type or a value out of range stops the server with the file, line and key named, so a typo can’t quietly leave a default in place. A layer that fails changes nothing.

--print-profile writes the merged profile as an ini file that loads back to the same values. Its header names the layers and a 64-bit hash of the values; the startup log and the nic_profile line in the stats file carry the same chain and hash, so two runs can be checked for the same device shape.

Key

Values

compat

Guest effect

identity.asic_type

none, capri, elba, giglio, salina

none

dev_info ASIC type. The driver runs its periodic doorbell re-check (ionic_doorbell_wa()) only for none and elba

identity.asic_rev

0 to 255

0

dev_info ASIC revision

identity.fw_version

1 to 31 printable ASCII characters; no \, no leading or trailing space

rocm-ernic-1.0

ethtool -i firmware-version

link.speed_mbps

1000, 10000, 25000, 40000, 50000, 100000, 200000, 400000

100000

port and LIF link speed: ethtool Speed, the RDMA port rate

link.xcvr

copper, fiber, none

copper

transceiver: ethtool Port is Direct Attach Copper, FIBRE or None

link.mtu_max

1500 to 9582

9198

the largest MTU the guest may set. The LIF reports a maximum frame of mtu_max + 18, and the port’s configured MTU is the smaller of mtu_max and 9000

stats.interval_ms

10 to 60000

1000

how often the LIF counters are rewritten in guest memory, which bounds how stale ip -s link can be. --stats-interval-ms overrides every layer

queues.eth

1 to 64

4

Ethernet queue pairs the LIF offers: the most ethtool -l can set combined to, and the rx_N_*/tx_N_* rows in ethtool -S. --eth-queues overrides every layer

queues.rdma_eqs

4 to 32

32

RDMA event queues; the RDMA driver needs at least 4

queues.intr_vectors

auto, or 2 to 128

32

MSI-X vectors the device advertises. auto is 1 + queues.eth + queues.rdma_eqs, enough for every queue and EQ to have its own

The built-in profiles:

Name

Shape

compat

the device described above: 4 queues, 32 EQs, 32 vectors, 100 Gb/s copper

small-vm

compat cut down for a guest with a few vCPUs: 2 queues, 4 EQs, auto (7) vectors

generic-100g

16 queues, 32 EQs, auto (49) vectors, 100 Gb/s fiber

generic-400g

64 queues, 32 EQs, auto (97) vectors, 400 Gb/s fiber

The guest driver (ionic_lif_size()) wants one vector for the admin and notify queues, one per queue pair and one per EQ, each count capped at the online vCPUs. If that is more than the device advertises it halves its notify queues, then its EQs, then its queue pairs until it fits. A profile whose queues.intr_vectors is below 1 + queues.eth + queues.rdma_eqs therefore works, but a guest with enough vCPUs gets fewer EQs or queues than the profile names; the server says so at startup. compat is such a profile (1 + 4 + 32 = 37 > 32), kept that way so its guest sees exactly what it saw before profiles existed.

Ethernet and TCP/IP#

The emulated LIF is a working Ethernet NIC. A Linux TAP interface is used as the host-side backend, which is the cheapest way to give the guest a real, routable Ethernet segment: the host end is an ordinary netdev, so ARP, ICMP, DHCP, and TCP all work against the host network stack with no protocol emulation inside the server.

Create a persistent TAP owned by the user that will run the server, then address it:

sudo ip tuntap add dev ernic0 mode tap user "$USER"
sudo ip addr add 192.168.77.1/24 dev ernic0
sudo ip link set ernic0 up

Start the server with --tap ernic0, boot the guest, and configure the guest end of the segment:

# in the guest
sudo ip addr add 192.168.77.2/24 dev enp0s4
sudo ip link set enp0s4 up
ping 192.168.77.1

Two Guests on One Bridge#

A single TAP with an address on it is enough for host-to-guest traffic, but every two-VM test in ansible/playbooks/ needs guest-to-guest IP on 192.168.200.x: sanity-tests.yml pings it, tcp-performance-tests.yml runs iperf3 over it, and performance-tests.yml uses it for the out-of-band exchange in perftest. That traffic leaves through the TAP rather than through the rocm-ernic TCP mesh, so each instance needs its own TAP and all of them need to share one host bridge:

sudo ip link add ernicbr0 type bridge
sudo ip link set ernicbr0 up
for n in 1 2; do
  sudo ip tuntap add dev "ernic-tap${n}" mode tap user "$USER"
  sudo ip link set "ernic-tap${n}" master ernicbr0 up
done

The ernic_host_setup role does this from tasks/tap.yml using ernic_tap_prefix, ernic_tap_bridge and ernic_tap_owner; the launcher then passes --tap ${ERNIC_TAP_PREFIX}${id} to each instance. A missing TAP is a warning rather than an error there, so a loopback-only host still starts — but the guest comes up with no Ethernet, and every guest-to-guest test then fails without saying why.

The bridge carries no IP of its own by default; the guests address each other directly across it. Set ernic_tap_bridge_ip if the host needs to join the segment for debugging.

The transmit path gathers the head fragment plus any scatter-gather elements out of guest memory and writes the frame to the TAP file descriptor. The receive path drains the TAP from the server’s main loop and DMAs each frame into a descriptor the guest has posted on one of its receive queues. Both directions run on the thread that owns the vfio-user context, because that is the only thread permitted to DMA into guest memory; no extra thread is created.

Received frames are spread over the receive queues the guest has enabled (Q_CONTROL; ethtool -L changes the set) by a fixed hash of the flow, so one flow always lands on one queue and each queue’s interrupt fires on its own vector. The hash is 32-bit FNV-1a over the IPv4 or IPv6 source and destination address, the protocol, and for TCP, UDP and SCTP the two ports, after up to two VLAN tags; fragments leave the ports out so every fragment of a datagram stays together. Other frames hash on the two MAC addresses. The queue is the hash modulo the number of enabled queues, in index order. The guest’s RSS key and indirection table are accepted and ignored. A frame from the TAP whose queue has no buffer posted is dropped and counted in rx_queue_empty; it does not fall back to another queue. A frame an in-process endpoint (the S3 store’s control plane, or the P4 engine) queues for the guest is held until the queue it steers to has a buffer, and the queued frames behind it wait too.

Note

The emulated LIF advertises no checksum, TSO, or scatter-gather offloads, so the guest stack always hands down linear skbs. This keeps the descriptor handling simple at the cost of transmit throughput.

Statistics#

The ionic driver never asks the device for Ethernet counters. At LIF_INIT it hands the device a struct ionic_lif_info in its own memory, and from then on reads struct ionic_lif_stats (byte 320 of that block) whenever ip -s link, /proc/net/dev or ethtool -S asks. The emulator counts every frame the LIF moves and rewrites that block from the main loop: once straight after LIF_INIT, then every stats.interval_ms (1000 by default). A real NIC’s firmware does the same.

# rewrite the counters every 100 ms
./build/rocm-ernic --tap ernic0 --stats-interval-ms 100

Frames are classed by destination MAC: the group bit makes a frame multicast, all-ones makes it broadcast, anything else is unicast. What moves:

Event

Counters

frame sent or received

{tx,rx}_{ucast,mcast,bcast}_{bytes,packets}; the driver sums them into ip -s link packets and bytes

receive with no buffer posted, or the queue of frames for the guest full

rx_*_drop_* and rx_queue_empty (rx_dropped and rx_over_errors)

frame larger than the posted buffer

rx_*_drop_* and rx_desc_data_error

DMA to or from guest memory failed

{tx,rx}_dma_error

the host backend refused a transmit

tx_*_drop_*

descriptor unreadable, or nonsensical

tx_desc_fetch_error, tx_desc_data_error

doorbell on a queue that was never initialized

tx_queue_disabled

RDMA payload sent or landed

{tx,rx}_rdma_ucast_{bytes,packets}, one packet per message

Frames are counted whether or not the server has a TAP: without one, transmitted frames are counted and discarded. The counters restart at zero on a device reset. After LIF_RESET or a device reset nothing more is written into guest memory until the next LIF_INIT, because the driver may have freed the block.

The stats file (--stats-file) carries the same counters in a LIF Statistics section, one line per named field of struct ionic_lif_stats whether or not it is zero, so a test can check what the guest reports against what the emulator counted. They agree to within one interval.

Not counted yet: rx_queue_disabled (the receive path skips queues that are not initialized).

The RDMA error counters at qwords 96 to 125, and {tx,rx}_rdma_cnp_packets and rx_rdma_ecn_packets, belong to the data path, which serves the same counters to the guest’s hw_counters; they are copied into this block whenever it is written or read, so the two agree. Those with no emulator event behind them yet (packet sequence, duplicates, flushes, CNP and ECN) read zero. A device reset zeroes them in the data path, so hw_counters restarts from zero along with this block.

ernic-exporter reads that section and publishes every counter as ernic_lif_counter_total, labelled with the ionic_if.h name; the Grafana dashboard graphs them. The guest’s separate RDMA hw_counters view, which the device serves from the admin queue rather than from this block, and how the two counter sets compare with the silicon, are in Prometheus Monitoring and Grafana Dashboard.

Port (MAC) statistics#

At PORT_INIT the driver hands the device a struct ionic_port_info. Its struct ionic_port_stats (byte 576, 89 counters) is what ethtool -S lists under the port names such as frames_rx_ok and frames_rx_64b. The emulator counts these at the wire, the way a MAC sees it, and publishes them on the same tick as the LIF block.

A frame’s wire length is its length padded to 60 bytes, as a sending MAC pads it, plus the 4-byte FCS. A 42-byte ARP from the TAP is therefore a 64-byte frame. What moves:

Event

Counters

frame sent

frames_tx_all, _ok, octets_tx_total, _ok, the size bucket, the cast class

frame received

frames_rx_all, _ok, octets_rx_all, _ok, the size bucket, the cast class

VLAN tag (0x8100, or a 0x88a8 outer tag)

frames_rx_vlan_good or frames_tx_vlan, and frames_{rx,tx}_pri_N from the tag’s PCP; untagged frames count as _pri_0

received frame over the port’s maximum (link.mtu_max + 18)

frames_rx_too_long, _oversized, _bad_all; the MAC drops it and the LIF never sees it

received with no buffer posted, or the queue of frames for the guest full

frames_rx_dropped, on top of the counters for a received frame: the MAC took it in, the LIF dropped it

The size buckets run 64, 65–127, 128–255, 256–511, 512–1023, 1024–1518, 1519–2047, 2048–4095, 4096–8191, 8192–9215 and 9216 and up (_other), so ping -s 1472 lands in frames_rx_1024b_1518b. Because every frame is padded, frames_rx_undersized and _less_than_64b stay zero, as do the counters only a real wire can move: bad FCS, jabber, fragments and pause frames.

PORT_RESET stops the port writes but not the LIF ones, and LIF_RESET does the reverse. PORT_RESET also zeroes the port counters, so the next PORT_INIT starts from 0; the LIF counters are kept. The stats file has a Port Statistics section, one line per counter.

Guest Driver: Patched Upstream ionic#

The guest driver is not built from this repository. The build fetches a pinned upstream Linux tree, applies the patches in ansible/files/patches/, and hands the result to DKMS.

Note

The patch series and the two scripts that apply it live in the Ansible collection so that a consumer installing sbates130272.rocm_ernic gets them at the same ref as the roles that apply them; see ansible/files/README.md. The installed layout under share/rocm-ernic is unchanged.

Patches currently carried:

Patch

Purpose

0002-ionic-allocate-an-address-handle-for-UC-queue-pairs.patch

Allocates an address handle for UC queue pairs, which the upstream RDMA driver otherwise omits.

Building and Installing#

Configure with -DERNIC_BUILD_KMOD=ON and use the generated targets. These run inside the guest, where the modules are needed:

cmake -B build -G Ninja -DERNIC_BUILD_KMOD=ON
cmake --build build --target fetch-ionic-sources
cmake --build build --target build-ionic-dkms
sudo cmake --build build --target install-ionic-dkms

# and to back it out again
sudo cmake --build build --target remove-ionic-dkms

fetch-ionic-sources sparse-clones only the two ionic subtrees — drivers/net/ethernet/pensando/ionic and drivers/infiniband/hw/ionic — from IONIC_KERNEL_REF (default v7.2.6) and applies ansible/files/patches/*.patch in filename order with git am. The DKMS package is registered as ionic-ernic.

Warning

IONIC_KERNEL_REF must be at least v6.18: drivers/infiniband/hw/ionic was merged for 6.18, so older refs cannot supply the RDMA half of the stack.

The guest kernel’s major.minor must also match the ref. The ionic sources track IB-core helpers that move between minor releases — v7.2.4 sources on a 7.0 kernel fail on ib_umem_get_va, ib_copy_validate_udata_in and ib_respond_udata — so the default v7.2.6 needs a 7.2.x guest. A point-release gap (v7.2.4 sources on 7.2.6) is fine. No Ubuntu stock kernel qualifies today: noble HWE is 6.17 and resolute GA is 7.0, so the guest image bakes a mainline kernel from the Ubuntu mainline PPA — the ionic flavour pins 7.2.6 — and ernic_guest_setup asserts the match before it starts the DKMS build.

Loading and Verifying#

sudo modprobe ionic
sudo modprobe ionic_rdma

sudo update-pciids
lspci -nnv -d 1dd8:1002 | grep '\[1dd8:5400\]'
ip link                  # the LIF appears as a normal netdev
ibv_devices              # the RDMA device appears here

Userspace: the Upstream Provider#

No patched rdma-core is needed: providers/ionic landed upstream in rdma-core v61, and the guest role installs v62.0. The ernic_guest_setup role therefore skips the provider injection and registration steps entirely and only verifies that libionic*.so landed in libibverbs/.

The one exception is GPU work. With ernic_gpu_passthrough on, the role applies rdma-core/ionic-gda/*.patch from the guest’s rocm-xio checkout before configuring, so the installed provider exposes the direct-verbs symbols the GDA_IONIC backend in rocm-xio links against. rocm-xio is then configured with -DGDA_IONIC=ON -DRDMA_CORE_BUILD=OFF so it uses that system rdma-core rather than building a second copy.

Bumping the Upstream Baseline#

Change IONIC_KERNEL_REF in cmake/ErnicKernelModule.cmake and re-run fetch-ionic-sources. If a patch no longer applies, rebase it and refresh the file in ansible/files/patches/. The ionic-patches CI job reads the pinned ref straight out of the cmake file and fails the pull request if any patch stops applying, so the two cannot drift apart.

The pin is also what the driver pack records for guests: the install rules write it to share/rocm-ernic/ionic-kernel-ref, and rocm-ernic-driver-pack copies that file into the tarball so the guest builds from the same baseline the patches beside it were generated against. Because it is captured at install time, a bump needs cmake --install re-run on the host before the next driver-pack; neither script carries a fallback, so a pack assembled without that file fails immediately rather than guessing a ref.

Testing#

tests/test_ionic_ci.sh is registered with CTest as ionic-ci and needs no VM. It checks that the server starts on each backend, announces 1dd8:1002 with subsystem 1dd8:5400, reports the expected BAR and MSI-X geometry, shuts down cleanly on SIGTERM, attaches to a TAP when one is available, comes up with no extra flags, and writes the full counter set to the stats file.

The TAP attach check is skipped unless ERNIC_TEST_TAP names an existing interface owned by the current user, because creating one needs CAP_NET_ADMIN:

sudo ip tuntap add dev ernic-ci0 mode tap user "$USER"
ERNIC_TEST_TAP=ernic-ci0 ctest --test-dir build -R ionic-ci

Current Status#

Working:

  • PCI enumeration, device identify, LIF init, and the admin queue

  • Ethernet transmit and receive over TAP, including ARP, ICMP, and bulk TCP in both directions

  • RC, UC, and UD queue pairs (ib_*_pingpong and the in-tree rdma_verify payload checks)

  • Fast registration: IB_WR_REG_MR and IB_WR_LOCAL_INV as local work requests, which is what a stock nvme-rdma initiator uses to produce the keys in its keyed SGLs

  • guest-to-guest rdma_cm, and the transfers that follow it: rping between two guests on a mesh connects, exchanges its payloads – SEND, RDMA READ and RDMA WRITE, verified at both ends – and exits 0

  • flushing a QP that enters the Error state, so that work requests still outstanding are completed with IBV_WC_WR_FLUSH_ERR rather than left to a consumer that waits for them forever. One completion per queue, which is what the device owes: it names no work request, and the driver completes what it has posted from its own records. A QP bound to an SRQ flushes its send queue only, because the receives in an SRQ belong to the SRQ and other QPs are still drawing from them

  • restarting a mesh worker. It comes back as a new node – it asks for an id of 0xFFFFFFFF, as any worker does when it starts, and its listening port is ephemeral, so nothing in the registration says which node it used to be – and the GIDs it claimed before are withdrawn so that they follow it to its new id rather than being claimed by two nodes at once and resolving to neither. The manager withdraws what an unreachable node claimed both when its health check notices one has died and when any worker registers, the second being what covers a worker that comes back inside the health-check interval. The withdrawal is reversible: the claims are kept and only stop counting, so a node that was merely silent and then answers a heartbeat, or that registers again under its own id, owns its addresses again without advertising them afresh

  • losing a mesh worker’s connection to the manager. The worker connects again and registers under the id it has when the connection ends, or when nothing has come on it for three health-check intervals, and the manager gives the id back if the worker asks from the address it registered and can’t still be answering on the connection it had. A worker whose id the manager no longer has (it evicted the node, or it has restarted) stops asking, and has to be restarted to rejoin. The manager closes the connection of a node that stops answering, rather than connecting to it. While a worker has no connection to the manager it hears nothing of the mesh, and reaches only the nodes it has a connection of its own to

Not yet working:

  • rping prints poll error -5 once as it tears down, from a bug in the rdma-core ionic provider rather than in the device. The provider keeps receive metadata in slots taken off a free list and records which slot a queue position used in rq.meta_idx; ionic_poll_recv() reads it back through that index, but ionic_flush_recv() uses the raw wqe_id as the slot number. The two agree only until the first completion permutes the free list, after which the flush walk reads a slot that has already been returned, calls it “wqe not posted” and gives ibv_poll_cq() -EIO. Present in rdma-core v61.0 and still in master (providers/ionic/ionic_verbs.c). The exchange itself is unaffected and rping exits 0.

    Establishment was broken until recently, and both reasons this page used to give were wrong: GSI traffic is forwarded between endpoints, which ernic-test-gsi-mesh demonstrates, and both the link-local and the v4-mapped GID are populated. What actually stopped it was the shape of the 40-byte header in front of a datagram – the device claimed RoCEv2 over IPv4 in the completion while writing an IPv6 GRH, so ib_cm read the middle of the destination GID as the sender’s address, ARPed for 0.0.255.255, and dropped every CM REQ with -ETIMEDOUT and no reject. Fixed, pinned in both directions by ionic-harness-ud, ionic-harness-nvmeof and gsi-mesh, and confirmed on a two-guest pair: with the previous build the client printed nothing and the server’s neighbour table held 0.0.255.255 FAILED; with the fix that entry is gone and the connection is accepted.

    The nvmeof backend does not depend on any of this: it answers the CM exchange inside the server rather than routing it to a peer, so nvme connect -t rdma has always had a responder (see NVMe-oF Controller)

  • Shared receive queues, which the upstream RDMA driver does not implement in its ib_device_ops

  • The PCI Express capability is not advertised, so the guest reports an unknown link speed and width; this is cosmetic