rocm-ernic: Emulated RDMA NIC for Virtual Machines

Introduction

rocm-ernic is a userspace RDMA device server built on the libvfio-user framework. It provides full RDMA (Remote Direct Memory Access) functionality to virtual machines without requiring physical RDMA hardware or an in-guest software stack such as Soft-RoCE.

Key Features

• Full PCIe device emulation in userspace

• An AMD Pensando ionic NIC, driven in the guest by the upstream Linux ionic and ionic_rdma drivers (see The ionic Device)

• Memory-mapped BARs (MSI-X, registers, doorbells)

• MSI-X interrupt support

• Multiple RDMA backends (loopback, TCP/IP)

• An in-process NVMe-oF target, so one VM and one server are a complete fabric (see NVMe-oF Controller)

• An in-process S3-over-RDMA object store, with its own in-band HTTP endpoint on the emulated wire, so the same single VM is a complete object fabric (see S3 over RDMA Object Store)

• Working Ethernet and TCP/IP to the host via a TAP interface

• An optional P4 match-action stage on the emulated wire, matching the programmable datapath of the silicon the device models (see P4 Match-Action Engine)

• Comprehensive statistics collection

Quick Start

sudo apt install cmake meson ninja-build pkg-config \
  libibverbs-dev librdmacm-dev libglib2.0-dev libjson-c-dev
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug
cmake --build build

./build/rocm-ernic \
  --socket /tmp/vfio-user-rocm-ernic.sock \
  --backend loopback --verbose

That presents the device to the guest as an upstream-driven ionic NIC. Add --tap to attach its Ethernet interface to a host TAP:

./build/rocm-ernic \
  --socket /tmp/vfio-user-rocm-ernic.sock \
  --backend loopback --tap ernic0

User Guide

• Building and Installing
  • Requirements
  • Compilation
  • Installation
  • Nightly Packages
  • Packages
  • Build Options
  • Warnings as Errors
  • Sanitizers
  • Code Coverage
  • include-what-you-use
  • clang-tidy
  • Guest ionic Modules
  • Building Documentation
• Architecture
  • The Short Version
  • RDMA in Brief
  • How a PCI Device Is Emulated in Userspace
  • The Device the Guest Sees
  • Inside the Server
  • Bringing the Device Up
  • The Control Path: the RDMA Admin Queue
  • The Data Path
  • More Than One VM: the TCP Mesh
  • The Ethernet Side
  • In-Process Targets: NVMe-oF and S3
  • The P4 Engine
  • Backends
  • Threads
  • Trust Boundaries
  • What Is Not Emulated
  • Where to Look in the Source
• Usage
  • Basic Usage
  • The Emulated Device
  • Log Levels
  • Launching a VM
  • Statistics Collection
  • Development Workflow: Hot Reload
• The ionic Device
  • Starting the Server
  • NIC Profiles
  • Ethernet and TCP/IP
  • Guest Driver: Patched Upstream ionic
  • Testing
  • Current Status
• NVMe-oF Controller
  • Starting the Server
  • Connecting from the Guest
  • How It Works
  • Testing
  • Performance
  • Limitations
• S3 over RDMA Object Store
  • Starting the Server
  • Reaching It from the Guest
  • How It Works
  • Testing
  • Performance
  • Limitations
• P4 Match-Action Engine
  • Status
  • The guest’s control surface
  • The port model
  • Programs
  • Loading a program
  • Table entries
  • Alignment with AMD Pensando silicon
  • Where this differs from the silicon
  • Security
  • Testing
• systemd Service and ernicctl
  • Overview
  • Installation
  • Configuration
  • Using the Service
  • ernicctl Command Reference
  • Manifest
  • Typical Workflows
• Prometheus Monitoring and Grafana Dashboard
  • Architecture
  • Quick Start
  • Configuration
  • Metrics Reference
  • Grafana Dashboard
  • Useful PromQL Examples
• Testing
  • Test Programs
  • Running Tests
  • Multi-VM RDMA Testing
  • Ansible-Based Testing
  • Guest Driver CI
  • Adding New Tests
• Performance
  • Test Environment
  • Running the Benchmarks
  • Interpreting the Results
  • Reference Measurements
  • Known Limitations
• NVMe-oF Performance
  • Test Environment
  • Reproducing
  • Results
  • Known Limits
• Performance trends
  • Send bandwidth over time
  • Send latency over time
  • NVMe-oF read bandwidth over time
  • S3-over-RDMA GET bandwidth over time
  • P4 fp16-to-fp8 conversion bandwidth over time

API Reference

• API Reference
  • Userspace Server API
  • ionic Emulation

Acknowledgments

The RDMA device logic in src/pvrdma/ is adapted from the QEMU PVRDMA implementation. The original authors of that work:

• Yuval Shaia <yuval.shaia@oracle.com> (Oracle)

• Marcel Apfelbaum <marcel@redhat.com> (Red Hat)

License

The build system, documentation, and the deployment and automation code are licensed under the MIT license. The emulator itself is GPL-2.0-or-later:

• Everything under src/ and tests/, and the fuzz harnesses in nix/analysis/fuzz/, which build against src/.

• The VMware/Linux uAPI header src/pvrdma/vmw_pvrdma-abi.h is instead dual GPL-2.0 / BSD-2-Clause.

The patches under ansible/files/patches/ modify the Linux ionic drivers and rdma-core’s ionic provider, and are licensed under the terms of the files they modify: mostly GPL-2.0 for the kernel drivers and GPL-2.0 OR Linux-OpenIB for rdma-core.

The groupings above are a summary; the per-file SPDX-License-Identifier notice is authoritative. See LICENSE.md for the full statement.