systemd Service and ernicctl ============================ rocm-ernic ships a systemd service and a companion ``ernicctl`` CLI for managing one or more server instances, launching VMs, performing live hotplug, and distributing the kernel driver to guests. .. contents:: On this page :local: :depth: 2 Overview -------- The service replaces the older ``setup-rocm-ernic`` script with a production-grade systemd unit. Key capabilities: - Start/stop N server instances in a TCP manager/worker mesh with a single ``systemctl`` command. - Deterministic per-instance MAC addresses derived from the host's ``/etc/machine-id``. - Dynamic mesh scaling -- add or remove worker instances at runtime. - Launch QEMU VMs via qemu-minimal's ``run-vm`` and hot-plug/unplug vfio-user devices via QMP. - Build and push a self-contained driver tarball to guest VMs. Prerequisites ^^^^^^^^^^^^^ - Python 3 (standard library only; no pip packages) - ``socat`` (for QMP communication) - QEMU 10.1+ with vfio-user support - qemu-minimal checkout with ``run-vm`` supporting ``QMP_SOCKET`` and ``VFIO_USERDEV`` .. code-block:: bash sudo apt install socat python3 Installation ------------ The nightly ``rocm-ernic`` and ``rocm-ernic-exporter`` packages install all of this under ``/usr``, with the configuration in ``/etc/rocm-ernic/rocm-ernic.env`` (see :ref:`nightly-packages`). Otherwise, build and install with CMake: .. code-block:: bash cmake -B build -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_INSTALL_PREFIX=/usr/local \ -DERNIC_INSTALL_SERVICE=ON cmake --build build sudo cmake --install build This installs: .. list-table:: :header-rows: 1 :widths: 40 60 * - Destination - Contents * - ``/usr/local/bin/`` - ``rocm-ernic``, ``ernicctl``, ``ernic-exporter`` * - ``/usr/local/libexec/rocm-ernic/`` - ``rocm-ernic-launcher``, ``rocm-ernic-driver-pack`` * - ``/usr/local/etc/rocm-ernic/`` - ``rocm-ernic.env`` * - ``/usr/local/share/rocm-ernic/`` - ``vm-driver-install.sh.in``, ``patches/``, ``fetch-ionic-sources.sh``, ``setup-ionic-dkms.sh``, ``ionic-kernel-ref``, ``requirements-exporter.txt``, ``grafana/ernic-dashboard.json`` * - ``/usr/local/lib/systemd/system/`` - ``rocm-ernic.service``, ``rocm-ernic-driver-pack.service``, ``ernic-exporter.service`` All destinations are relative to ``CMAKE_INSTALL_PREFIX``, shown here for the default ``/usr/local``, except that with the prefix ``/usr``, the environment file goes in ``/etc/rocm-ernic/``. The installed units, scripts, ``ernicctl``, and environment file name each other by their paths under the prefix the project was configured with. .. note:: The systemd units and ``ernicctl`` read the environment file from ``/etc/rocm-ernic/rocm-ernic.env``. Under any prefix but ``/usr``, the install rule places it under the prefix: with the default prefix it lands in ``/usr/local/etc/rocm-ernic/``, so copy or symlink it into ``/etc/rocm-ernic/`` before starting the service. Likewise, systemd does not read units from ``/usr/local/lib/systemd/system`` by default; install with ``-DCMAKE_INSTALL_PREFIX=/usr`` or copy the units into ``/etc/systemd/system/``. After installation, reload systemd and optionally enable the service at boot: .. code-block:: bash sudo systemctl daemon-reload sudo systemctl enable rocm-ernic Configuration ------------- All settings live in ``/etc/rocm-ernic/rocm-ernic.env`` (a shell-sourceable key=value file used by systemd's ``EnvironmentFile=`` directive). Edit this file to match your environment. The defaults below are for an install under ``/usr/local``; under another prefix, such as the packages' ``/usr``, the paths are under that prefix. Server settings ^^^^^^^^^^^^^^^ .. list-table:: :header-rows: 1 :widths: 30 20 50 * - Variable - Default - Description * - ``ERNIC_BIN`` - ``/usr/local/bin/rocm-ernic`` - Path to the server binary * - ``ERNIC_RUN_DIR`` - ``/run/rocm-ernic`` - Runtime directory (sockets, stats, manifest) * - ``ERNIC_LOG_DIR`` - ``/var/log/rocm-ernic`` - Log directory * - ``ERNIC_INSTANCES`` - ``4`` - Total instances (1 manager + N-1 workers) * - ``ERNIC_TCP_PORT`` - ``6320`` - TCP port for the manager/worker mesh * - ``ERNIC_MANAGER_IP`` - ``127.0.0.1`` - Host name, or IPv4 or IPv6 address, workers connect to. An IPv6 address needs no brackets. * - ``ERNIC_GUEST_GIDS_`` - derived - Comma-separated guest RoCE addresses owned by instance *n*, in ``inet_pton`` notation. Peers match a destination GID against these to pick the destination node, so an address missing here is unreachable and its peers fail ``MODIFY_QP``. Defaults to the IPv6 link-local address derived from the instance MAC; set it when the guest also carries a static address peers will target. Passed to the server as ``ERNIC_TCP_GUEST_GIDS`` * - ``ERNIC_LOG_LEVEL`` - ``warn`` - Server log verbosity: ``none|error|warn|info|debug`` * - ``ERNIC_SOCKET_GROUP`` - (unset) - Group whose members may connect to the instance sockets, such as the group of a QEMU not run as root. Passed to the server as ``--socket-group``. Unset, the sockets are ``0600`` and only root can connect. * - ``ERNIC_VERBOSE`` - ``false`` - Deprecated shorthand for ``ERNIC_LOG_LEVEL=debug`` * - ``ERNIC_TAP_PREFIX`` - ``ernic-tap`` - Instance *n* is started with ``--tap ${ERNIC_TAP_PREFIX}n``; a missing interface is a warning and the instance starts without Ethernet * - ``ERNIC_TAP_BRIDGE`` - ``ernicbr0`` - Host bridge the TAPs are enslaved to, so guests can reach each other. Created by the ``ernic_host_setup`` Ansible role, not by the launcher MAC address overrides ^^^^^^^^^^^^^^^^^^^^^ By default, each instance gets a deterministic MAC derived from ``/etc/machine-id`` and the instance index. To assign explicit MACs, set ``ERNIC_MAC_1`` through ``ERNIC_MAC_N``: .. code-block:: bash ERNIC_MAC_1=02:aa:bb:cc:00:01 ERNIC_MAC_2=02:aa:bb:cc:00:02 VM and QEMU settings ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: :header-rows: 1 :widths: 30 20 50 * - Variable - Default - Description * - ``ERNIC_QEMU_MINIMAL`` - ``$HOME/Projects/qemu-minimal`` - Path to qemu-minimal checkout * - ``ERNIC_QEMU_PATH`` - ``/opt/qemu-v10.1.2/bin/`` - Directory containing ``qemu-system-x86_64`` * - ``ERNIC_VM_IMAGE_DIR`` - ``$ERNIC_QEMU_MINIMAL/images`` - Directory with qcow2 VM images * - ``ERNIC_VM_NAME`` - ``stebates-test-vm`` - Base VM name (qcow2 filename stem) * - ``ERNIC_VM_VCPUS`` - ``4`` - vCPUs per VM * - ``ERNIC_VM_MEM`` - ``8192`` - Memory per VM (MB) * - ``ERNIC_VM_SSH_USER`` - ``ubuntu`` - SSH user for guest access * - ``ERNIC_VM_SSH_HOST`` - ``localhost`` - SSH destination host for guest commands (use an ``ssh_config`` Host alias when you do not use ``localhost``). * - ``ERNIC_VM_SSH_CONFIG`` - (unset) - Path passed to ``ssh``/``scp`` as ``-F``. When unset and ``ernicctl`` runs as ``root`` under ``sudo``, the invoking user's ``~/.ssh/config`` (``SUDO_USER``) is used if that file exists. ``/root/.ssh`` is never read for this implicit default; set ``ERNIC_VM_SSH_CONFIG`` when there is no ``SUDO_USER`` (e.g. a systemd unit or bare ``su -``). * - ``ERNIC_VM_SSH_BASE_PORT`` - ``2222`` - Base SSH port (instance N uses base + N - 1) * - ``ERNIC_VM_SSH_IDENTITY`` - (unset) - SSH private key path for the VM_LOAD column (SSH fallback). When unset and ``ernicctl`` runs as ``root`` under ``sudo``, a recent ``ernicctl`` may use ``SUDO_USER``'s default ``~/.ssh/id_rsa`` (or ``id_ed25519`` / ``id_ecdsa``) automatically. ``/root/.ssh`` keys are not auto-discovered. Otherwise set this for ``ERNIC_VM_SSH_USER@ERNIC_VM_SSH_HOST``. * - ``ERNIC_VM_BACKING`` - (unset) - Golden backing qcow2. When set and a per-instance overlay does not exist, ``ernicctl vm-launch`` creates one automatically via ``qemu-img create``. Driver settings ^^^^^^^^^^^^^^^ .. list-table:: :header-rows: 1 :widths: 30 25 45 * - Variable - Default - Description * - ``ERNIC_DRIVER_SOURCE`` - ``/usr/local/share/rocm-ernic/patches`` - Patched ionic sources to pack for the guest * - ``ERNIC_DRIVER_TARBALL`` - ``/tmp/rocm-ernic-driver.tar.gz`` - Path to the built tarball Using the Service ----------------- Start, stop, and check status: .. code-block:: bash sudo systemctl start rocm-ernic sudo systemctl status rocm-ernic sudo systemctl stop rocm-ernic Or use ``ernicctl`` which wraps ``systemctl``: .. code-block:: bash sudo ernicctl start ernicctl status sudo ernicctl stop The ``ernicctl status`` command reports the emulated device, then a table of all instances with their PID, MAC, socket, VM attachment, IP and RDMA byte counts (TX/RX), liveness state, uptime, and log file path: .. code-block:: text Device: ionic 1dd8:1002 (TAP ernic-tap) ID ROLE PID STATE UPTIME MAC SOCKET VM IP (TX/RX) RDMA (TX/RX) LOG 1 manager 12345 running 2h15m 02:a1:b2:c3:d4:01 /run/rocm-ernic/1.sock 67890:2222 1.2K/3.4K 45M/12M /var/log/rocm-ernic/1.log 2 worker 12346 running 2h14m 02:a1:b2:c3:d4:02 /run/rocm-ernic/2.sock 67891:2223 0B/0B 0B/0B /var/log/rocm-ernic/2.log 3 worker 12347 running 2h14m 02:a1:b2:c3:d4:03 /run/rocm-ernic/3.sock - - - /var/log/rocm-ernic/3.log 4 worker 0 dead - 02:a1:b2:c3:d4:04 /run/rocm-ernic/4.sock - - - /var/log/rocm-ernic/4.log The VM column shows ``vm_pid:ssh_port`` when a VM is attached, ``detached`` when detached, or ``-`` when no VM is associated. The IP and RDMA columns show TX/RX byte totals with auto-scaled units (B, K, M, G, T). The UPTIME column shows how long each server process has been running (``45s``, ``3m47s``, ``2h15m``, ``3d04h``). Pass ``--json`` for machine-readable output (includes raw ``ip_bytes_tx``, ``ip_bytes_rx``, ``rdma_bytes_tx``, ``rdma_bytes_rx`` fields). Pass ``--rate SECS`` to show byte deltas measured over a SECS-second window instead of cumulative totals. This is useful with ``watch`` to monitor live throughput. ernicctl Command Reference -------------------------- Service lifecycle ^^^^^^^^^^^^^^^^^ .. code-block:: bash ernicctl start # systemctl start ernicctl stop # systemctl stop ernicctl restart # systemctl restart ernicctl reload # re-read env ernicctl status [--json] # instance table ernicctl status --rate 5 # 5s byte deltas ernicctl status --debug-vm-load # VM_LOAD QMP/SSH trace ernicctl logs [N] # tail logs ernicctl stats [N] # raw stats dump ernicctl stats --summary # one-row summary The ``ernicctl stats --summary`` command prints a concise table with one row per instance: .. code-block:: text ID CONN IP (TX/RX) RDMA-SR (TX/RX) RDMA-RW (R/W) MMIO (R/W) CMDS INTS QPS FLR 1 connected 1.2K/3.4K 45M/12M 8.1M/22M 123K/456K 89 789 3 0 2 connected 0B/0B 0B/0B 0B/0B 0B/0B 0 0 0 0 Columns: CONN = connection state, RDMA-SR = Send/Recv bytes, RDMA-RW = Read/Write bytes, MMIO = register reads/writes, CMDS = commands processed, INTS = interrupts, QPS = active queue pairs, FLR = Function Level Reset count. Dynamic mesh management ^^^^^^^^^^^^^^^^^^^^^^^ Add or remove worker instances at runtime without restarting the manager or existing workers: .. code-block:: bash # Add a new worker to the TCP mesh sudo ernicctl add-instance # Remove instance 4 (detaches from VM if needed) sudo ernicctl remove-instance 4 Live VM attach and detach ^^^^^^^^^^^^^^^^^^^^^^^^^ Hot-plug or hot-unplug a server instance's vfio-user device into/from a running VM via QMP: .. code-block:: bash # Attach instance 2 to a running VM sudo ernicctl attach 2 \ --qmp /run/rocm-ernic/qmp-2.sock # Detach instance 2 (optionally unload guest driver) sudo ernicctl detach 2 --unload-driver The ``attach`` command accepts optional overrides: - ``--device-id`` -- QEMU device ID (default ``ernicN``) - ``--root-port`` -- PCIe root port bus (default ``pcie-vfu.1``) - ``--ssh-port`` -- guest SSH port - ``--ssh-user`` -- guest SSH user Hot reload ^^^^^^^^^^ Perform a full hot-reload cycle for an instance: detach from VM, stop server, rebuild, restart, re-attach, and optionally update the guest driver: .. code-block:: bash sudo ernicctl hot-reload 1 sudo ernicctl hot-reload 1 --update-driver sudo ernicctl hot-reload 1 --no-build This is the service-aware equivalent of the standalone ``scripts/hot-reload.sh`` script. VM management ^^^^^^^^^^^^^ Launch, stop, and list QEMU VMs. ``vm-launch`` uses qemu-minimal's ``run-vm`` script with automatic QMP, hotplug, and manifest integration: .. code-block:: bash # Launch a VM connected to instance 1 sudo ernicctl vm-launch 1 # Launch with overrides sudo ernicctl vm-launch 2 \ --vm-name my-test-vm \ --ssh-port 2223 \ --vcpus 8 --mem 16384 # List running VMs ernicctl vm-list # Stop a VM sudo ernicctl vm-stop 1 The ``vm-launch`` command sets the following environment variables for ``run-vm``: - ``VFIO_USERDEV`` -- the instance's vfio-user socket - ``QMP_SOCKET`` -- placed at ``$ERNIC_RUN_DIR/qmp-N.sock`` - ``QEMU_PATH``, ``VM_NAME``, ``IMAGES``, ``SSH_PORT``, ``VCPUS``, ``VMEM`` Driver distribution ^^^^^^^^^^^^^^^^^^^ Build a self-contained driver tarball and push it to guest VMs: .. code-block:: bash # Build the tarball sudo ernicctl driver-pack # Push to a VM and install automatically ernicctl driver-push ubuntu@localhost \ --port 2222 --install # Push without auto-install ernicctl driver-push ubuntu@localhost --port 2222 The tarball extracts to ``/tmp/rocm-ernic-driver/`` on the guest and includes ``vm-driver-install.sh`` which: 1. Fetches the upstream ionic sources at the pinned ``ionic-kernel-ref`` and applies ``patches/``. 2. Builds and installs ``ionic`` and ``ionic_rdma`` via DKMS. 3. Loads ``ib_core``, ``ib_uverbs``, ``ionic`` and ``ionic_rdma``. 4. Verifies the RDMA device appears via ``ibv_devices``. The install script also supports ``--unload`` to remove the modules and ``--build-only`` to build without loading. See :doc:`ionic` for the sources and patches it uses. Manifest -------- The service maintains a JSON manifest at ``$ERNIC_RUN_DIR/instances.json`` that tracks all running instances and their VM attachment state. The manifest is the single source of truth for ``ernicctl`` and the launcher. Example: .. code-block:: json { "instances": [ { "id": 1, "role": "manager", "pid": 12345, "mac": "02:a1:b2:c3:d4:01", "socket": "/run/rocm-ernic/1.sock", "vm": { "qmp_socket": "/run/rocm-ernic/qmp-1.sock", "device_id": "ernic1", "root_port": "pcie-vfu.1", "ssh_port": 2222, "ssh_user": "ubuntu", "attached": true, "vm_pid": 67890, "vm_name": "stebates-test-vm" } } ] } When ``vm`` is ``null``, the instance is running but not attached to any VM. Typical Workflows ----------------- Single-server development ^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash # Edit /etc/rocm-ernic/rocm-ernic.env: # ERNIC_INSTANCES=1 sudo systemctl start rocm-ernic sudo ernicctl vm-launch 1 ernicctl driver-push ubuntu@localhost \ --port 2222 --install ssh -p 2222 ubuntu@localhost Multi-server TCP mesh ^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash # Edit /etc/rocm-ernic/rocm-ernic.env: # ERNIC_INSTANCES=3 sudo systemctl start rocm-ernic ernicctl status # Launch VMs for instances 1 and 2 sudo ernicctl vm-launch 1 sudo ernicctl vm-launch 2 # Push driver to both VMs ernicctl driver-push ubuntu@localhost \ --port 2222 --install ernicctl driver-push ubuntu@localhost \ --port 2223 --install Adding a server to a running mesh ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash sudo ernicctl add-instance # Prints: Added instance 4. sudo ernicctl vm-launch 4 ernicctl status Automated workflow via Ansible ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``ansible/`` directory contains playbooks that automate the entire workflow: build, install, configure the env file, create golden VM images, launch VMs, provision guests with the driver and custom rdma-core, and run iperf3 / perftest sanity tests. See :doc:`testing` for details. .. code-block:: bash cd ansible ansible-playbook site.yml Hot-reloading after a code change ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: bash # Rebuild server and cycle instance 1 sudo ernicctl hot-reload 1 --update-driver Guest VM Hardening ^^^^^^^^^^^^^^^^^^ The ``vm-driver-install.sh`` script automatically configures systemd to avoid long shutdown hangs caused by stale user session scopes: - ``KillUserProcesses=yes`` in ``/etc/systemd/logind.conf`` - ``DefaultTimeoutStopSec=15s`` in ``/etc/systemd/system.conf`` These settings are applied on first driver install and persist across reboots.