NUMA placement for a DRAM tier

UMBP can split a DRAM tier into one buffer per NUMA node and prefer the writer GPU’s local buffer. All buffers remain in one backend and one shared keyspace. Readers on any GPU can still access any object.

Configuration

export UMBP_DRAM_NUMA_NODE=0,1
# Optional: fail if a mapping cannot be bound, and forbid physical-page spill.
export UMBP_DRAM_NUMA_STRICT=1
# Optional: 1 for serial prefault; 0 (default) selects up to 16 workers.
export UMBP_DRAM_PREFAULT_THREADS=0

With no NUMA node configured (unset, blank, or -1), the tier keeps the original single-buffer layout and allocation order. A single node retains MPOL_BIND. Multiple nodes use MPOL_PREFERRED unless strict mode is enabled. Duplicate nodes, negative nodes mixed into a list, and capacities too small for one allocator page per node are rejected.

Binding policy and syscall failure handling are separate:

Nodes

STRICT

Policy

If mbind fails

Unset / -1

Either

No binding

mbind is not called

One node

0 (default)

BIND

Warn and continue

Multiple nodes

0 (default)

PREFERRED

Log degradation and continue

One or multiple nodes

1

BIND

Free the mapping and fail allocation

Without STRICT, ENOSYS emits one unavailable warning and continues. With STRICT, every binding failure, including ENOSYS, is fatal: startup must not claim strict placement on a system that cannot bind memory. A successful BIND still restricts physical pages to the selected node even when STRICT is zero; STRICT controls error handling, not whether that policy can spill on a fault.

In a container, mbind needs CAP_SYS_NICE: under Docker’s default seccomp profile it fails with EPERM. Run with --cap-add SYS_NICE, --security-opt seccomp=unconfined or --privileged. Without one of them a DRAM tier handles the failure as the table above says, and the standalone server’s ranged scratch, which requires its binding, falls back to one arena per direction.

With UMBP_DRAM_USE_HUGEPAGES=1, reserve huge pages per node, not only in total. The hugetlb reservation made at mmap time is global, but a fault under BIND takes pages only from the bound node, so a node whose own pool is short fails the fault even though the global count looked sufficient. Reserve at least each buffer’s size on its node, for example echo N > /sys/devices/system/node/node<i>/hugepages/hugepages-2048kB/nr_hugepages.

The capacity is split evenly in whole allocator pages, with the remainder in the final buffer. The policy JSON’s DRAM numa_node also accepts a list:

{"type": "dram", "capacity": "40GiB", "numa_node": [0, 1]}

Policy-created DRAM backends inherit the configured hugepage, prefault, strict binding, prefault thread and memory-region size options. Python exposes dram.numa_nodes, dram.numa_strict, dram.prefault_threads, and dram.max_mr_bytes. The deprecated dram.numa_node property reads the first node (or -1); assigning it replaces the entire list. The standalone client’s child server receives the same configuration.

UMBPHostMemAllocator.alloc(..., numa_node=...), including SGLang’s HiCache L2 allocator, keeps its scalar API and existing best-effort MPOL_BIND behavior.

Allocation and startup

The local write path derives one advisory NUMA hint per ranged batch and one per object for whole-object Put/BatchPut. CPU buffers and unknown GPUs have no preference. If the preferred buffer can satisfy a request, contiguous pages are preferred over scattered pages within that buffer. Otherwise, a global capacity check precedes allocation of the local remainder and then pages from other buffers. Insufficient global space leaves all bitmaps unchanged. HBM and SSD backends ignore the hint.

A host buffer larger than UMBP_DRAM_MAX_MR_BYTES (default 512 GiB) is registered with the NIC as several memory regions of at most that size, in whole backing and allocator pages. AMD Pensando AINIC firmware before 1.117.5-a-190 translates only the first 2^18 page entries of a memory region and silently aliases the rest, so the limit must not exceed 2^18 backing pages: 512 GiB with 2 MiB pages, 1 GiB with 4 KiB pages. On such a host the DRAM tier and the ranged scratch warn at startup about a region with more, counted from the pages that actually back it, since a hugepage request that fell back to 4 KiB pages has 512 times the entries; the warning names the firmware. 0 registers each buffer whole. Each region is its own buffer on the same node; a node preference takes the first of that node’s regions that holds the request.

Logical page spill and physical memory spill are distinct: the former uses another tier buffer when the local buffer is full; MPOL_PREFERRED also lets the kernel back a buffer with another node’s physical pages when necessary. Startup logs distinguish target placement from sampled physical placement. Failure to query placement is logged rather than inferred as success.

Every tier of at least 64 MiB prefaults in parallel with a total budget of at most 16 workers (UMBP_DRAM_PREFAULT_THREADS=1 restores the serial path). Large buffers can subdivide that budget into page-aligned chunks; small chunks stay serial. A bound buffer’s workers run on its node’s allowed CPUs. An unbound buffer’s workers alternate between nodes in whole rounds, so first touch splits it evenly across sockets instead of filling one node first. Workers restore their previous affinity. UMBP_DRAM_PREFAULT=0 disables prefault. GPU registration and the transfer path are unchanged.

This policy favors locality over the opportunity to put an entire object in a remote contiguous run. Its net benefit depends on fragmentation and workload. It does not migrate existing objects or replicate shared keys across sockets.

Remote writes

A write routed to another node carries the same advisory hint in its slot request: the node of the writer’s GPU, also when a ranged put stages the bytes through the ranged PUT arena, otherwise the node of a registered host source. The peer prefers its regions on that node. Without a hint (a CPU caller, or a writer that sends none), a tier with buffers on several nodes starts on the node with the most free pages instead of the first one, so the nodes fill evenly; a tier on one node keeps its order.

Each side of an RDMA transfer picks its NICs from the node of its own memory, so the object then travels between NICs on the same node at both ends when the reader is the GPU that wrote it.

Ranged I/O

A remote ranged GET or PUT stages through a host arena, and every GPU on a node shares the standalone server’s. In one piece, the arena sits on whichever node first touch placed it, so its NUMA node picks the NICs of every rank’s RDMA leg and half the GPUs copy across sockets. The standalone server therefore lays each arena out by NUMA node (UMBP_DISTRIBUTED_RANGED_SCRATCH_NUMA): the configured bytes are split evenly over the nodes that have GPUs, each part is bound to its node and registered on its own, and is cut into one shard per GPU on it unless UMBP_DISTRIBUTED_RANGED_SCRATCH_SHARDS sets the arena’s total. A call stages through a shard on its caller buffer’s node, or that node’s whole part when it needs more than a shard. Memory is the same as one arena per direction. Each part is one NIC memory region, so on 4 KiB pages keep a part within 1 GiB, the AINIC limit above, or back the arenas with hugepages (UMBP_DISTRIBUTED_RANGED_SCRATCH_USE_HUGEPAGES).

Measured effect

One MI355X node (two sockets, eight GPUs), DeepSeek-V4-Pro at TP8, SGLang with the UMBP linker and an agentx replay. Split tier (UMBP_DRAM_NUMA_NODE=0,1) against the single-buffer tier:

Concurrency

DRAM tier

Window

Runs (single / split)

Total token throughput

TTFT p50

16

512 GiB, 4K pages

900 s

3 / 5

within 1%

no consistent difference

128

740 GB, hugetlb, STRICT

3600 s

2 / 2

+0.2%

-0.4%

Placement does not change serving throughput at these points. At concurrency 16 tier reads overlap compute. At 128 the load wait that reaches the critical path is about 1.3% of forward time with or without the split. A hugetlb tier reserved to capacity already fills node 0 and then node 1.

The split does shorten startup. A standalone server with a 740 GB tier is ready in 64-81 s with 4K pages and in 10-13 s with hugetlb. A single-buffer tier used to take 274 s and 36 s: one thread prefaulted it, which placed the whole tier on that thread’s node, and registering a tier packed onto one node for GPU access takes several times longer. Its workers now alternate between nodes, and it is ready in 79-90 s and 12-14 s.

Replicating shared keys on both nodes was measured at concurrency 128 and not adopted. Throughput fell by 10.9%: storing each shared key twice raised recomputed prefill by 25%, and exposed load wait rose by 30%.