Pure-SSD mode and multi-drive SSD tiers
Run a UMBP node with no host-DRAM tier, SSD carrying the whole cache, with one node’s SSD tier spanning several drives at once.
The point of the mode is aggregate NVMe bandwidth: N nodes × M drives all writing and reading in parallel. Most of this page is about not accidentally giving that up.
Quick start
Two pieces: the extra_config your engine passes to UMBPStore, and a handful
of env vars. This is a validated 2-drive-per-node deployment under SGLang
HiCache.
extra_config, per engine:
{
"medium": "SSD",
"ssd_enabled": true,
"ssd_storage_dir": "/umbp_ssd/drive1,/umbp_ssd/drive2",
"ssd_capacity_bytes": 68719476736,
"ssd_backend": "file",
"ssd_io_backend": "io_uring",
"master_address": "<UMBP_MASTER>",
"node_address": "<NODE_IP>",
"io_engine_port": "19600",
"peer_service_port": "19700",
"cache_remote_fetches": false,
"kv_events_subscriber": true,
"kv_events_endpoint": "tcp://localhost:6557",
"staging_buffer_size": 2147483648,
"ssd_staging_use_hugepages": true,
"ssd_staging_hugepage_size": 2097152
}
Env vars, on each engine:
export UMBP_SSD_VERIFY_CRC=0 # skip checksums, matching the DRAM tier
export UMBP_SSD_TIER_IO_THREADS=4
export UMBP_SSD_DURABILITY=strict # ~free once direct I/O is on
export UMBP_SSD_READ_LEASE_MS=30000
export UMBP_DRAM_USE_HUGEPAGES=1
export UMBP_DISTRIBUTED_SSD_STAGING_USE_HUGEPAGES=1
# Only if the engine issues ranged (sub-object) I/O -- the SGLang tree
# connector does. Zero is the default and turns the feature off, and on a
# pure-SSD deployment every node is SSD, so leaving it unset costs ranged I/O
# cluster-wide. Size it above the largest object a remote ranged batch moves.
export UMBP_DISTRIBUTED_RANGED_SCRATCH_BYTES=268435456
Ranged I/O on an SSD medium used to be refused outright regardless of this variable; that exclusion is gone. Note what it does and does not buy: a ranged put runs at full speed, while a ranged get still reads the whole object off the drive and saves only the final copy into the caller’s buffers, because a resolve stages the object before the requested ranges are known. See design-ssd-ranged-io.md.
Deliberately not in that list: UMBP_SSD_TIMING=1. It is a diagnostic, not a
production setting — it prints several [SsdPerf/*] lines per batch, which on a
busy node is a lot of log. Turn it on when investigating throughput, off again
after; see runtime-env-vars.md for what it reports.
Env var, on umbp_master — not the engine:
export UMBP_ROUTE_PUT_SELECT_ALGO=random # see "Spreading writes"
Prerequisite, on every node before launching anything: reserve hugepages —
sudo sysctl -w vm.nr_hugepages=350000 — or engine/worker startup fails with
RegisterRdmaMemoryRegionAuto failed ... errno:12 (Cannot allocate memory).
Enabling the mode
medium: SSD is the switch. A node serves exactly one medium, and naming
SSD is the whole opt-in: the node registers an SsdBackend, advertises SSD
capacity, and takes puts directly. No separate “pure-SSD” flag and no
UMBP_ROUTE_PUT_SSD_MODE — the routing plane does not rank tiers, so every
advertised medium is an equally valid put target and an SSD node needs no
special case (see route_put_strategy.h).
This also means the mode is per node, not per cluster: SSD, DRAM and HBM nodes coexist in one pool, and which medium a key lands on is a property of the node master chose, not of the key.
Selecting SSD with ssd.capacity_bytes or ssd.segment_size_bytes unset is
rejected at startup rather than silently serving a zero-capacity tier.
How the selector reaches the node. UMBPDistributedConfig::medium is
exposed three ways: the pybind field (config.distributed.medium), and
UMBP_DISTRIBUTED_MEDIUM=SSD for the standalone server binary. The "medium"
key shown in the extra_config above needs a matching entry in sglang’s own
umbp_store.py parser — the same gap single_flight_reads has. Until that
lands, an SGLang engine selects SSD through the pybind field or the env var, not
through extra_config JSON.
Spreading writes
Set UMBP_ROUTE_PUT_SELECT_ALGO=random. This is the single most important
knob in the mode, and the default is wrong for it.
The default, most_available, sends each key to the emptiest node. Node capacity
only refreshes on a heartbeat, so every batch in between sees the same “emptiest”
node and piles onto it — and concurrent writers herd onto it together. Aggregate
write bandwidth collapses to one node’s drives. random is capacity-weighted
(a node that cannot fit a block is never chosen), so it still fills evenly
without the herding.
Leave UMBP_ROUTE_PUT_NODE_AFFINITY at none. same and local trade fan-out
for locality, which is the opposite of what this mode is for.
Both are read by the master process. Setting them on an engine does nothing. The master logs its resolved strategy at startup — check that line if a change seems to have no effect.
Sizing
Setting |
Meaning |
|---|---|
|
Comma-separated, one directory per physical drive. More than one turns on multi-drive: keys are balanced across drives and batch I/O runs on all of them at once. |
|
The total across every drive, split evenly — not per drive. |
|
Size of the staging arena, in pages ( |
(writes) |
Remote puts stage through the same arena; local ones do not. There is no separate write-staging region to size. |
Only reads and remote writes stage. A read batch that does not fit waits for a
later round rather than failing: busy resolves are retried for up to
UMBP_RESOLVE_BUSY_TIMEOUT_MS (30 s by default), and only a batch whose own
working set exceeds the whole arena fails at once. A remote put is not retried:
when the arena is full the put fails at once (see “Puts fail on an SSD node”
below).
Each key lives whole on one drive, so a single-key read costs one drive’s I/O.
Capacity is charged in padded on-disk bytes: values are padded to 4 KiB, so
a 512 B value occupies 8 KiB. Negligible for multi-MiB KV pages, but size
ssd_capacity_bytes with your real value size in mind if yours are small.
Direct I/O
Direct I/O is the default. Keep it on for any deployment whose numbers you intend to believe.** Without it the tier opens buffered, and reads are served from the page cache — a whole working set can be served while moving zero bytes to the device, which makes drive-count and DRAM-vs-SSD comparisons meaningless.
Pair it with ssd_io_backend: io_uring; the posix driver on the direct path is
far slower, and the tier warns about it.
Env var |
Default |
Meaning |
|---|---|---|
|
|
|
|
|
|
|
|
Parallelism within one drive’s batch. Worth ~42% on reads versus 1. |
|
|
Parallelism across drives. |
|
|
With direct I/O on, strict costs ~1.4% instead of 42%. Little reason to relax it. |
Confirm it is on: run iostat -x 1 during a read-heavy phase. Buffered shows
~1% device utilisation while reporting absurd bandwidth; direct shows the drives
busy at something near their rating.
Storage-only nodes (no engine, no GPU)
To add SSD capacity to the pool from a node that isn’t running an SGLang
engine — a dedicated storage box, or extra drives on a node already busy with
compute — join it as a bare UMBPStore client instead:
python tests/python/umbp/umbp_store_node.py \
--hicache-storage-backend-extra-config \
'{"master_address": "<UMBP_MASTER>", "node_id": "ssd-node-1",
"node_address": "<NODE_IP>", "medium": "SSD",
"ssd_enabled": true, "ssd_storage_dir": "/mnt/nvme0,/mnt/nvme1",
"ssd_capacity_bytes": 1000000000000}'
It registers with umbp_master, heartbeats, and idles — same config surface
(JSON extra_config or the matching UMBP_* env vars) as an engine’s
UMBPStore, just with mem_pool_host=None. Run it in the same
image/container as the engine (needs sglang + mori importable). Ctrl-C /
SIGTERM flushes and exits cleanly; there is no remote-clear signal wired up,
so clearing one means restarting the process.
If you want compute nodes to contribute no cache capacity of their own —
all SSD capacity coming from dedicated storage nodes instead — set the
engine’s own ssd_capacity_bytes to something unusably tiny (e.g. 1048576,
1 MB) rather than disabling its SSD tier: ssd_enabled: true is still needed
for HiCache’s L3 path to initialize, but random routing skips any node that
“cannot fit a block,” so a 1 MB node is never chosen as a put target.
Troubleshooting
Symptom |
Likely cause |
|---|---|
Absurd read bandwidth, |
You are reading the page cache: either |
Adding drives or nodes doesn’t add bandwidth |
Writes are skewed. Set |
|
It was set on the engine instead of |
Reads report misses under load |
Staging stayed full for longer than |
Puts fail on an SSD node |
The staging arena is exhausted, so |
Node comes up with an empty cache after an upgrade |
On-disk record format changed; older records are dropped on version mismatch |
Tier fills far earlier than expected |
Capacity is charged in padded bytes — small values pay up to 16× |
SSD used is well above the live data |
Used counts dead records until their segment file empties. Compare |
Startup rejects |
|
A config key seems to do nothing |
|
Reads keep missing after killing/relaunching a node several times |
The master drops a dead client only after it has missed heartbeats for |
Resend right after a write misses instead of hitting |
Write-through acks drain asynchronously; a flush/resend that lands before the SSD write actually completes evicts the key with no L3 trace. Give it a moment (scales with value size) before flushing/resending |
Measured: L1 vs L2 vs L3
sglang HiCache, Kimi-K2.6-MXFP4, TP8, one fixed-length request per round. “Request
length” is the prefix; every request appends and generates exactly 1 token on
top, so the number is TTFT. Each tier isolated by forcing a resend to hit only
that tier, verified against /metrics per-tier hit counters rather than timing.
8 physical drives / 4 shards, 2 local + 6 remote across 3 standalone workers on a
second node. The engine’s own SSD capacity was shrunk to 1 MB so it can never be
capacity-selected as a put target and leak reads back to its own idle disk.
Request length |
Cold recompute |
L1 hit |
L2 hit |
L3 hit |
|---|---|---|---|---|
4K |
≈0.18s |
≈0.10s |
≈0.20s |
≈0.14s |
16K |
≈0.73s |
≈0.12s |
≈0.20s |
≈0.20s |
32K |
≈1.42s |
≈0.14s |
≈0.21s |
≈0.27s |
128K |
≈7.7s |
≈0.29s |
≈0.45s |
≈0.84s |
L1 always wins. L3 closes almost all the way to L2 at small sizes — beating it at 4K — but the gap widens with length, since an L3 hit still pays an SSD round trip. L3 is ~1.3x faster than cold recompute at 4K and ~9x by 128K.
Notes
Scope traps.
UMBP_ROUTE_PUT_SELECT_ALGOis read by the master.UMBP_DISTRIBUTED_SSD_STAGING_USE_HUGEPAGESis read only by the standalone server binary — on the SGLang path thessd_staging_use_hugepagesJSON key is the one that takes effect; setting both covers either launch mode.UMBP_DRAM_USE_HUGEPAGESdoes nothing on a node whose medium is SSD — that node registers no DRAM pool.cache_remote_fetches: falsekeeps remote-fetch cost honest by not re-caching whole-object remote reads. It does not cover remote ranged reads: those land the object in the local medium whenever it is already in hand (the fetch brought all of it), which costs nothing on the wire. The extra background pull, which does cost a second copy on the wire, isranged_locality_prefetch: falselocally. Turn it on for a production hit-rate setup.SPDK.
UMBP_SPDK_REACTOR_MASKdefaults to0x1— a single core polling all drives. Give it at least one core per drive (0x3,0xf, …) before expecting multi-drive scaling on that path. Multi-drive tiers and SPDK RAID0 compose; a drive in the list may itself be a RAID0 bdev.Every setting above is also a plain field on
UMBPConfig/UMBPDistributedConfigfor callers driving the client directly, without SGLang.
Full env-var reference: runtime-env-vars.md.