UMBP transfer calibration
The optional probes train profiles with real engines, then compare independent real/sim workloads. Use the same build for the probes and the UMBP simulator.
cmake --build build --target umbp_sim_probe umbp_sim_remote_probe -j 8
python examples/umbp/calibrate.py \
--probe build/src/umbp/umbp_sim_probe --output /shared/calibration \
--paths all --device 0 --threads 1 4 --cpus 4,5,6,7 --ssd-dir /data/calibration \
--ssd-io io_uring --ssd-read-prefill-mib 256 --ssd-api-settle-ms 250 \
--local-address 10.0.0.1 --remote-address 10.0.0.2 \
--remote-command 'ssh node2 /path/to/build/src/umbp/umbp_sim_remote_probe'
The default is CPU-only. For local GPU calibration, select
--paths cpu_cpu cpu_gpu gpu_cpu gpu_gpu and omit remote settings.
Both remote nodes need working RDMA, the same build and access to the output/profile
path. Use reachable IPv4 addresses. The driver appends quoted owner arguments to
--remote-command and records owner.log; an existing SSH/container launcher must
provide the environment. The driver does not deploy images or allocate nodes.
GPU/RDMA callers use the full --cpus set to allow CQ polling alongside submission.
The owner launcher receives that affinity; forward it or select suitable remote CPUs.
Native probes accept --block-bytes B,... and --batches N,.... The remote probe
also accepts --gap-bytes (default 64) and --staged; local staged SSD API probes
accept --gpu-caller. The driver selects these modes automatically during validation.
Path |
Real engine and validation |
|---|---|
|
|
|
|
|
|
|
|
|
Actual HbmCopyEngine gather kernels, fitted separately from memcpy. |
|
Serial copyin/copyout on actual MoriIo bounce buffers; RDMA refresh precedes copyout timing. |
|
|
|
|
|
|
|
SSDTier reads without CRC; automatically trained and engine-validated alongside CRC reads. |
Real probes verify complete payloads; sim uses inaccessible virtual buffers.
Allocation, registration, planning, handshake, eight warmups and payload checks stay
outside Submit+Wait timers. API timers include actual control work and trace output
to /dev/null; API/control observations never enter the transfer fit. Remote API
control timing is available separately through the simulator trace. gpu_gpu follows
HbmCopyEngine’s host return without extra device synchronization, so its fitted
bandwidth does not measure device-drain throughput.
Sampling
Option |
Behavior |
|---|---|
|
Defaults: 24 and 3. |
|
Default: 300 seconds per invocation; increase for large grids or settle delays. |
|
Replace default grids; positive entries only, sampled TRAIN/heldout overlap rejected. |
|
Default: 256 MiB aggregate workload across callers, including SSD record padding or remote gaps. Both probes honor this cap. |
|
CPU copy-thread curves; GPU/network/staging fits remain scalar. |
|
Default: 1/2/4; heldout validation only. Callers share an engine and use disjoint buffers. |
|
Local host-source cache condition; default |
Keep CPU/NUMA/GPU placement and cache conditions consistent when reusing profiles. Cold flushing requires x86. Small reused buffers can overestimate DRAM bandwidth; API staging may still warm internal sources, so engine/API cache conditions can differ.
SSD configuration and preparation
Use an explicit writable --ssd-dir on the SSD being calibrated, not NFS/tmpfs.
Probes remove only their own temporary child directories. Match deployment settings:
--ssd-io posix|io_uring: probe defaultposix, UMBP defaultio_uring. Unavailable io_uring, including seccomp restrictions, fails explicitly.--ssd-durability relaxed|strict: defaultstrict, matching UMBP.--ssd-no-crc: staged-read rows usefile_cpu_staged_no_crc; an unmeasured CRC curve remains an estimate. Without this flag, both read curves are sampled.SSD payload sampling uses the default four
tier_io_threads;--threadscontrols CPU copies. API GPU staging uses HBM curves, which should also be calibrated.
--ssd-read-prefill-mib defaults to zero. A positive value writes unrelated records
outside timing after seeding each read fixture, with matching preparation for TRAIN
and heldout engines. API reads then retain seeded keys and repeat only loads. Fixtures
reserve storage/staging for preparation. Choose prefill size for the device; it can
displace recent writes but does not guarantee cold caches or identical fresh extents.
--ssd-settle-ms defaults to 10 ms before engine reads. --ssd-api-settle-ms defaults
to zero for immediate reload. Retained loads may reuse DRAM staging leases; with the
usual 200 ms lease reaper, use 250 ms to validate repeated storage reads, and check
that loads reach storage. Adjust for deployment’s reaper interval. Both waits are
outside timers and do not flush device caches. Report reload, staging-hit and retained
storage-read conditions separately; they are recorded with prefill in report.json.
Staged TRAIN timing includes complete payload work and write-buffer cleanup, excluding
index, locks and planning. wall_elapsed_ns keeps whole-call time, model_bytes keeps
padded I/O bytes and segments keeps actual geometry. Engine/API validation uses whole
wall time. GDS first-FD registration fits separately from TRAIN api=register rows and
is validated with the same lifecycle; registration_latency_ns applies once per live FD.
Profiles and accuracy
Only real, single-caller TRAIN data fits nonnegative fixed overhead, per-copy/segment
cost and effective bandwidth. CPU thread curves interpolate in time/byte. Include small
and deployment-sized large blocks, with independent validation shapes. Unresolved byte
cost keeps the measured copy cost and uses the largest finite bandwidth, reported in
zero_byte_cost_paths; this does not identify physical bandwidth.
GPU TRAIN separates gather/memcpy using actual kernel launches. Sim applies production registration/geometry/cutoff rules and falls back to base curves when variants are absent. Measured H2D/D2H base curves set
interleave_copies=truefor blocking per-copy fallback; old profiles retain batch scheduling.Directional staging can fit
max(0, bytes - size_knee_bytes) * large_size_ns_per_byteonly with identifiable TRAIN sizes on both sides. Staging sampling follows timed wire transfers, and read ranges are cleared before RDMA refresh. Older reads fall back tocpu_staging, then serialcpu_cpu. Model calculation stays inside stage budgets.SSD writes fit combined preparation/I/O/durability/cleanup. Reads optionally fit nonnegative per-I/O
log_size_latency_nsand per-batchbatch_log_size_latency_ns, multiplying log physical bytes per I/O. Unidentifiable or zero terms are omitted. Native SSDTier reads emitserialize_latency=truefor their I/O mutex. Compatible no-CRC curves reserve queued I/O on the shared channel; remaining CRC work completes outside that reservation. Missing/incompatible curves and mixed submits retain full payload service cost; writes serialize service. Older staged paths use direct curves.
--base-profile defaults to complete example estimates. Measured paths replace entries;
unmeasured paths remain estimates, listed in path_sources. Custom bases must include
unmeasured/reverse API paths. profile.json identifies measured paths and estimate count;
report.json retains raw data, settings, profile/base hashes, medians, signed errors and
passes at max(2 µs, 10% of real median). Operation, fit or isolation failure prevents
publishing a completed profile; large timing errors remain visible in a successful run.
Concurrent RDMA uses an approximate serialized channel model; independent processes/ranks, GPU RDMA endpoints, inter-GPU P2P and distributed framework API totals are not calibrated. Gather validation uses one GPU/direction per submit; mixed batches aggregate fixed costs per direction. SSD models do not track device cache/placement/post-write state, queue saturation or preparation/cleanup overlap; mixed read/write submits are not independently calibrated. Check heldout errors for the actual deployment before reusing parameters.
Run fitting/isolation tests with NumPy and pytest:
python -m pytest -q tests/python/umbp/test_sim_calibration.py
Hardware accuracy is verified with actual probes, not generated-token correctness.