hipObject cuObject Interoperability

hipObject cuObject Interoperability#

This guide summarizes compatibility with NVIDIA cuObject v1.2.0 and documents how to test hipObject on lab hardware.

Compatibility matrix#

Feature

cuObject v1.2.0

hipObject

S3 x-amz-rdma-token

Yes

Yes

S3 x-amz-rdma-reply

Yes

Yes

GET / PUT / multipart

Yes

Yes (via SDK)

RDMA transport

DC (ConnectX)

RC (bnxt/ionic)

Direct cuObject server

N/A

adapter

Control plane#

hipObject is wire-compatible with cuObject at the S3 header level. Applications embed a hex-encoded RDMA token in x-amz-rdma-token and read x-amz-rdma-reply from the HTTP response.

x-amz-rdma-token

The header value is exactly one fixed-width RDMA token encoded as 88 hex characters (44 binary bytes). No suffix or additional :-separated component is permitted in the header value.

The binary payload layout is:

Byte

Width

Type

Meaning

0

1 byte

u8

transport

1

4 bytes

u32 LE

QP number

5

16 bytes

bytes

GID, copied verbatim

21

4 bytes

u32 LE

rkey

25

8 bytes

u64 LE

remote address

33

8 bytes

u64 LE

length

41

1 byte

u8

port number

42

2 bytes

u16 LE

LID

The fields are packed with no padding between them. The integers are little-endian whatever the byte order of the host that encodes or decodes the token.

Enumerated values:

  • transport = 0x00: DC

  • transport = 0x01: RC

encodeRdmaToken() emits lowercase hex, but receivers accept either hex case. A peer that validates the token as a fixed-length hex string will reject any value longer than 88 hex characters.

x-amz-rdma-reply

Clients must accept these forms:

  • ok — legacy success marker, deprecated

  • err — legacy error marker, deprecated

  • <http-status> — decimal HTTP status code such as 200 or 501

  • 200:<server-token-hex> — RC peer handshake reply carrying one 88-hex RDMA token

Only 200:<server-token-hex> carries a peer token. Legacy ok / err remain accepted for cuObject compatibility but new implementations should prefer numeric status codes, and RC peer handshakes should use the 200:<server-token-hex> form.

Data plane#

cuObject v1.2.0 requires Dynamic Connection (DC) transport on NVIDIA ConnectX NICs. hipObject uses Reliable Connection (RC) on Broadcom Thor-2 and AMD Pensando Pollara NICs.

A stock libcuobjserver stack cannot serve RC clients directly. An RC-to-DC adapter bridges AMD RC clients to cuObject-equipped storage gateways in production deployments.

Testing paths#

Layer 1 — Unit tests (CI)#

Token encoding, reply parsing, and header formatting run in GitHub Actions without GPU or RDMA hardware:

cmake -B build -DBUILD_TESTING=ON
cmake --build build
ctest --test-dir build -R test-rdma-token --output-on-failure

Layer 2 — RC integration test server#

hipobj-rdma-test-server is an in-repo S3+RDMA harness that speaks RC and returns peer tokens for the client-side QP handshake:

cmake -B build -DBUILD_TESTING=ON
cmake --build build --target hipobj-rdma-test-server
./build/test/integration/rdma-test-server/hipobj-rdma-test-server 9000

On a GPU client node with libcurl installed:

cmake -B build -DBUILD_TESTING=ON
cmake --build build --target get-object
./build/hipobj-examples/get-object 4096 --live http://127.0.0.1:9000

Layer 3 — MinIO AIStor bridge#

Build the minio-cpp bridge for production-style S3 signing and retries:

cmake -B build -DHIPOBJ_MINIO_CLIENT=ON
cmake --build build --target minio-getput-rdma

See integrations/minio-cpp/TESTING.md for AIStor + cuObjServer setup.

Layer 4 — cuObject server probe (optional)#

When libcuobjserver is installed locally:

cmake -B build \
  -DHIPOBJ_FIND_CUOBJECT_SERVER=ON \
  -DCUOBJSERVER_ROOT=/opt/nvidia/cuobjserver \
  -DBUILD_TESTING=ON
cmake --build build --target hipobj-cuobject-probe

This verifies linkage only. End-to-end cuObject interop still requires ConnectX hardware and the RC adapter on the storage gateway.

Hardware topology#

AMD GPU client                Storage server
+----------------+            +------------------+
| MI300 + hipObj |  RoCEv2 RC  | RC adapter       |
| bnxt / ionic   | ----------> | cuObjServer (DC) |
+----------------+            | MinIO AIStor     |
                              +------------------+

CMake options#

HIPOBJ_FETCH_CUOBJECT_CLIENT

Download libcuobjclient v1.2.0.59 from the NVIDIA CUDA CDN.

HIPOBJ_FIND_CUOBJECT_SERVER

Locate a system libcuobjserver and build hipobj-cuobject-probe.

HIPOBJ_INTEGRATION_TESTS

Build hipobj-rdma-test-server (default ON when testing is enabled).

HIPOBJ_MINIO_CLIENT

Build the minio-cpp RDMA bridge and minio-getput-rdma example.