API Reference

API Reference#

This page documents the hipObject public C API, extracted from annotated source headers by Doxygen and rendered via Breathe.

Core Functionality#

typedef void *hipObjHandle_t#: Opaque handle to an S3 object for RDMA I/O.

const char *hipObjGetVersionString(void)#

Return the library version as a string.

Thread safety: Concurrent: can be called from any thread at any time, including from a callback, as long as calls that run at the same time don’t share a buffer that one of them writes to or a token that one of them releases. See Thread Safety.

hipObjError_t hipObjInit(hipObjConfig_t *config)#

Initialize the hipObject library.

Opens the RDMA device, selects the closest NIC to the GPU, and prepares internal state. Must be called before any other hipObj* function.

Thread safety: Serialized: runs one at a time with the other serialized functions, and returns hipObjInvalidValue if called from a callback. See Thread Safety.

Parameters:: config – Pointer to configuration struct
Returns:: hipObjError_t

hipObjError_t hipObjShutdown(void)#

Shut down the hipObject library.

Releases all RDMA resources. All registered buffers are implicitly deregistered.

Thread safety: Serialized: runs one at a time with the other serialized functions, and returns hipObjInvalidValue if called from a callback. See Thread Safety.

Returns:: hipObjError_t

HIPOBJ_VERSION_MAJOR#: hipObject major version number

HIPOBJ_VERSION_MINOR#: hipObject minor version number

HIPOBJ_VERSION_PATCH#: hipObject patch version number

struct hipObjConfig_t#

#include <hipobj.h>

Configuration for hipObject initialization.

Public Members

const char *accessKey#: S3 access key (optional)

const char *endpoint#: S3 endpoint URL

uint32_t flags#: Reserved, set to 0

int gpuDevice#: HIP device index, or -1

const char *nicHint#: NIC name hint, or NULL

const char *region#: AWS region

const char *secretKey#: S3 secret key (optional)

struct hipObjOps_t#

#include <hipobj.h>

Callback struct for S3 SDK integration.

The application provides these callbacks so that hipObject can inject RDMA tokens into S3 requests and receive RDMA reply tags from the server. This mirrors cuObject’s CUObjOps_t pattern.

The callbacks run while hipObject holds its API lock, so they can’t call serialized functions; see Thread Safety.

Public Members

int (*recvReply)(void *ctx, char *reply, size_t *replyLen)#

Called by hipObject to receive the RDMA reply. The application should extract x-amz-rdma-reply from the S3 response headers.

Param ctx:: User-supplied context pointer
Param reply:: Buffer to receive the reply tag
Param replyLen:: In: buffer size; out: actual length
Return:: 0 on success, negative on failure

int (*sendRequest)(void *ctx, const char *token, size_t tokenLen)#

Called by hipObject to send the RDMA token. The application should embed the token in the x-amz-rdma-token S3 request header.

Param ctx:: User-supplied context pointer
Param token:: Hex-encoded RDMA token
Param tokenLen:: Length of token in bytes
Return:: 0 on success, negative on failure

Thread Safety#

hipObject can be called from any number of threads. Each function’s documentation says which of these it is:

Serialized functions use the library’s state: whether it’s initialized, the registered buffers, and the RDMA connection. They run one at a time across the whole process. Each one takes a library-wide lock and holds it until it returns, so a serialized call waits for the one in progress, on any thread, to finish.
Concurrent functions don’t use the library’s state. They can be called from any thread at any time, including while a serialized function runs and from a callback. As with any C function, calls that run at the same time mustn’t share a buffer that one of them writes to, or a token that one of them releases.

Since serialized calls run one at a time, only one transfer is in progress at a time, and calling hipObject from more threads doesn’t move data any faster. A call can also wait a long time for another thread’s transfer, which holds the lock for as long as the transfer takes: while the application’s callbacks send the S3 request and wait for the response, while hipObject copies the buffer through host memory (for a buffer that the NIC can’t reach directly), and while it waits for the RDMA transfer to complete.

hipObjShutdown() waits for the serialized call in progress to finish. Calls made after it, on any thread, that need an initialized library fail with hipObjNotInitialized until hipObjInit() is called again.

Callbacks. hipObjGet() and hipObjPut() run the hipObjOps_t callbacks with the lock held. A callback can call concurrent functions. If it calls a serialized function, that function returns hipObjInvalidValue instead of waiting for a lock its own thread holds. A callback also mustn’t wait for another thread that’s calling a serialized function, since that thread is waiting for the lock: neither can continue, and hipObject can’t detect it.

Transfers made of several calls. The lock covers one call, not a sequence of calls. A transfer that the application drives itself, with hipObjGetRdmaToken(), its own S3 request, and hipObjBufSync(), is several calls, and other threads’ calls can run between them. hipObject uses one RDMA connection for every transfer, so the application must make sure that such a transfer doesn’t overlap any other transfer, on any thread.

Serializing the library is how hipObject works now, not a promise: a later release may run transfers concurrently.

Errors and Error Handling#

enum hipObjOpError_t#

Operation-level error codes.

Values:

enumerator hipObjSuccess#

enumerator hipObjInvalidValue#

enumerator hipObjNotInitialized#

enumerator hipObjAlreadyInitialized#

enumerator hipObjRdmaError#

enumerator hipObjS3Error#

enumerator hipObjBufNotRegistered#

enumerator hipObjBufAlreadyRegistered#

enumerator hipObjNicNotFound#

enumerator hipObjDmabufNotSupported#

enumerator hipObjSizeTooLarge#

enumerator hipObjInternalError#

const char *hipObjGetErrorString(hipObjOpError_t err)#

Return a human-readable string for an operation error code.

Thread safety: Concurrent: can be called from any thread at any time, including from a callback, as long as calls that run at the same time don’t share a buffer that one of them writes to or a token that one of them releases. See Thread Safety.

HIPOBJ_BASE_ERR#: Base value for hipObject error codes.

struct hipObjError_t#: #include <hipobj.h>

Compound error type carrying both an operation error and an optional HIP error.

Buffer Registration#

hipObjError_t hipObjBufDeregister(void *devPtr)#

Deregister a previously registered buffer.

Thread safety: Serialized: runs one at a time with the other serialized functions, and returns hipObjInvalidValue if called from a callback. See Thread Safety.

Parameters:: devPtr – Pointer previously passed to hipObjBufRegister or hipObjBufRegisterHost
Returns:: hipObjError_t

hipObjError_t hipObjBufRegister(void *devPtr, size_t size)#

Exports the GPU buffer via dmabuf and registers it with the RDMA NIC. Maximum 4 GiB per registration. The buffer must lie within a single HIP allocation: returns hipObjInvalidValue if it runs past the end of the allocation that holds devPtr.

Thread safety: Serialized: runs one at a time with the other serialized functions, and returns hipObjInvalidValue if called from a callback. See Thread Safety.

Parameters:

devPtr – Pointer returned by hipMalloc
size – Size of the buffer in bytes

Returns:

hipObjError_t

hipObjError_t hipObjBufRegisterHost(void *hostPtr, size_t size)#

Registers user-owned CPU RAM, such as a buffer allocated with malloc() or hipHostMalloc(), with the RDMA NIC. hipObject does not allocate or free the host buffer. Maximum 4 GiB per registration.

Thread safety: Serialized: runs one at a time with the other serialized functions, and returns hipObjInvalidValue if called from a callback. See Thread Safety.

Parameters:

hostPtr – Pointer to CPU-accessible memory
size – Size of the buffer in bytes

Returns:

hipObjError_t

Data Transfer (GET / PUT)#

hipObjError_t hipObjBufSync(void *devPtr, size_t size, off_t offset, int direction)#

Stage a registered buffer between device and host memory.

When a device pointer cannot be registered with the NIC directly (no dmabuf support, or a fabric with no peer-to-peer path to the GPU), hipObjBufRegister() registers a host staging buffer instead and the peer’s RDMA reads and writes land there. Callers driving a transfer through hipObjGetRdmaToken() must therefore call this with HIPOBJ_SYNC_TO_HOST before a PUT and HIPOBJ_SYNC_TO_DEVICE after a GET. It is a no-op, reporting success, for a directly registered buffer; hipObjGet() and hipObjPut() do it themselves.

Thread safety: Serialized: runs one at a time with the other serialized functions, and returns hipObjInvalidValue if called from a callback. See Thread Safety.

hipObjError_t hipObjGet(hipObjHandle_t handle, void *devPtr, size_t size, off_t offset, hipObjOps_t *ops, void *ctx)#

GET: fetch an S3 object into a registered buffer via RDMA.

The server performs an RDMA WRITE to push data into the registered buffer, starting offset bytes into it. The bytes from offset to offset + size must lie inside the buffer. hipObject doesn’t send an object offset; the request the callbacks make selects the part of the object to fetch.

Thread safety: Serialized: runs one at a time with the other serialized functions, and returns hipObjInvalidValue if called from a callback. See Thread Safety.

Parameters:

handle – S3 object handle (from application)
devPtr – Registered GPU or host buffer
size – Number of bytes to transfer
offset – Byte offset into the registered buffer
ops – S3 SDK callbacks
ctx – User context passed to callbacks

Returns:

hipObjError_t

hipObjError_t hipObjGetRdmaToken(const void *devPtr, size_t size, int op, char **outToken)#

Mint a hex-encoded RC RDMA token for a registered buffer.

The caller must release *outToken with hipObjPutRdmaToken(). op is HIPOBJ_RDMA_OP_PUT or HIPOBJ_RDMA_OP_GET (reserved).

Thread safety: Serialized: runs one at a time with the other serialized functions, and returns hipObjInvalidValue if called from a callback. See Thread Safety.

hipObjError_t hipObjParseRdmaReply(const char *reply, size_t replyLen, int *httpCode)#

Parse an x-amz-rdma-reply header value.

On success writes the HTTP-style reply code (200, 204, 206, 501) to httpCode.

Thread safety: Concurrent: can be called from any thread at any time, including from a callback, as long as calls that run at the same time don’t share a buffer that one of them writes to or a token that one of them releases. See Thread Safety.

hipObjError_t hipObjPut(hipObjHandle_t handle, const void *devPtr, size_t size, off_t offset, hipObjOps_t *ops, void *ctx)#

PUT: store a registered buffer to an S3 object via RDMA.

The server performs an RDMA READ to pull data from the registered buffer, starting offset bytes into it. The bytes from offset to offset + size must lie inside the buffer. hipObject doesn’t send an object offset; the request the callbacks make selects where the data goes in the object.

Thread safety: Serialized: runs one at a time with the other serialized functions, and returns hipObjInvalidValue if called from a callback. See Thread Safety.

Parameters:

handle – S3 object handle (from application)
devPtr – Registered GPU or host buffer
size – Number of bytes to transfer
offset – Byte offset into the registered buffer
ops – S3 SDK callbacks
ctx – User context passed to callbacks

Returns:

hipObjError_t

hipObjError_t hipObjPutRdmaToken(char *token)#

Release a token allocated by hipObjGetRdmaToken()

Thread safety: Concurrent: can be called from any thread at any time, including from a callback, as long as calls that run at the same time don’t share a buffer that one of them writes to or a token that one of them releases. See Thread Safety.

hipObjError_t hipObjTokenClientNic(const char *token, char *nicIp, size_t nicIpLen)#

Extract client NIC IPv4 from a minted RDMA token.

Writes a dotted-quad address into nicIp when the token GID carries an IPv4-mapped RoCEv2 suffix. Returns hipObjSuccess with an empty string when no address is encoded.

Thread safety: Concurrent: can be called from any thread at any time, including from a callback, as long as calls that run at the same time don’t share a buffer that one of them writes to or a token that one of them releases. See Thread Safety.

HIPOBJ_RDMA_OP_GET#: RDMA token operation: server RDMA WRITE (client GET)

HIPOBJ_RDMA_OP_PUT#: RDMA token operation: server RDMA READ (client PUT)

HIPOBJ_RDMA_REPLY_NOT_IMPLEMENTED#: x-amz-rdma-reply: server declined RDMA (HTTP fallback)

HIPOBJ_SYNC_TO_DEVICE#: hipObjBufSync direction: staging buffer -> device

HIPOBJ_SYNC_TO_HOST#: hipObjBufSync direction: device -> staging buffer

API Reference

Contents

API Reference#

Core Functionality#

Thread Safety#

Errors and Error Handling#

Buffer Registration#

Data Transfer (GET / PUT)#