hipObject: RDMA-Accelerated S3 Object Storage for GPUs

Warning

hipObject is an experimental library. It is not intended for production use.

Introduction

hipObject enables direct data transfers between AMD GPU VRAM and S3-compatible object storage using RDMA over RoCEv2. It interoperates with NVIDIA cuObject-equipped storage servers via the x-amz-rdma-token S3 header protocol, providing a vendor-neutral client for GPU-direct object storage.

Architecture

hipObject separates control and data planes:

• Control plane: Standard S3 REST requests augmented with x-amz-rdma-token / x-amz-rdma-reply headers, made by the application’s S3 client.

• Data plane: RDMA READ/WRITE over RoCEv2 RC transport directly between GPU VRAM and storage server buffers.

See Architecture for how the library works.

Transfers

hipObject doesn’t make S3 requests. The application’s S3 client sends each request, with an RDMA token from hipObject in its x-amz-rdma-token header, and the server moves the data by RDMA:

Call	The application’s S3 request	The server
hipObjGet()	GetObject, for a whole object or a byte range	RDMA WRITEs into the buffer
hipObjPut()	PutObject, or UploadPart of a multipart upload	RDMA READs from the buffer

The request selects the object, and the part of it, to transfer; the token only describes the buffer. The buffer can be in GPU or host memory.

Getting Started

Building hipObject covers building and installing hipObject, and using it from a CMake project. The README’s Quick Start is a short program that reads an object into GPU memory, and the examples, which are installed in share/doc/hipobject<mm>/examples, are complete programs that make their S3 requests with libcurl.

User Guide

• Building hipObject
  • Prerequisites
  • Build Steps
  • Using hipObject
  • CMake Options
  • Packaging
  • MinIO C++ RDMA Bridge
  • cuObject Interoperability
  • Building Documentation
  • Environment Variables
• Architecture
  • What the Library Does
  • Source Map
  • Layers
  • The Instance
  • The Platform
  • NIC and GID Selection
  • Connections
  • Buffer Registration
  • A Transfer
  • Staging Copies
  • Concurrency
  • Errors
  • Configuration
  • Open Questions
• hipObject cuObject Interoperability
  • Compatibility matrix
  • Testing paths
  • Hardware topology
  • CMake options

API Reference

• API Reference
  • Core Functionality
  • Thread Safety
  • Errors and Error Handling
  • Buffer Registration
  • Data Transfer (GET / PUT)

License

MIT License