Triggered Plans#

Experimental (iris.experimental.plan): may change between releases without deprecation.

Copy-engine transfers prepared once and released by kernels, every round. See Triggered Plans for the model and its ordering rules.

TriggeredPlan#

class TriggeredPlan(ctx, buffer, schedule, timeout=30.0, destination=None, prepare_ahead=False)[source]#

Bases: object

Sends regions of buffer from the rank that writes them to other ranks, using copy engines that a kernel releases chunk by chunk.

Each rank passes its own schedule: the chunks it writes, in the order its producer completes them. A chunk goes to its peers (by default every other rank) and lands at each region’s destination offset in their destination (by default the same offset in their buffer). Chunk ids are global: rank r’s chunks come after those of ranks 0..r-1 (see chunk_id), and schedules[r] holds rank r’s schedule. A rank that writes nothing passes []. No two ranks may write the same bytes of a rank’s destination.

Collective: every rank creates the plan for the same symmetric buffer and destination. Use it as a context manager, or call close() on every rank when done.

Parameters:
  • ctx – Iris instance, created with engines=("sdma",).

  • buffer – Contiguous tensor on the symmetric heap.

  • schedule – This rank’s list of Chunk, in the order it completes them.

  • timeout – Seconds wait(), close() and the round fence of next_round() wait before raising.

  • destination – Contiguous tensor on the symmetric heap that peers’ chunks land in; buffer if None. It must not overlap buffer unless it is buffer.

  • prepare_ahead – Allow prepare_next(). The plan then holds queue space for two rounds instead of one, so one round may use half as much of each queue’s ring.

counters#

Contiguous int32 tensor, one counter per chunk id. Adding 1 to a chunk’s counter is what mark_ready does, so another kernel (such as a GEMM epilogue that signals per tile) may increment this rank’s counters directly, under the same rules.

__init__(ctx, buffer, schedule, timeout=30.0, destination=None, prepare_ahead=False)[source]#
chunk_id(rank, index)[source]#

Global id of chunk index in rank rank’s schedule.

start()[source]#

Start the prepared round and return the handle tensor its kernels take.

The round is set in the handle by work queued on the current stream: launch the kernels that take it on that stream, or make their stream wait for it.

wait(timeout=None, blocking=True)[source]#

Wait until every chunk is delivered this round: own ones at every peer, others here.

With blocking=False, queue the wait on the current stream instead and return at once: kernels queued after it see the round’s data, and the host never synchronizes. An error (a timeout, extra marks, a peer that failed the round, or a round fence that timed out) is then raised by the next blocking wait(), or by close().

prepare_next()[source]#

Prepare the next round’s copies now, so next_round() does not have to. Not collective.

May be called once the current round has started, on a plan created with prepare_ahead=True. The copies wait behind gates that no producer can open before every rank has passed next_round(). If preparing fails, next_round() raises the error after its fence, so no rank is left waiting in it.

next_round()[source]#

Collective: move to the next round, preparing it first unless prepare_next() did.

Calling it declares that this rank is done reading the current round’s data: once every rank has called it, a peer may start the next round, and its copies can overwrite buffer at any time. The fence is a kernel on the current stream, so the host does not wait: work queued on that stream before the call is covered, and nothing issued after it may read buffer until the next wait(). A peer that never reaches the fence is recorded as a timeout after the plan’s timeout, and raised by the next blocking wait() or close().

If this rank cannot prepare the next round, it still takes part in the fence, telling its peers, whose wait() for that round then raises; then this call raises. A rank whose plan has recorded an error (from a stream-ordered wait()) does the same, without raising here.

run(timeout=None)[source]#

Run one round with no producer kernel: send every own chunk as it stands, then wait.

close(timeout=None, check=True)[source]#

Collective: wait until every rank is done with the plan, then free it.

Prepared copies cannot be withdrawn from the queue, so closing a plan whose round has not finished sends this rank’s chunks as they stand (for every prepared round). A peer’s copies and polls may still touch this rank’s buffer and gates, so every rank must call close() before any of them is freed. Then raises any error a stream-ordered wait() recorded, unless check is false.

Plans may be closed in any order. If a plan prepared earlier on the same queues still has copies waiting, this plan’s copies are queued behind them and cannot run yet: close() then returns with the plan CLOSING, and closing the earlier plan finishes this one too.

Schedules#

class Region(offset_bytes, row_bytes, rows=1, stride_bytes=None, dst_offset_bytes=None, dst_stride_bytes=None)[source]#

Bases: object

One copy: rows rows of row_bytes bytes, stride_bytes apart (packed if None), starting offset_bytes into the plan’s buffer. Each peer receives them dst_offset_bytes into its destination (the plan’s destination, by default the buffer itself), rows dst_stride_bytes apart; both default to the source’s.

Parameters:
  • offset_bytes (int)

  • row_bytes (int)

  • rows (int)

  • stride_bytes (int | None)

  • dst_offset_bytes (int | None)

  • dst_stride_bytes (int | None)

offset_bytes: int#
row_bytes: int#
rows: int = 1#
stride_bytes: int | None = None#
dst_offset_bytes: int | None = None#
dst_stride_bytes: int | None = None#
property row_stride#

Bytes between the starts of consecutive rows (row_bytes if packed).

property end#

Byte offset one past the last byte this region copies.

property dst_offset#

Where the region lands in each peer’s destination.

property dst_row_stride#

Bytes between the starts of consecutive rows at the destination.

property dst_end#

Destination byte offset one past the last byte this region writes.

__init__(offset_bytes, row_bytes, rows=1, stride_bytes=None, dst_offset_bytes=None, dst_stride_bytes=None)#
Parameters:
  • offset_bytes (int)

  • row_bytes (int)

  • rows (int)

  • stride_bytes (int | None)

  • dst_offset_bytes (int | None)

  • dst_stride_bytes (int | None)

Return type:

None

class Chunk(regions, writers=1, peers=None, group=None)[source]#

Bases: object

Regions sent together once all writers have marked the chunk ready, every round.

peers are the ranks it goes to (every other rank if None). Chunks with the same group (a small integer, shared across ranks) signal one arrival counter at each receiver, so a consumer can wait for all of them at once with wait_group.

Parameters:
regions: tuple#
writers: int = 1#
peers: tuple | None = None#
group: int | None = None#
recipients(sender, world_size)[source]#

The ranks this chunk goes to, when sender sends it.

property is_2d#

True if any region has more than one row.

__init__(regions, writers=1, peers=None, group=None)#
Parameters:
Return type:

None

whole(view, base=None, writers=1, dst=None, dst_base=None, peers=None, group=None)[source]#

One chunk covering a contiguous view, marked ready by writers programs per round.

By default it lands where it is, in each peer’s buffer. With dst (a contiguous view of the same shape inside dst_base, the plan’s destination), it lands there instead. peers limits the ranks it goes to, and group is its arrival group.

chunks(view, n, base=None, writers=1, dst=None, dst_base=None, peers=None, group=None)[source]#

Split a contiguous view along dim 0 into n chunks, in order. Suits producers that finish row blocks in order: row-block, streaming and elementwise kernels. dst, dst_base, peers and group are as for whole(); every chunk joins group.

tile_chunks(view, block_shape, tiles_per_chunk, order=None, base=None, dst=None, dst_base=None, peers=None, group=None)[source]#

Group the tiles of a tiled producer into chunks of tiles_per_chunk, in completion order.

view is a 2D tensor with unit column stride, split into block_shape tiles (edge tiles clipped; tile ids are row-major). order gives the tile the producer makes at each step i: a list of tile ids, or a function of i. Chunk k holds the tiles of steps k * tiles_per_chunk up to the next chunk, one region per tile, and is marked ready once per tile. The producer must mark the tile of step i ready as part of chunk i // tiles_per_chunk, so pass the mapping the kernel itself uses: a @triton.jit mapping written with plain integer arithmetic runs on the host as lambda i: mapping.fn(i, ...).

For a persistent kernel of P programs where program p takes steps p, p + P, …, use tiles_per_chunk=P: chunk k is every program’s k-th tile, so chunks complete in order.

With dst (a 2D view of the same shape and unit column stride inside dst_base, the plan’s destination), each tile lands at its place in dst instead, with dst’s row pitch. peers limits the ranks every chunk goes to, and every chunk joins group.

TriggeredView (device)#

class TriggeredView(view, gates, round, rank, num_chunks)[source]#

Bases: object

Device handle for one round of a triggered plan.

Usage:

@triton.jit
def producer(out, handle, ...):
    cv = TriggeredView.initialize(handle)
    ...                          # write this program's part of chunk c
    cv.mark_ready(c)             # whole CTA, after its writes

@triton.jit
def consumer(out, handle, ...):
    cv = TriggeredView.initialize(handle)
    cv.wait(c)                   # chunk c's data is now visible
    x = tl.load(out + ...)
Parameters:
  • view (tensor)

  • gates (tensor)

  • round (tensor)

  • rank (tensor)

  • num_chunks (tensor)

static initialize(handle)[source]#

Read the plan’s handle tensor (from TriggeredPlan.start()).

owns(chunk)[source]#

True if this rank produces chunk chunk.

mark_ready(chunk)[source]#

Declare this CTA’s writes to chunk chunk done; the chunk is sent once all of its writers have.

Call from the whole CTA, outside divergent branches, after the CTA’s writes to the chunk.

arrived(chunk)[source]#

True once chunk chunk is visible here this round: for this rank’s own chunks, once marked ready; for another rank’s chunk in a group, once the whole group has arrived here. Always true of chunks not sent to this rank.

Call uniformly across the CTA: the result reaches the other waves through a CTA barrier, so every wave must make the same number of calls (true of the other waits too).

wait(chunk)[source]#

Spin until arrived(chunk). Each wave first polls with its own loads, which do not serialize with the copy engines’ atomics on the word; the acquire check that follows is CTA-uniform.

group_arrived(group)[source]#

True once every chunk of group sent to this rank has arrived this round. Always true of a group with no chunks sent here, including a group no chunk uses (a debug build asserts on that, since it is likely a typo).

wait_group(group)[source]#

Spin until group_arrived(group), polling each wave’s own loads first (see wait).

sent()[source]#

True once every peer has received this rank’s chunks this round, so they may be overwritten.

wait_sent()[source]#

Spin until sent(), polling each wave’s own loads first (see wait).

TriggeredView (Gluon device)#

class TriggeredView(view, gates, round, rank, num_chunks)[source]#

Bases: object

Gluon device handle for one round of a triggered plan.

Usage:

@gluon.jit
def producer(out, handle, ...):
    cv = TriggeredView.initialize(handle)
    ...                          # write this program's part of chunk c
    cv.mark_ready(c)             # whole CTA, after its writes

@gluon.jit
def consumer(out, handle, ...):
    cv = TriggeredView.initialize(handle)
    cv.wait(c)                   # chunk c's data is now visible
    x = gl.load(out + ...)
Parameters:
  • view (tensor)

  • gates (tensor)

  • round (tensor)

  • rank (tensor)

  • num_chunks (tensor)

static initialize(handle)[source]#

Read the plan’s handle tensor (from TriggeredPlan.start()).

owns(chunk)[source]#

True if this rank produces chunk chunk.

mark_ready(chunk)[source]#

Declare this CTA’s writes to chunk chunk done; the chunk is sent once all of its writers have.

Call from the whole CTA, after the CTA’s writes to the chunk: outside divergent branches and outside gl.warp_specialize partitions.

arrived(chunk)[source]#

True once chunk chunk is visible here this round: for this rank’s own chunks, once marked ready; for another rank’s chunk in a group, once the whole group has arrived here. Always true of chunks not sent to this rank.

Call uniformly across the CTA: the result reaches the other waves through a CTA barrier, so every wave must make the same number of calls (true of the other waits too).

wait(chunk)[source]#

Spin until arrived(chunk). Each wave first polls with its own loads, which do not serialize with the copy engines’ atomics on the word; the acquire check that follows is CTA-uniform.

group_arrived(group)[source]#

True once every chunk of group sent to this rank has arrived this round. Always true of a group with no chunks sent here, including a group no chunk uses (a debug build asserts on that, since it is likely a typo).

wait_group(group)[source]#

Spin until group_arrived(group), polling each wave’s own loads first (see wait).

sent()[source]#

True once every peer has received this rank’s chunks this round, so they may be overwritten.

wait_sent()[source]#

Spin until sent(), polling each wave’s own loads first (see wait).