Triggered Plans#
Experimental (iris.experimental.plan): may change between releases without deprecation.
Copy-engine transfers prepared once and released by kernels, every round. See Triggered Plans for the model and its ordering rules.
TriggeredPlan#
- class TriggeredPlan(ctx, buffer, schedule, timeout=30.0, destination=None, prepare_ahead=False)[source]#
Bases:
objectSends regions of
bufferfrom the rank that writes them to other ranks, using copy engines that a kernel releases chunk by chunk.Each rank passes its own
schedule: the chunks it writes, in the order its producer completes them. A chunk goes to itspeers(by default every other rank) and lands at each region’s destination offset in theirdestination(by default the same offset in theirbuffer). Chunk ids are global: rankr’s chunks come after those of ranks0..r-1(seechunk_id), andschedules[r]holds rankr’s schedule. A rank that writes nothing passes[]. No two ranks may write the same bytes of a rank’s destination.Collective: every rank creates the plan for the same symmetric
bufferanddestination. Use it as a context manager, or callclose()on every rank when done.- Parameters:
ctx – Iris instance, created with
engines=("sdma",).buffer – Contiguous tensor on the symmetric heap.
schedule – This rank’s list of
Chunk, in the order it completes them.timeout – Seconds
wait(),close()and the round fence ofnext_round()wait before raising.destination – Contiguous tensor on the symmetric heap that peers’ chunks land in;
bufferifNone. It must not overlapbufferunless it isbuffer.prepare_ahead – Allow
prepare_next(). The plan then holds queue space for two rounds instead of one, so one round may use half as much of each queue’s ring.
- counters#
Contiguous int32 tensor, one counter per chunk id. Adding 1 to a chunk’s counter is what
mark_readydoes, so another kernel (such as a GEMM epilogue that signals per tile) may increment this rank’s counters directly, under the same rules.
- start()[source]#
Start the prepared round and return the handle tensor its kernels take.
The round is set in the handle by work queued on the current stream: launch the kernels that take it on that stream, or make their stream wait for it.
- wait(timeout=None, blocking=True)[source]#
Wait until every chunk is delivered this round: own ones at every peer, others here.
With
blocking=False, queue the wait on the current stream instead and return at once: kernels queued after it see the round’s data, and the host never synchronizes. An error (a timeout, extra marks, a peer that failed the round, or a round fence that timed out) is then raised by the next blockingwait(), or byclose().
- prepare_next()[source]#
Prepare the next round’s copies now, so
next_round()does not have to. Not collective.May be called once the current round has started, on a plan created with
prepare_ahead=True. The copies wait behind gates that no producer can open before every rank has passednext_round(). If preparing fails,next_round()raises the error after its fence, so no rank is left waiting in it.
- next_round()[source]#
Collective: move to the next round, preparing it first unless
prepare_next()did.Calling it declares that this rank is done reading the current round’s data: once every rank has called it, a peer may start the next round, and its copies can overwrite
bufferat any time. The fence is a kernel on the current stream, so the host does not wait: work queued on that stream before the call is covered, and nothing issued after it may readbufferuntil the nextwait(). A peer that never reaches the fence is recorded as a timeout after the plan’s timeout, and raised by the next blockingwait()orclose().If this rank cannot prepare the next round, it still takes part in the fence, telling its peers, whose
wait()for that round then raises; then this call raises. A rank whose plan has recorded an error (from a stream-orderedwait()) does the same, without raising here.
- run(timeout=None)[source]#
Run one round with no producer kernel: send every own chunk as it stands, then wait.
- close(timeout=None, check=True)[source]#
Collective: wait until every rank is done with the plan, then free it.
Prepared copies cannot be withdrawn from the queue, so closing a plan whose round has not finished sends this rank’s chunks as they stand (for every prepared round). A peer’s copies and polls may still touch this rank’s buffer and gates, so every rank must call close() before any of them is freed. Then raises any error a stream-ordered
wait()recorded, unlesscheckis false.Plans may be closed in any order. If a plan prepared earlier on the same queues still has copies waiting, this plan’s copies are queued behind them and cannot run yet: close() then returns with the plan
CLOSING, and closing the earlier plan finishes this one too.
Schedules#
- class Region(offset_bytes, row_bytes, rows=1, stride_bytes=None, dst_offset_bytes=None, dst_stride_bytes=None)[source]#
Bases:
objectOne copy:
rowsrows ofrow_bytesbytes,stride_bytesapart (packed ifNone), startingoffset_bytesinto the plan’s buffer. Each peer receives themdst_offset_bytesinto its destination (the plan’sdestination, by default the buffer itself), rowsdst_stride_bytesapart; both default to the source’s.- Parameters:
- property row_stride#
Bytes between the starts of consecutive rows (
row_bytesif packed).
- property end#
Byte offset one past the last byte this region copies.
- property dst_offset#
Where the region lands in each peer’s destination.
- property dst_row_stride#
Bytes between the starts of consecutive rows at the destination.
- property dst_end#
Destination byte offset one past the last byte this region writes.
- class Chunk(regions, writers=1, peers=None, group=None)[source]#
Bases:
objectRegions sent together once all
writershave marked the chunk ready, every round.peersare the ranks it goes to (every other rank ifNone). Chunks with the samegroup(a small integer, shared across ranks) signal one arrival counter at each receiver, so a consumer can wait for all of them at once withwait_group.- property is_2d#
True if any region has more than one row.
- whole(view, base=None, writers=1, dst=None, dst_base=None, peers=None, group=None)[source]#
One chunk covering a contiguous
view, marked ready bywritersprograms per round.By default it lands where it is, in each peer’s buffer. With
dst(a contiguous view of the same shape insidedst_base, the plan’s destination), it lands there instead.peerslimits the ranks it goes to, andgroupis its arrival group.
- chunks(view, n, base=None, writers=1, dst=None, dst_base=None, peers=None, group=None)[source]#
Split a contiguous
viewalong dim 0 intonchunks, in order. Suits producers that finish row blocks in order: row-block, streaming and elementwise kernels.dst,dst_base,peersandgroupare as forwhole(); every chunk joinsgroup.
- tile_chunks(view, block_shape, tiles_per_chunk, order=None, base=None, dst=None, dst_base=None, peers=None, group=None)[source]#
Group the tiles of a tiled producer into chunks of
tiles_per_chunk, in completion order.viewis a 2D tensor with unit column stride, split intoblock_shapetiles (edge tiles clipped; tile ids are row-major).ordergives the tile the producer makes at each stepi: a list of tile ids, or a function ofi. Chunkkholds the tiles of stepsk * tiles_per_chunkup to the next chunk, one region per tile, and is marked ready once per tile. The producer must mark the tile of stepiready as part of chunki // tiles_per_chunk, so pass the mapping the kernel itself uses: a@triton.jitmapping written with plain integer arithmetic runs on the host aslambda i: mapping.fn(i, ...).For a persistent kernel of
Pprograms where programptakes stepsp,p + P, …, usetiles_per_chunk=P: chunkkis every program’sk-th tile, so chunks complete in order.With
dst(a 2D view of the same shape and unit column stride insidedst_base, the plan’s destination), each tile lands at its place indstinstead, withdst’s row pitch.peerslimits the ranks every chunk goes to, and every chunk joinsgroup.
TriggeredView (device)#
- class TriggeredView(view, gates, round, rank, num_chunks)[source]#
Bases:
objectDevice handle for one round of a triggered plan.
Usage:
@triton.jit def producer(out, handle, ...): cv = TriggeredView.initialize(handle) ... # write this program's part of chunk c cv.mark_ready(c) # whole CTA, after its writes @triton.jit def consumer(out, handle, ...): cv = TriggeredView.initialize(handle) cv.wait(c) # chunk c's data is now visible x = tl.load(out + ...)
- Parameters:
view (tensor)
gates (tensor)
round (tensor)
rank (tensor)
num_chunks (tensor)
- mark_ready(chunk)[source]#
Declare this CTA’s writes to chunk
chunkdone; the chunk is sent once all of its writers have.Call from the whole CTA, outside divergent branches, after the CTA’s writes to the chunk.
- arrived(chunk)[source]#
True once chunk
chunkis visible here this round: for this rank’s own chunks, once marked ready; for another rank’s chunk in a group, once the whole group has arrived here. Always true of chunks not sent to this rank.Call uniformly across the CTA: the result reaches the other waves through a CTA barrier, so every wave must make the same number of calls (true of the other waits too).
- wait(chunk)[source]#
Spin until
arrived(chunk). Each wave first polls with its own loads, which do not serialize with the copy engines’ atomics on the word; the acquire check that follows is CTA-uniform.
- group_arrived(group)[source]#
True once every chunk of
groupsent to this rank has arrived this round. Always true of a group with no chunks sent here, including a group no chunk uses (a debug build asserts on that, since it is likely a typo).
- wait_group(group)[source]#
Spin until
group_arrived(group), polling each wave’s own loads first (seewait).
TriggeredView (Gluon device)#
- class TriggeredView(view, gates, round, rank, num_chunks)[source]#
Bases:
objectGluon device handle for one round of a triggered plan.
Usage:
@gluon.jit def producer(out, handle, ...): cv = TriggeredView.initialize(handle) ... # write this program's part of chunk c cv.mark_ready(c) # whole CTA, after its writes @gluon.jit def consumer(out, handle, ...): cv = TriggeredView.initialize(handle) cv.wait(c) # chunk c's data is now visible x = gl.load(out + ...)
- Parameters:
view (tensor)
gates (tensor)
round (tensor)
rank (tensor)
num_chunks (tensor)
- mark_ready(chunk)[source]#
Declare this CTA’s writes to chunk
chunkdone; the chunk is sent once all of its writers have.Call from the whole CTA, after the CTA’s writes to the chunk: outside divergent branches and outside
gl.warp_specializepartitions.
- arrived(chunk)[source]#
True once chunk
chunkis visible here this round: for this rank’s own chunks, once marked ready; for another rank’s chunk in a group, once the whole group has arrived here. Always true of chunks not sent to this rank.Call uniformly across the CTA: the result reaches the other waves through a CTA barrier, so every wave must make the same number of calls (true of the other waits too).
- wait(chunk)[source]#
Spin until
arrived(chunk). Each wave first polls with its own loads, which do not serialize with the copy engines’ atomics on the word; the acquire check that follows is CTA-uniform.
- group_arrived(group)[source]#
True once every chunk of
groupsent to this rank has arrived this round. Always true of a group with no chunks sent here, including a group no chunk uses (a debug build asserts on that, since it is likely a typo).
- wait_group(group)[source]#
Spin until
group_arrived(group), polling each wave’s own loads first (seewait).