aorta

Aorta Probe Agent — Closed-Loop Mitigation Search

Author: AORTA platform · June 2026

Agenda

Problem — static probe matrices and manual correlation
Design — one agent loop on top of aorta probe
Safety boundaries
Reuse — shared engine and resume-as-memo
Relationship to cluster-scale agent systems
Build phases
Implementation plan (engineering appendix)

Problem — Static Matrices, Manual Correlation

aorta probe runs an opaque user launch command across a pre-written mitigation_axis × diagnostic_axis cartesian product. A human must:

Guess which mitigations belong in the matrix and in what order.
Read per-cell failure_detectors_fired and capture fields by hand.
Name the failure category and decide when to stop.
Package artifacts for handoff.

Most time is spent in signal correlation, not in running the repro. The agent compresses that loop while leaving execution and verdicts to the existing deterministic probe machinery.

Design — One Agent Loop on Top of Probe

Unlike cluster-scale three-agent systems (preflight / watchdog / autopsy at the scheduler layer), the Probe Agent is a standalone application-layer loop: it wraps the same run_recipe engine aorta probe uses, but replaces the static matrix with a hypothesis-driven search.

flowchart TD
    seed["Seed: opaque repro argv + ticket + optional symptom"] --> baseline["Run baseline cell none-none via run_recipe"]
    baseline --> classify["Deterministic 5-tier classifier -> verdict + detectors + capture"]
    classify --> decide{"agent step (LLM)"}
    decide -->|"label + propose next registered mitigation"| run["Grow mitigation_axis -> run_recipe (flat_resume skips done cells)"]
    run --> classify
    decide -->|"a mitigation passed"| converged["Emit autopsy report + winning mitigation"]
    decide -->|"budget exhausted / no hypotheses"| giveup["Emit report: best-effort category + unknown"]
    converged --> bundle["aorta bundle (redacted handoff)"]
    giveup --> bundle

Agent step output (structured JSON)

Field	Meaning
`category`	One of the eight probe labels — `PROBE_CATEGORIES`, see below. Not the full eleven-label shared taxonomy: `validate_step` refuses the three evidence-only names here
`hypothesis`	Short natural-language explanation
`next_mitigations`	Registered mitigation names to try next (never raw argv)
`confidence`	0.0–1.0 self-reported confidence
`stop`	True when the agent believes search should end

Autopsy category taxonomy

The vocabulary has eleven names and is shared with aorta.cia, so a report means the same thing whichever front door wrote it. A probe step may use only the eight below; the three after them are reachable only by reading an instrument.

Probe category (`PROBE_CATEGORIES`)	Typical probe signals
`rccl_hang`	`tier2:*` hang detectors, RCCL timeout patterns
`thermal_throttle`	Sustained perf drop + thermal context (when available)
`illegal_mem`	`tier4:hip_error`, illegal-access regex in stderr
`oom_fragment`	OOM / exit 137 patterns
`checkpoint_race`	Checkpoint save/load boundary signatures
`launch_error`	Early exit, launch failures
`perf_regression`	Pass with warn detectors or confound regression
`unknown`	No confident mapping

Three more are evidence-only — reachable by aorta.cia reading an instrument, never by a probe step:

Category	Established by
`gpu_race`	ConSan / waitcheck findings naming sites inside one kernel
`numeric_silent`	`tier4:nan_signature`, Inf/overflow, out-of-tolerance drift
`tooling_gap`	The instrument could not run, or produced no records

The set is closed: AgentPolicy.validate_step raises PolicyViolation on anything outside PROBE_CATEGORIES, so the loop stops rather than recording a label nothing downstream can route on. The vocabulary is defined once, in aorta.agent.llm.AUTOPSY_CATEGORY_GUIDANCE, as name → one-line gloss; AUTOPSY_CATEGORIES is derived from that mapping, PROBE_CATEGORIES is that set less the evidence-only three, and the proposer prompt renders the glosses of the probe subset — so a new label cannot reach the validator without also reaching the model, and the model is never shown a label it has no way to reach.

Three distinctions the glosses exist to enforce, because the names alone do not:

checkpoint_race is about checkpoint I/O, not about kernels. An intra-wave LDS race is gpu_race. The two were previously conflated — checkpoint_race was the nearest available name for a kernel race, and it was the wrong one.
tooling_gap is not unknown. unknown is evidence that fits no label; tooling_gap is evidence never collected, because the sanitizer was rejected or produced no records. Reading the second as the first turns “we did not look” into “we looked and found nothing”.
unknown is a real answer, not a failure to answer. It is the correct label when the evidence supports none of the others, and validate_step accepts it. Guessing a specific label to avoid unknown is worse than unknown, because the loop routes on the label.

Safety Boundaries (Non-Negotiable)

Verdict source of truth — Only the deterministic 5-tier classifier in aorta.probe.classifier sets pass / fail. The LLM never overrides it.
Registered mitigations only — Proposals resolve through aorta.registry.get_mitigation. Raw shell or argv changes are rejected.
Bounded autonomy — max_iterations and max_walltime_sec caps; optional approval gate before running mitigations flagged as needing ack.
Optional LLM dependency — LiteLLM lives behind the [agent] extra; a deterministic fake proposer supports offline tests with zero API calls.

Reuse — Shared Engine, Resume as Memo

Component	Role in agent loop
`aorta.triage.runner.run_recipe`	Same entry point as `aorta probe`
`layout=flat_resume`, `resume_existing=True`	Skips completed cells — search tree memo
`aorta.probe.classifier`	Per-trial verdict + detectors
`aorta.instrumentation.environment`	`host_env.json` per ticket
`aorta bundle`	Redacted handoff at loop end

Each iteration grows mitigation_axis on the probe recipe; cells already present under <output>/<ticket>/ are not re-executed.

Relationship to Cluster-Scale Agent Systems

Cluster intelligence proposals (preflight / watchdog / autopsy at the scheduler + omnistat + TraceLens layer) target fleet-wide failure modes. The Probe Agent targets single-repro mitigation search at the application layer.

Layer	Scope	This agent
Cluster autopsy	Post-mortem across nodes, traces, Prometheus	Future interop: cluster agent could invoke `aorta agent mitigate` with a frozen argv
Probe agent	Mitigation search on one opaque command	Shipped here

Standalone today; integration is a one-slide handoff, not a hard dependency.

Build Phases

Phase	Deliverable
A	`aorta agent mitigate` CLI + fake-LLM loop + `agent_log.jsonl` + tests (no API)
B	LiteLLM client + structured output + `agent_report.md`
C	Approval gates + `wake()` resume + optional `aorta bundle` at end

Implementation Plan (Engineering Appendix)

Package layout

src/aorta/agent/
  __init__.py
  loop.py      # orchestration -> run_recipe
  llm.py       # AgentStep, FakeLLMProposer, LiteLLMProposer
  policy.py    # budget, registry filter, approval
  state.py     # agent_log.jsonl, wake()
  report.py    # agent_report.md writer
src/aorta/cli/agent.py
tests/agent/
docs/agent/aorta-probe-agent.md   # this file

CLI

aorta agent mitigate \
  --output ./agent_results \
  --ticket ROCM-EXAMPLE \
  --max-iterations 8 \
  --symptom "illegal memory access after long run" \
  -- \
  python3 my_repro.py --steps 100

Install LLM support: pip install 'amd-aorta[agent]' (pulls litellm). Default backend is fake (deterministic, offline-safe).

State file

<output>/<ticket>/agent_log.jsonl — append-only JSON lines for resume and audit. wake() replays tried mitigations and last category.

CLI outcomes

Outcome	Meaning
`baseline_pass`	`none-none` passed; repro OK without mitigations
`converged`	A non-baseline mitigation cell passed
`exhausted_candidates`	No more registered mitigations left to try
`agent_stop`	Proposer ended search for another reason
`proposal_unresolved`	Every mitigation the proposer named was dropped by the candidate filter (unregistered, already tried, outside the allowlist, or the `none` baseline, which is never a candidate), and the proposer did not ask to stop. An agent-side name-resolution failure, not a decision by the model — the names are in `unresolved_mitigations` in `agent_log.jsonl`

Tests

Mock run_recipe; assert growing mitigation_axis and flat_resume.
Policy rejects unregistered mitigation names.
Fake-LLM loop converges when a mocked cell returns pass.
wake() skips already-tried mitigations.

agentic-testing-guide.md — usage, examples, LLM vs fake, under the hood
docs/probe-188/usage.md for probe-mode recipes
src/aorta/agent/ for the implementation.

This site is open source. Improve this page.