aorta

Aorta Probe Agent — Closed-Loop Mitigation Search

Author: AORTA platform · June 2026

Agenda

Problem — static probe matrices and manual correlation
Design — one agent loop on top of aorta probe
Safety boundaries
Reuse — shared engine and resume-as-memo
Relationship to cluster-scale agent systems
Build phases
Implementation plan (engineering appendix)

Problem — Static Matrices, Manual Correlation

aorta probe runs an opaque user launch command across a pre-written mitigation_axis × diagnostic_axis cartesian product. A human must:

Guess which mitigations belong in the matrix and in what order.
Read per-cell failure_detectors_fired and capture fields by hand.
Name the failure category and decide when to stop.
Package artifacts for handoff.

Most time is spent in signal correlation, not in running the repro. The agent compresses that loop while leaving execution and verdicts to the existing deterministic probe machinery.

Design — One Agent Loop on Top of Probe

Unlike cluster-scale three-agent systems (preflight / watchdog / autopsy at the scheduler layer), the Probe Agent is a standalone application-layer loop: it wraps the same run_recipe engine aorta probe uses, but replaces the static matrix with a hypothesis-driven search.

flowchart TD
    seed["Seed: opaque repro argv + ticket + optional symptom"] --> baseline["Run baseline cell none-none via run_recipe"]
    baseline --> classify["Deterministic 5-tier classifier -> verdict + detectors + capture"]
    classify --> decide{"agent step (LLM)"}
    decide -->|"label + propose next registered mitigation"| run["Grow mitigation_axis -> run_recipe (flat_resume skips done cells)"]
    run --> classify
    decide -->|"a mitigation passed"| converged["Emit autopsy report + winning mitigation"]
    decide -->|"budget exhausted / no hypotheses"| giveup["Emit report: best-effort category + unknown"]
    converged --> bundle["aorta bundle (redacted handoff)"]
    giveup --> bundle

Agent step output (structured JSON)

Field	Meaning
`category`	One of the eight probe labels — `PROBE_CATEGORIES`, see below. Not the full eleven-label shared taxonomy: `validate_step` refuses the three evidence-only names here
`hypothesis`	Short natural-language explanation
`next_mitigations`	Registered mitigation names to try next (never raw argv)
`confidence`	0.0–1.0 self-reported confidence
`stop`	True when the agent believes search should end

Autopsy category taxonomy

The vocabulary has eleven names and is shared with aorta.cia, so a report means the same thing whichever front door wrote it. A probe step may use only the eight below; the three after them are reachable only by reading an instrument.

Probe category (`PROBE_CATEGORIES`)	Typical probe signals
`rccl_hang`	`tier2:*` hang detectors, RCCL timeout patterns
`thermal_throttle`	Sustained perf drop + thermal context (when available)
`illegal_mem`	`tier4:hip_error`, illegal-access regex in stderr
`oom_fragment`	OOM / exit 137 patterns
`checkpoint_race`	Checkpoint save/load boundary signatures
`launch_error`	Early exit, launch failures
`perf_regression`	Pass with warn detectors or confound regression
`unknown`	No confident mapping

Three more are evidence-only — reachable by aorta.cia reading an instrument, never by a probe step:

Category	Established by
`gpu_race`	ConSan / waitcheck findings naming sites inside one kernel
`numeric_silent`	`tier4:nan_signature`, Inf/overflow, out-of-tolerance drift
`tooling_gap`	The instrument could not run, or produced no records

The set is closed: AgentPolicy.validate_step raises PolicyViolation on anything outside PROBE_CATEGORIES, so the loop stops rather than recording a label nothing downstream can route on. The vocabulary is defined once, in aorta.agent.llm.AUTOPSY_CATEGORY_GUIDANCE, as name → one-line gloss; AUTOPSY_CATEGORIES is derived from that mapping, PROBE_CATEGORIES is that set less the evidence-only three, and the proposer prompt renders the glosses of the probe subset — so a new label cannot reach the validator without also reaching the model, and the model is never shown a label it has no way to reach.

Three distinctions the glosses exist to enforce, because the names alone do not:

checkpoint_race is about checkpoint I/O, not about kernels. An intra-wave LDS race is gpu_race. The two were previously conflated — checkpoint_race was the nearest available name for a kernel race, and it was the wrong one.
tooling_gap is not unknown. unknown is evidence that fits no label; tooling_gap is evidence never collected, because the sanitizer was rejected or produced no records. Reading the second as the first turns “we did not look” into “we looked and found nothing”.
unknown is a real answer, not a failure to answer. It is the correct label when the evidence supports none of the others, and validate_step accepts it. Guessing a specific label to avoid unknown is worse than unknown, because the loop routes on the label.

Declaring a detector’s category

--llm-backend fake labels a failure without a model. It reads what the failure detectors that fired declare first — custom_patterns[*].category in the recipe. Only when the declarations settle nothing (none was made, or every one was unknown) does it infer a category from the IDs of the undeclared detectors, and then from --symptom. A warn or info pattern may declare a category too, but it is not a failure detector, so this proposer never reads it.

Declare a category whenever an ID could mislead: custom: IDs are free-form, so custom:consan_host_data_race names a sanitizer and a race and reads as gpu_race, although the race it reports is on the host.

custom_patterns:
  - id: consan_host_data_race
    match:
      regex: "host data race"
    category: unknown   # fits no label; read off its name it would be gpu_race

unknown is a declaration too, and it abstains: it stops the name being read without contradicting another detector’s label. Undeclared detectors beside it are still read off their IDs, and --symptom is still heard.
Fired detectors that declare two different labels give unknown; neither is ranked above the other, and --symptom does not break the tie.
Any of the eleven names may be declared, because a declaration describes evidence. A probe step still reports an evidence-only label as unknown.
Built-in tier1–tier4 detectors declare nothing and are still read off their IDs. Only the offline proposer reads declarations: a real backend takes its category from the model, and no prompt carries them.

Safety Boundaries (Non-Negotiable)

Verdict source of truth — Only the deterministic 5-tier classifier in aorta.probe.classifier sets pass / fail. The LLM never overrides it.
Registered mitigations only — Proposals resolve through aorta.registry.get_mitigation. Raw shell or argv changes are rejected.
Bounded autonomy — max_iterations and max_walltime_sec caps; optional approval gate before running mitigations flagged as needing ack.
Optional LLM dependency — LiteLLM lives behind the [agent] extra; a deterministic fake proposer supports offline tests with zero API calls.

Reuse — Shared Engine, Resume as Memo

Component	Role in agent loop
`aorta.triage.runner.run_recipe`	Same entry point as `aorta probe`
`layout=flat_resume`, `resume_existing=True`	Skips completed cells — search tree memo
`aorta.probe.classifier`	Per-trial verdict + detectors
`aorta.instrumentation.environment`	`host_env.json` per ticket
`aorta bundle`	Redacted handoff at loop end

Each iteration grows mitigation_axis on the probe recipe; cells already present under <output>/<ticket>/ are not re-executed.

Relationship to Cluster-Scale Agent Systems

Cluster intelligence proposals (preflight / watchdog / autopsy at the scheduler + omnistat + TraceLens layer) target fleet-wide failure modes. The Probe Agent targets single-repro mitigation search at the application layer.

Layer	Scope	This agent
Cluster autopsy	Post-mortem across nodes, traces, Prometheus	Future interop: cluster agent could invoke `aorta agent mitigate` with a frozen argv
Probe agent	Mitigation search on one opaque command	Shipped here

Standalone today; integration is a one-slide handoff, not a hard dependency.

Build Phases

Phase	Deliverable
A	`aorta agent mitigate` CLI + fake-LLM loop + `agent_log.jsonl` + tests (no API)
B	LiteLLM client + structured output + `agent_report.md`
C	Approval gates + `wake()` resume + optional `aorta bundle` at end

Implementation Plan (Engineering Appendix)

Package layout

src/aorta/agent/
  __init__.py
  loop.py      # orchestration -> run_recipe
  llm.py       # AgentStep, FakeLLMProposer, LiteLLMProposer
  policy.py    # budget, registry filter, approval
  state.py     # agent_log.jsonl, wake()
  report.py    # agent_report.md writer
src/aorta/cli/agent.py
tests/agent/
docs/agent/aorta-probe-agent.md   # this file

CLI

aorta agent mitigate \
  --output ./agent_results \
  --ticket ROCM-EXAMPLE \
  --max-iterations 8 \
  --symptom "illegal memory access after long run" \
  -- \
  python3 my_repro.py --steps 100

Install LLM support: pip install 'amd-aorta[agent]' (pulls litellm). Default backend is fake (deterministic, offline-safe).

State file

<output>/<ticket>/agent_log.jsonl — append-only JSON lines for resume and audit. wake() replays tried mitigations and last category.

CLI outcomes

Outcome	Meaning
`baseline_pass`	`none-none` passed; repro OK without mitigations
`converged`	A cell with one non-baseline name passed: `{mitigation}-none`, or `none-{diagnostic}` when the diagnostic is a built-in that changes behaviour. A logging knob such as `amd_log_level_4` on the diagnostic axis is not a fix, so its pass does not converge; neither does a pass under a knob whose variable nothing reads, such as `tf32_off` (`INERT_MITIGATIONS`), or under a plugin or sidecar diagnostic, which the registry cannot classify
`exhausted_candidates`	No more registered mitigations left to try
`agent_stop`	Proposer ended search for another reason
`proposal_unresolved`	Every mitigation the proposer named was dropped by the candidate filter (unregistered, already tried, outside the allowlist, or the `none` baseline, which is never a candidate), and the proposer did not ask to stop. An agent-side name-resolution failure, not a decision by the model — the names are in `unresolved_mitigations` in `agent_log.jsonl`
`proposal_redundant`	Every mitigation the proposer named was removed by validation as redundant — only the `none` baseline can empty a proposal this way — and the proposer did not ask to stop. Agent-side normalisation, not a decision by the model — the names are in `redundant_mitigations` in `agent_log.jsonl`

Tests

Mock run_recipe; assert growing mitigation_axis and flat_resume.
Policy rejects unregistered mitigation names.
Fake-LLM loop converges when a mocked cell returns pass.
wake() skips already-tried mitigations.

agentic-testing-guide.md — usage, examples, LLM vs fake, under the hood
docs/probe-188/usage.md for probe-mode recipes
src/aorta/agent/ for the implementation.

This site is open source. Improve this page.