Multi-agent incident triage with OpenClaw#

Author: Kalyan Archakam
Knowledge level: Intermediate
Publication date: August 20, 2026

This tutorial shows how to build a multi-agent incident triage system that runs on a single AMD Instinct™ MI300X GPU using OpenClaw and vLLM. You will serve a local Qwen3.6-35B-A3B model, configure a graph with three specialist agents, and run them against real observability data from a microservices fault injection study.

From prompt engineering to graph engineering#

Over time, the way practitioners structure AI workloads has evolved through several stages:

Prompt engineering: A single model call, shaped by carefully crafted input text. The model is a black box, and the lever is your input.

Context engineering: The model call is the same as prompt engineering, but the surrounding context is curated: retrieval-augmented generation (RAG), system prompts, tool descriptions, few-shot examples, etc. Quality of the context window determines quality of the output.

Loop engineering: A single agent runs a plan → act → observe cycle, retrying until a termination condition is met. This is the default pattern for most AI coding agents today.

Graph engineering: The loop is replaced by an explicit graph. Multiple agents run in parallel on different sub-problems, with their outputs converging to a coordinator that makes a decision. The structure between nodes (for example, which agent sees which data, and in what order) becomes as important as the agents themselves.

The shift from loop to graph engineering is not about adding complexity. It is about matching the approach with the structure of the problem. Incident triage is naturally parallel: logs, metrics, and traces are independent signals that can be read simultaneously. A loop processes them sequentially and asks one agent to hold all three in context at once. On the other hand, a graph assigns each signal to a specialist and uses a coordinator to correlate the results, which is similar to how an experienced site reliability engineering (SRE) team usually works.

What you’ll build#

This figure illustrates the graph engineering approach taken by this tutorial for multi-agent triaging.

Fan-out/fan-in multi-agent triage graph: a coordinator spawns logs, metrics and traces specialists that each write findings to their own workspace, then correlates them into a root-cause report

How the graph maps onto OpenClaw:

Graph element

OpenClaw primitive

Configured in

Nodes

openclaw agents, one workspace per specialist

Part 5.1

Typed edges

Each agent’s AGENTS.md briefing

Part 5.2

Fan-out

sessions_spawn × 3, before the first yield

Part 5.4

Fan-in

sessions_yield + per-agent findings.md

Part 5.4

Adjacency list

agents.defaults.subagents.allowAgents

Part 5.3

Bounded cycle

One-retry rule in the coordinator protocol

Part 5.4

Here is a summary of the steps this tutorial will guide you through:

  • Make sure all prerequisites for this project are satisfied (Prerequisites).

  • Get vLLM and OpenClaw ready (Part 1 to Part 3).

  • Fetch incident data to be triaged, and curate the inputs for the different agents (Part 4).

  • Configure the fan-out/fan-in multi-agent triage graph (Part 5).

  • Run the triage graph (Part 6).

  • Clean up after the exercise (Part 7).

Prerequisites#

This tutorial was developed and tested using the following setup.

Operating system#

Ubuntu 22.04 or 24.04.

Hardware#

AMD Instinct GPUs: This tutorial was tested on a single AMD Instinct MI300X GPU (192 GB HBM3). Ensure you are using an AMD Instinct GPU or compatible hardware with ROCm support and that your system meets the official requirements.

You will also need approximately 70 GB of free disk space for the model weights.

Software#

ROCm 7.0 or later: Install and verify ROCm by following the ROCm install guide. This tutorial was tested on ROCm 7.2.3. After installation, confirm your setup using:

amd-smi

This command lists your AMD GPUs with relevant details.

Docker: Ensure Docker is installed and configured correctly. Follow the Docker installation guide for your operating system.

Note: Ensure Docker permissions are correctly configured. To allow non-root access, run:

sudo usermod -aG docker $USER
newgrp docker

Verify Docker is working correctly:

docker run hello-world

Launch the vLLM container and JupyterLab: You run JupyterLab inside the vLLM container. The model server starts from a notebook cell in Part 1, and OpenClaw is installed in the same container in Part 2. The notebook, the OpenClaw gateway, and the vLLM server all run on localhost, so they can communicate over 127.0.0.1.

In the droplet terminal, start the container. The -v $(pwd)/hf-cache:/root/.cache/huggingface flag mounts a local directory over the container’s Hugging Face cache so the downloaded weights persist across container restarts:

docker run -it --rm \
    --ipc=host \
    --network=host \
    --device=/dev/kfd \
    --device=/dev/dri \
    --group-add video \
    -v $(pwd)/hf-cache:/root/.cache/huggingface \
    -v $(pwd):/workspace/notebooks \
    -w /workspace/notebooks \
    --entrypoint /bin/bash \
    vllm/vllm-openai-rocm:v0.31.0

vllm/vllm-openai-rocm is the official vLLM image built for ROCm and includes support for the Qwen3.6 architecture (Qwen3_5MoeForCausalLM). The rocm/vllm image used in some other tutorials predates this architecture and does not support it.

Inside the container, install JupyterLab and start it:

pip install jupyterlab
jupyter-lab --ip=0.0.0.0 --port=8888 --no-browser --allow-root

Note: Ensure port 8888 is not already in use. If it is, replace --port=8888 with another port, for example --port=8890.

After running the jupyter-lab command, click the link in the terminal to open JupyterLab. The link has the form http://127.0.0.1:<PORT>/lab?token=<TOKEN_VALUE>.

Note: Make sure this notebook is in the directory you started the container from, or upload it after Jupyter starts. You can download this notebook from the AI Developer Hub GitHub repository.

Run the remaining steps in this tutorial from inside this notebook (or a JupyterLab terminal) after the Jupyter server is up.

Note: The container runs with --rm, so OpenClaw and its configuration (~/.openclaw) are gone when it stops. After restarting the container, re-run Parts 2 to 5 to install, connect and configure OpenClaw again.

Part 1: Start the vLLM server#

The cells below serve the Qwen3.6-35B-A3B model with vllm serve in the background, logging to vllm_server.log, and wait until the server answers. The first start downloads the model, so it can take several minutes. These helpers start a server, wait until it answers (or stop with its log if it fails), and stop it again:

The server runs with a 128K context (--max-model-len 131072): OpenClaw 2026.10.4 assumes a 128K context window for this model, and with all of its agent tools the coordinator’s prompts do not fit in 32K tokens. The weights take about 70 GB; vLLM uses the rest of the MI300X’s 192 GB for the KV cache, which leaves room for 128K-token requests.

Note: Replace abc-123 below with a secure, unique API key.

Hide code cell source

import os
import signal
import subprocess
import time
import urllib.request


def server_is_up(url, headers=None):
    try:
        with urllib.request.urlopen(urllib.request.Request(url, headers=headers or {}),
                                    timeout=5):
            return True
    except Exception:
        return False


def ensure_server(command, url, log_path, timeout=1200, headers=None):
    """Start `command` in the background (its output goes to log_path) unless
    url already answers, then wait until it does. Returns the server process,
    or None if a server was already running. Raises, with the end of the log,
    if the server exits or is not ready within timeout seconds."""
    if server_is_up(url, headers):
        print(f"Already running: {url}")
        return None
    log = open(log_path, "w")
    server = subprocess.Popen(command, shell=True, stdout=log, stderr=subprocess.STDOUT,
                              start_new_session=True)
    deadline = time.time() + timeout
    while time.time() < deadline and server.poll() is None:
        if server_is_up(url, headers):
            print(f"Ready after {timeout - int(deadline - time.time())}s: {url}")
            return server
        time.sleep(5)
    tail = "".join(open(log_path).readlines()[-40:])
    state = f"exited with code {server.returncode}" if server.poll() is not None else "timed out"
    raise RuntimeError(f"Server {state} before {url} answered. Last log lines:\n{tail}")


def stop_server(server):
    """Stop a server started by ensure_server, and everything it started."""
    if server is not None and server.poll() is None:
        os.killpg(server.pid, signal.SIGTERM)
        try:
            server.wait(timeout=60)
        except subprocess.TimeoutExpired:
            os.killpg(server.pid, signal.SIGKILL)

Start the vLLM server in the background with these helpers, reusing one that is already running:

vllm_server = ensure_server(
    "vllm serve Qwen/Qwen3.6-35B-A3B"
    " --served-model-name Qwen3.6-35B-A3B"
    " --host 0.0.0.0 --port 8000"
    " --api-key abc-123"
    " --max-model-len 131072"  # OpenClaw assumes a 128K context for this model
    " --reasoning-parser qwen3"
    " --enable-auto-tool-choice"
    " --tool-call-parser qwen3_xml"
    " --trust-remote-code",
    "http://localhost:8000/v1/models", "vllm_server.log", timeout=3600,
    headers={"Authorization": "Bearer abc-123"},
)

List the models the server is serving:

import urllib.request, json

req = urllib.request.Request("http://localhost:8000/v1/models",
                             headers={"Authorization": "Bearer abc-123"})
with urllib.request.urlopen(req, timeout=10) as r:
    models = json.loads(r.read())
for m in models.get("data", []):
    print(f"Model: {m['id']}")

Troubleshooting

If the server does not become ready, the launch cell stops with the end of vllm_server.log. Check these common issues:

  • A server is already running: If the launch cell reports Already running, a server from a previous run still answers on port 8000. Stop that server from a terminal (pkill -f "vllm serve") or restart the container, then re-run the launch cell.

  • Out of memory (OOM): The model’s weights need roughly 70 GB of GPU memory, and vLLM reserves most of the rest for the 128K-context KV cache. Check free memory with this command, and free up some memory if needed by closing other GPU-intensive applications:

    amd-smi monitor
    
  • Startup errors: Inspect the full server log:

    tail -n 100 vllm_server.log
    

Part 2: Ensure OpenClaw is installed#

OpenClaw runs your agents as a local service called the gateway. The cell below installs the openclaw CLI, version 2026.10.4 (the release this tutorial was tested with), with the official installer unless that version is already present. Onboarding happens in Part 3, so the installer skips its own setup prompts:

%%bash
set -e
# This tutorial was tested with OpenClaw 2026.10.4; other releases may behave differently.
OPENCLAW_VERSION=2026.10.4
if ! openclaw --version 2>/dev/null | grep -qF "$OPENCLAW_VERSION"; then
    curl -fsSL --proto '=https' --tlsv1.2 https://openclaw.ai/install.sh \
        | bash -s -- --no-onboard --no-prompt --version "$OPENCLAW_VERSION"
fi
openclaw --version

When it prints a version, OpenClaw is ready. Continue to Part 3.

Part 3: Connect OpenClaw to the model and start the gateway#

This part connects OpenClaw to the running vLLM server and starts the gateway process.

3.1 Connect OpenClaw to the vLLM endpoint#

Onboard the local vLLM endpoint by registering it with OpenClaw and associating it with a model. The model id and the api-key must match the values specified for --served-model-name and --api-key, respectively, from Part 1.

%%bash
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url "http://localhost:8000/v1" \
  --custom-model-id "Qwen3.6-35B-A3B" --custom-provider-id "vllm" \
  --custom-compatibility openai --custom-api-key "abc-123" \
  --secret-input-mode plaintext \
  --skip-channels --skip-search --skip-hooks --skip-ui --skip-health --accept-risk

3.2 Keep the full tool surface (lean mode and tool search off)#

Leave lean mode off for this graph. Lean mode shrinks the tool surface and moves tools such as sessions_spawn and sessions_yield behind a tool_search step. A small local model might fail to discover the tools then, preventing the coordinator from fanning out and resulting in an error such as sessions_spawn tool is not available. Keeping the full surface exposed allows main to call the spawn and yield tools directly.

For local models, OpenClaw also hides optional tools behind Tool Search by default, even with lean mode off, so turn that off too (tools.toolSearch: false).

%%bash
openclaw config set agents.defaults.experimental.localModelLean false
openclaw config set tools.toolSearch false --strict-json
echo "lean mode and tool search off (sessions_spawn/yield stay directly available)"

3.3 Start the gateway in the background#

This container has no systemd, so run the gateway directly in the background, and wait until it listens on its port (18789).

%%bash
pkill -f openclaw-gateway 2>/dev/null || true
sleep 1
# setsid: its own session, so the gateway keeps running after this cell ends
setsid nohup openclaw gateway run > ~/gateway.log 2>&1 < /dev/null &
for i in $(seq 60); do
    curl -s -o /dev/null http://127.0.0.1:18789/ && break
    sleep 2
done
curl -s -o /dev/null http://127.0.0.1:18789/ || { tail -n 30 ~/gateway.log; exit 1; }
tail -n 8 ~/gateway.log

3.4 Test the model#

Perform a smoke test by running this command. It goes through the gateway started in 3.3 (with a gateway running, OpenClaw 2026.10.4’s --local mode, which runs the agent in this process instead, conflicts with it):

!openclaw agent --session-id smoke --message "Reply with exactly: pong" --thinking off

It should reply pong.

Part 4: Fetch the incident data#

This tutorial uses one fault window from the Nezha observability dataset: real logs, metrics, and traces captured from the OnlineBoutique microservices application during a controlled fault injection experiment. A fault was injected into exactly one service. All three signals carry a trace of it, but no single signal is sufficient to identify the cause.

The raw per-minute CSV files are 4–15 MB each. The cells below first configure the fault window and a download helper, then curate each signal (metrics, traces, and logs) into compact, model-readable files in /workspace/incident/.

4.1 Configure the fault window and download helper#

First, pin the incident to a specific date and a specific three-minute fault window in the Nezha dataset, list the ten OnlineBoutique pods, and define a small fetch() helper for pulling a raw CSV file from GitHub. This cell only sets things up and doesn’t download or write any data.

import os, io, csv, json, urllib.request

NEZHA_DATE   = "2022-08-22"
FAULT_WINDOW = ["04:44", "04:45", "04:46"]   # fault injected at 04:44:16
BASELINE_MIN = "04_36"
FAULT_MIN    = "04_45"
INCIDENT_DIR = "/workspace/incident"
RAW = f"https://raw.githubusercontent.com/IntelligentDDS/Nezha/main/rca_data/{NEZHA_DATE}/"

PODS = [
    "adservice-5f6585d649-fnmft",        "cartservice-579f59597d-wc2lz",
    "checkoutservice-578fcf4766-9csqn",  "currencyservice-cf787dd48-vpjrd",
    "emailservice-55fdc5b988-f6xth",     "frontend-579b9bff58-t2dbm",
    "paymentservice-9cdb6588f-554sm",    "productcatalogservice-668d5f85fb-wckp8",
    "recommendationservice-6cfdd55578-gfj6q", "shippingservice-7b598fc7d-lmggd",
]
svc = lambda pod: pod.split("-")[0]
os.makedirs(INCIDENT_DIR, exist_ok=True)

def fetch(path):
    req = urllib.request.Request(RAW + path, headers={"User-Agent": "notebook"})
    return urllib.request.urlopen(req, timeout=180).read().decode("utf-8", "replace")

4.2 Curate the metrics signal#

Download each service’s per-minute metric CSV file, keeping only the rows inside the fault window, and extracting the CPU usage, server-latency P95, and workload. The curated result is written to /workspace/incident/metrics.csv and the cell prints the file’s size when it finishes.

# Metrics: CPU usage, server-latency 95th-percentile (P95), and workload for each service across the fault window.
rows = []
for pod in PODS:
    for r in csv.DictReader(io.StringIO(fetch(f"metric/{pod}_metric.csv"))):
        if r["Time"][11:16] in FAULT_WINDOW:
            rows.append([svc(pod), r["Time"][11:19],
                         round(float(r["CpuUsageRate(%)"]), 1),
                         round(float(r["PodServerLatencyP95(s)"]), 3),
                         round(float(r["PodWorkload(Ops)"]), 1)])
with open(f"{INCIDENT_DIR}/metrics.csv", "w") as f:
    f.write("service,time,cpu_usage_rate_pct,server_latency_p95_s,workload_ops\n")
    f.write("\n".join(",".join(map(str, r)) for r in rows) + "\n")
print(f"metrics.csv  {os.path.getsize(f'{INCIDENT_DIR}/metrics.csv')} bytes")

4.3 Curate the traces signal#

Download the trace CSV files for the fault minute and a baseline minute captured eight minutes earlier, compute the span-latency P95 per service, and write the per-service slowdown to /workspace/incident/traces.csv.

# Traces: span-latency P95 per service: fault minute vs a baseline minute.
def p95(vals): return sorted(vals)[int(len(vals) * 0.95)] if vals else 0

def p95_by_svc(minute):
    d = {}
    for r in csv.DictReader(io.StringIO(fetch(f"trace/{minute}_trace.csv"))):
        d.setdefault(svc(r["PodName"]), []).append(int(r["Duration"]))
    return {s: p95(v) for s, v in d.items()}

base, fault = p95_by_svc(BASELINE_MIN), p95_by_svc(FAULT_MIN)
with open(f"{INCIDENT_DIR}/traces.csv", "w") as f:
    f.write("service,baseline_p95_ms,fault_p95_ms,slowdown_x\n")
    for s in sorted(set(base) | set(fault)):
        b, x = base.get(s, 0) / 1e6, fault.get(s, 0) / 1e6
        f.write(f"{s},{round(b,2)},{round(x,2)},{round(x/b,1) if b else 0}\n")
print(f"traces.csv   {os.path.getsize(f'{INCIDENT_DIR}/traces.csv')} bytes")

4.4 Curate the logs signal#

Finally, download the fault-minute log CSV file and summarize it per service, including the line count, the error and warning count, and a sample message. The summary is written to /workspace/incident/logs.txt.

# Logs: per-service line count, error/warning count, and one sample message.
counts, errors, sample = {}, {}, {}
for r in csv.DictReader(io.StringIO(fetch(f"log/{FAULT_MIN}_log.csv"))):
    s = svc(r["PodName"]); counts[s] = counts.get(s, 0) + 1
    if any(k in r["Log"].lower() for k in ('"error"','"warn"',"exception","failed","panic")):
        errors[s] = errors.get(s, 0) + 1
    if s not in sample:
        try:    sample[s] = json.loads(json.loads(r["Log"])["log"])["message"][:100]
        except: sample[s] = r["Log"][:100]
with open(f"{INCIDENT_DIR}/logs.txt", "w") as f:
    f.write(f"# OnlineBoutique logs: fault minute {FAULT_MIN.replace('_',':')}\n")
    f.write("# service: lines | errors/warnings | sample\n\n")
    for s in sorted(counts, key=lambda k: -counts[k]):
        f.write(f"{s}: {counts[s]} lines | {errors.get(s,0)} err/warn | {sample.get(s,'')}\n")
print(f"logs.txt     {os.path.getsize(f'{INCIDENT_DIR}/logs.txt')} bytes")

What the data looks like#

After running the cells above, /workspace/incident/ contains three files. Each one is small enough for a language model to read in full, but rich enough to require cross-signal reasoning.

metrics.csv records CPU usage rate (%), server-side latency P95 (seconds), and request workload (ops/s) for all 10 services across the three-minute fault window:

service,time,cpu_usage_rate_pct,server_latency_p95_s,workload_ops
adservice,04:44:19,2.7,0.04,4.9
checkoutservice,04:44:19,7.6,0.242,0.4
checkoutservice,04:45:19,98.8,4.5,0.2
checkoutservice,04:46:19,100.8,4.625,0.3
frontend,04:45:19,25.2,0.935,7.9
...

traces.csv gives span-latency P95 per service for the fault minute compared to a baseline minute captured 8 minutes earlier:

service,baseline_p95_ms,fault_p95_ms,slowdown_x
checkoutservice,0.3,1.8,5.9
frontend,0.22,0.18,0.8
productcatalogservice,0.01,0.01,1.0
...

logs.txt summarizes log activity per service with a line count, an error/warning count, and a sample message:

# OnlineBoutique logs - fault minute 04:45
# service: lines | errors/warnings | sample

productcatalogservice: 9964 lines | 0 err/warn | Query product with name and description successfully
currencyservice: 7383 lines | 0 err/warn | Get currency data successful
checkoutservice: 416 lines | 0 err/warn | PlaceOrder user_id="45bf39a9..." ...
paymentservice: 32 lines | 0 err/warn | Transaction processed: visa ending ...

A few things worth noting before running the graph:

  • productcatalogservice generates the most log lines by a wide margin, but has zero errors.

  • checkoutservice CPU is normal at 04:44 and pegged near 100% from 04:45 onward.

  • The traces table shows a 5.9x slowdown for checkoutservice; every other service is near 1.0x.

  • No service logs any errors or warnings during the fault window.

Here is the ground truth from the data the agents should discover: At 04:44:16 a CPU-contention fault was injected into checkoutservice. It appears as CPU utilization rose to nearly 100%, server-latency P95 increased from ~0.24s to ~4.5s in the metrics data, and trace data shows a ~6x increase in span latency. No error logs were generated, indicating a resource issue rather than an application bug. The high log volume on productcatalogservice is noise, and the service remains healthy. The graph has to correlate all three signals to identify checkoutservice as the root cause of the issue, rather than incorrectly focusing on the noisiest service.

Part 5: Configure the graph#

This part defines the graph structure as OpenClaw configuration: the participating agents, the context available to each agent, the tools exposed to the coordinator, and the control flow governing the collaboration.

5.1 Add the graph nodes#

In a directed graph, a node is a unit of computation with its own state and a defined interface. Each specialist agent is a node: it has an isolated workspace (its state), reads exactly one signal (its input interface), and writes to a fixed findings.md file (its output interface).

Keeping workspaces isolated enforces the typed-edge defined in 5.2 below. A node that can access every signal is no longer a specialist, but a general-purpose agent. The isolation is what gives the graph its structure.

The following cell adds three nodes named logs, metrics, and traces. They are wired together by edges defined in 5.3.

%%bash
for name in logs metrics traces; do
  openclaw agents add "$name" --non-interactive --workspace "/root/.openclaw/workspace/$name"
  rm -f "/root/.openclaw/workspace/$name/BOOTSTRAP.md"
done
openclaw agents list

5.2 Define the typed edges#

In graph terminology, a typed edge carries a label that restricts what flows along it. In this case the label is the agent’s briefing, which specifies that signal the agent reads and what it ignores. Changing the briefing changes the edge type—that rewrites the graph without changing any code.

You define each edge by giving the agent its briefing. The briefing becomes the agent’s AGENTS.md, which persists and loads at the start of every session.

The cell below writes each specialist’s briefing to its AGENTS.md. Each briefing ends with “do not analyze anything now”, because you are configuring the agent, not running it yet.

Tip: You can also give a briefing interactively: open a Jupyter terminal, run openclaw tui, switch to the specialist with /agent <name>, paste the briefing, and ask the agent to save it as its AGENTS.md. The agent writes the file and confirms; type /exit when done.

The AGENTS.md files are loaded when a session starts, so the gateway is restarted in 5.5 before the triage runs.

import os

BRIEFINGS = {
    "logs": """You are the Logs Specialist for an incident-triage graph. Your job:
- When given a task, read ONLY the file /workspace/incident/logs.txt: a per-service log
  summary from the OnlineBoutique microservices system showing line counts, error/warning
  counts, and a sample message per service.
- Report: which services (if any) show error or warning lines, which is noisiest by volume,
  and whether the evidence points to a code bug or a resource issue. High volume alone is
  NOT a fault. Ignore metrics and traces.
- Save your findings to /root/.openclaw/workspace/logs/findings.md and reply concisely.

Do not analyze anything now: wait for a task.""",
    "metrics": """You are the Metrics Specialist for an incident-triage graph. Your job:
- When given a task, read ONLY the file /workspace/incident/metrics.csv: a cross-service
  table of CPU usage (%), server-latency P95 (s), and workload (ops) for a fault window in
  the OnlineBoutique system.
- Report: which service's CPU and/or latency deviates most from the rest, at what time, and
  which services look normal. Name the standout service clearly. Ignore logs and traces.
- Save your findings to /root/.openclaw/workspace/metrics/findings.md and reply concisely.

Do not analyze anything now: wait for a task.""",
    "traces": """You are the Traces Specialist for an incident-triage graph. Your job:
- When given a task, read ONLY the file /workspace/incident/traces.csv: a per-service
  span-latency comparison table with baseline P95, fault-window P95, and a slowdown factor
  (x) for the OnlineBoutique system.
- Report: which service has the highest slowdown_x, which stayed near 1.0x, and what that
  pattern implies about where the fault is localised. Ignore logs and metrics.
- Save your findings to /root/.openclaw/workspace/traces/findings.md and reply concisely.

Do not analyze anything now: wait for a task.""",
}

for name, briefing in BRIEFINGS.items():
    path = os.path.expanduser(f"~/.openclaw/workspace/{name}/AGENTS.md")
    with open(path, "w") as f:
        f.write(briefing + "\n")
    print("wrote", path)

5.3 Define the adjacency list#

In a directed graph, the adjacency list records for each node, the nodes reachable through its outgoing edges. allowAgents is that list for main: it specifies which agents the coordinator is permitted to spawn. Agents not on the list cannot be reached. The graph topology is enforced by configuration, not by trust.

tools.alsoAllow adds the sessions_spawn and sessions_yield primitives to main’s tool surface. Without them, the coordinator has no mechanism to fan out or wait for results.

Define the adjacency list by running the following cell:

%%bash
openclaw config set agents.defaults.subagents.allowAgents '["logs","metrics","traces"]' --strict-json
openclaw config set tools.alsoAllow '["sessions_spawn","sessions_yield","subagents"]' --strict-json
echo "main may now spawn: logs, metrics, traces"

5.4 Define the control flow#

The coordinator protocol is the program that runs on the edges of the graph: it activates (fans-out) nodes, waits for their outputs (fans-in), combines the results, and decides what to do if agents disagree (bounded retry). It is written as natural language appended to main’s AGENTS.md—the same mechanism used for individual node briefings is applied to the coordinator that owns the graph structure.

The cell is idempotent: re-running it replaces the previous version.

import os

def write_file(path, content):
    path = os.path.expanduser(path)
    os.makedirs(os.path.dirname(path), exist_ok=True)
    with open(path, "w") as f:
        f.write(content)
    print("wrote", path)
    return path

PROTOCOL = """

## Incident Triage Protocol

You coordinate three specialist agents: "logs", "metrics", and "traces". You analyze NOTHING
yourself. Each specialist writes to a fixed file in ITS OWN workspace (the only place the sandbox
lets it write): logs -> /root/.openclaw/workspace/logs/findings.md,
metrics -> /root/.openclaw/workspace/metrics/findings.md,
traces -> /root/.openclaw/workspace/traces/findings.md. You can READ these files but must NOT try
to create or delete them yourself. Each specialist OVERWRITES its own findings file every run, so no
pre-cleaning is needed. These files are the source of truth; on every wake, check which files exist.

The incident signals live in /workspace/incident/: logs.txt, metrics.csv, traces.csv.

When the user says "run incident triage on <DIR>":

1. Inputs are <DIR>/logs.txt, <DIR>/metrics.csv, <DIR>/traces.csv.
2. Spawn ALL THREE specialists back-to-back, THEN yield once. Use context "isolated":
   - sessions_spawn agentId="logs"    context="isolated" task="Analyze the log summary <DIR>/logs.txt. Overwrite your findings to /root/.openclaw/workspace/logs/findings.md and reply."
   - sessions_spawn agentId="metrics" context="isolated" task="Analyze the metrics table <DIR>/metrics.csv. Overwrite your findings to /root/.openclaw/workspace/metrics/findings.md and reply."
   - sessions_spawn agentId="traces"  context="isolated" task="Analyze the trace-latency table <DIR>/traces.csv. Overwrite your findings to /root/.openclaw/workspace/traces/findings.md and reply."
   - sessions_yield with no parameters.
3. On EVERY wake: exec `ls /root/.openclaw/workspace/logs/findings.md /root/.openclaw/workspace/metrics/findings.md /root/.openclaw/workspace/traces/findings.md 2>/dev/null | wc -l`.
   If 3, go to step 4. If less than 3, call sessions_yield again (do not reply yet).
4. CORRELATE: cat the three files and give the SINGLE root cause consistent with ALL THREE signals:
   - Root cause: the culprit SERVICE and the fault type (one sentence)
   - Evidence: one line each from logs, metrics, traces
   - Confidence: high/medium/low
   - Remediation
   - Other suspects considered and why rejected (e.g. a noisy-but-healthy service).
5. If the signals do NOT converge, repeat step 2 EXACTLY ONCE with the other agents' findings added
   to each task, then always finish (termination guard).

Rules: context MUST be "isolated"; spawn all three before the first yield; never analyze a signal
yourself; never create or delete the specialists' files yourself; at most ONE re-run.
"""

AGENTS_MAIN = "/root/.openclaw/workspace/AGENTS.md"
existing = open(AGENTS_MAIN).read() if os.path.exists(AGENTS_MAIN) else ""
# Idempotent: drop any previous protocol block, then append the current one,
# so re-running this cell always installs the latest version.
idx = existing.find("## Incident Triage Protocol")
if idx != -1:
    existing = existing[:idx].rstrip() + "\n"
with open(AGENTS_MAIN, "w") as f:
    f.write(existing + PROTOCOL)
print("Triage Protocol written (idempotent)")

5.5 Restart the gateway#

The gateway loads AGENTS.md at startup. Restart it to pick up the briefings written in 5.2 and the coordinator protocol written in 5.4:

%%bash
pkill -f openclaw-gateway 2>/dev/null || true
sleep 1
# setsid: its own session, so the gateway keeps running after this cell ends
setsid nohup openclaw gateway run > ~/gateway.log 2>&1 < /dev/null &
for i in $(seq 60); do
    curl -s -o /dev/null http://127.0.0.1:18789/ && break
    sleep 2
done
curl -s -o /dev/null http://127.0.0.1:18789/ || { tail -n 30 ~/gateway.log; exit 1; }
openclaw agents list

Part 6: Run the triage#

This part demonstrates three interactions in sequence: a cold triage request, a deliberate challenge to the conclusion, and an optional direct query to a specialist. Each of these steps exercises a different capability of the graph you built.

6.1 The investigation#

You are the on-call engineer. Three signals just landed in your inbox from the OnlineBoutique system within a 3-minute window during which something went wrong. Your graph is standing by.

The cell below gives main the alert with openclaw agent, through the gateway. To follow along interactively instead, open a Jupyter terminal, launch the TUI, switch to main, and give it the same alert:

openclaw tui
  1. You might land in the Crestodian setup helper; switch to your coordinator with /agent main.

  2. Type /reset for a clean context.

  3. Send the alert:

Three signals just came in from the OnlineBoutique system: logs, metrics, and traces for a
3-minute fault window starting around 04:44 UTC. Run incident triage on /workspace/incident
and tell me: which service is the culprit, what type of fault it is, and how confident you are.

Watch the tool cards appear as main fans out to the three specialists, then waits with sessions_yield. When all three findings are in, main correlates and reports. Compare the graph’s answer with the ground truth described in What the data looks like above.

The logs alone would make you blame the wrong service. Only correlating the output from all three agents gives you the right answer, which is the whole point of the graph.

main replies as soon as it has spawned the specialists. The cell then waits until all three have written their findings and asks main for its correlated answer, the reply you would see in the TUI once the specialists finish.

import json, subprocess

def ask_agent(message, agent="main", session="triage", timeout=1800):
    """Send one message to an OpenClaw agent through the gateway and print its reply."""
    result = subprocess.run(
        ["openclaw", "agent", "--agent", agent, "--session-id", session,
         "--timeout", str(timeout), "--json", "--message", message],
        capture_output=True, text=True, timeout=timeout + 120,
    )
    if result.returncode != 0:
        raise RuntimeError(f"openclaw agent failed:\n{(result.stderr or result.stdout)[-3000:]}")
    payloads = json.loads(result.stdout)["result"]["payloads"]  # gateway replies nest under "result"
    reply = "\n".join(p.get("text", "") for p in payloads)
    print(reply)
    return reply

ALERT = """Three signals just came in from the OnlineBoutique system: logs, metrics, and traces for a
3-minute fault window starting around 04:44 UTC. Run incident triage on /workspace/incident
and tell me: which service is the culprit, what type of fault it is, and how confident you are."""
# Findings from an earlier run would make the wait below finish at once; start clean.
for name in ("logs", "metrics", "traces"):
    path = os.path.expanduser(f"~/.openclaw/workspace/{name}/findings.md")
    if os.path.exists(path):
        os.remove(path)

triage = ask_agent(ALERT)

# main replies once it has spawned the specialists and yielded; they finish in the background.
# Wait until each has written its findings (the coordinator then correlates them).
import time
pending = {name: os.path.expanduser(f"~/.openclaw/workspace/{name}/findings.md")
           for name in ("logs", "metrics", "traces")}
deadline = time.time() + 1200
while pending and time.time() < deadline:
    for name, path in list(pending.items()):
        if os.path.isfile(path):
            print(f"\n--- {name}/findings.md ---\n" + open(path).read()[:800])
            del pending[name]
    time.sleep(10)
assert not pending, f"no findings from: {', '.join(pending)}"

# All three findings are in: ask main for the correlated root cause.
print("\n--- main's correlated answer ---")
answer = ask_agent("All three specialists have written their findings. Correlate them and give me the root cause: "
                   "the culprit service, the fault type, and how confident you are.")
assert answer.strip(), "main gave no answer"

6.2 Challenge the result#

Once main gives its answer, push back on it, in the same session. The logs signal contains a deliberately misleading clue to distract investigators from the true root cause. Use the clue to challenge the result:

productcatalogservice had the most log lines by far: nearly 10,000 in that minute.
Are you sure it's not the culprit? Justify using evidence from all three specialists.

main already has the three findings.md files on disk: it doesn’t need to re-spawn the specialists. It can read them directly and explain why high log volume alone doesn’t prove fault. The coordinator can defend its conclusion using the three findings files already on disk. No re-spawn is needed, just a read and a reasoned response.

CHALLENGE = """productcatalogservice had the most log lines by far: nearly 10,000 in that minute.
Are you sure it's not the culprit? Justify using evidence from all three specialists."""
ask_agent(CHALLENGE)

6.3 Query a specialist directly (optional)#

The specialists are persistent agents. You can query any of them independently at any time. For example, you can ask the metrics specialist a more precise follow-up question directly from a Jupyter terminal (no TUI needed):

openclaw agent --agent metrics \
  --message "At exactly which minute did checkoutservice's CPU cross 50%? What was its workload at that moment?"

Or ask the traces specialist to rank all services by slowdown:

openclaw agent --agent traces \
  --message "Rank all services from highest to lowest slowdown_x. Which ones are clearly within normal range?"

Part 7: Summary and cleanup#

What you built#

You implemented a fan-out/fan-in multi-agent graph running on a single AMD Instinct™ MI300X GPU:

you  ──►  main (coordinator)
           ├──sessions_spawn──►  logs    ──► logs/findings.md
           ├──sessions_spawn──►  metrics ──► metrics/findings.md
           └──sessions_spawn──►  traces  ──► traces/findings.md
                sessions_yield
           └──────────────────────────────►  correlate  ──►  root cause

Each piece maps onto a concrete primitive you defined in OpenClaw. They form a graph that helped you triage an incident.

The key insight from this exercise is: a single signal is not always sufficient for identifying an issue. Sometimes, you need to correlate different signals. Instead of a single agent carrying out the cross-signal correlation, it can be achieved by a parallelism with typed reasoning: each node inside a graph structure sees exactly one signal and reports its findings to a coordinator, who correlates the results to draw the right conclusion.

Cleanup#

When you are done, run the cell below to stop the OpenClaw gateway and the vLLM server. OpenClaw’s state in ~/.openclaw is discarded with the container (--rm), so after a restart, re-run Parts 2 to 5.

The model weights persist on the host in hf-cache/ (the container’s Hugging Face cache mount), so the next run reuses them instead of downloading them again.

Further exploration#

  • Add a fourth signal: Add a node agent that reads the node-level CPU and memory columns from the Nezha metric CSV files; wire it into allowAgents and the protocol.

  • Try a different fault: Change FAULT_WINDOW and FAULT_MIN in Part 4 to one of the other 19 Nezha windows (e.g. cartservice network_delay at 04:27, or paymentservice cpu_contention at 06:01) and re-run the graph. The specialists’ briefings can stay the same; only change the data.

  • Add a remediation agent: After correlation, spawn a fourth agent whose job is to propose and write a concrete fix (e.g., a configuration patch or a scale-out command) based on the fault type the coordinator identified.

  • Swap the model: Replace Qwen/Qwen3.6-35B-A3B in the Part 1 launch cell with another vLLM-supported model by adjusting --tool-call-parser and --reasoning-parser.

Further reading#

subprocess.run(["pkill", "-f", "openclaw-gateway"])
stop_server(vllm_server)