Checking your setup with rocprofv3 –doctor

Checking your setup with rocprofv3 –doctor#

rocprofv3 --doctor inspects your machine and your ROCprofiler-SDK installation, and reports what would prevent rocprofv3 from working.

When rocprofv3 fails with a message that does not obviously point at a cause, run rocprofv3 --doctor first. It checks the installation, the driver and device nodes, the runtime library stack, counter-collection permissions, the environment, container device passthrough, the Python environment, the output filesystem, and the companion tools – then prints a copy-pasteable remediation for anything it finds.

It goes beyond checking that files exist: it loads the ROCm libraries rocprofv3 depends on, and when a command it runs fails, it reads the error output and explains the cause.

The tool makes no persistent changes to your system and has no --fix mode. It does more than read files, though: see What a run does for the temporary files, child processes, and GPU initialization a run involves.

Run it as the user who runs rocprofv3, without sudo. It needs no elevated privileges, and a root process sees a different system: root opens /dev/kfd regardless of group membership, holds every capability, and sudo resets variables such as LD_LIBRARY_PATH. The environ.run-as-user check warns when the tool runs as root on behalf of another account. Running as root is expected inside a container whose processes run as root; there, the group-membership checks pass because they do not apply.

What a run does#

Every check declares a probe category, shown by --list-checks and in JSON output as probe:

Probe	What it does	When it runs
`passive`	Reads files, `/sys`, `/proc`, and the environment, and runs read-only system queries such as `ldconfig -p`. The output-directory checks create and immediately delete one exclusively-created temporary file.	Every run
`process`	Runs code in a child process: loads the ROCm libraries (`runtime.libraries-load`), imports Python modules and opens an in-memory `sqlite3` database (the `python.*` checks), and runs `rocprofv3-avail --help`. A crash or hang in the child costs one check, never the report.	Every run
`gpu`	Initializes the GPU runtime: `counters.avail-enumeration` runs `rocprofv3-avail info --pmc`, which opens every GPU agent.	Every run; skip it with `--skip counters.avail-enumeration`
opt-in	The `smoke` checks run `rocprofv3-avail info` and `rocprofv3` itself.	Only with `--run-smoke-test` or when named with `--only`

Every child process has a timeout, and on timeout the child’s whole process tree is killed. A default run takes about a second on a healthy machine.

Installation#

--doctor is an option of rocprofv3, so there is nothing separate to install or put on your PATH: wherever rocprofv3 runs, rocprofv3 --doctor does. (The checks themselves are installed under <prefix>/libexec/rocprofiler-sdk/, which is not meant to be run directly.) rocprofv3 is in the bin directory of your ROCm installation, which depends on how ROCm was installed:

Installation method	`bin` directory
ROCm system packages (`apt`/`dnf`)	`/opt/rocm/bin` (a link to `/opt/rocm-X.Y.Z/bin`)
TheRock system packages (`amdrocm-*`)	`/opt/rocm/core-X.Y/bin`, also reachable as `/opt/rocm/bin`
TheRock Python packages (`pip install rocm[...]`)	the virtual environment’s `bin`; `rocm-sdk path --bin` prints it
TheRock tarball, source build, or custom prefix	`<prefix>/bin`

If rocprofv3 is not already on your PATH, add that directory, for example:

export PATH=$PATH:/opt/rocm/bin

Basic usage#

Run it with no further options to check everything:

rocprofv3 --doctor

Options after --doctor configure the check rather than profiling, for example rocprofv3 --doctor --format json. rocprofv3 --doctor --help lists them. Do not combine --doctor with profiling options: it checks the setup and exits, and runs no application.

Understanding the output#

The report is grouped by subsystem. Each line shows a status tag, the check id, and a short title:

rocprofv3 --doctor v1.4.0  (ROCm 10.0.0)
Checking: /opt/rocm-10.0.0  (found via install prefix of rocprofv3; ROCm system packages)

  Installation
  ============
  [ PASS ]  install.rocm-root                   ROCm root directory found
  [ PASS ]  install.sdk-library                 librocprofiler-sdk.so present
  [ PASS ]  install.kokkosp-library             librocprofiler-sdk-tool-kokkosp.so present

  Driver / Device
  ===============
  [ PASS ]  driver.kfd-device                   GPU device node exists (/dev/kfd, or /dev/dxg on WSL2)
  [ FAIL ]  driver.kfd-readable                 GPU device node readable and writable by the current user

  Summary: 41 passed, 6 warning(s), 2 failed, 3 skipped (52 total)

The status tags mean:

Status	Meaning
`PASS`	The condition is satisfied.
`WARN`	The condition may be intentional or may only affect an optional feature. Warnings do not change the exit code.
`FAIL`	The condition definitively prevents some form of profiling. Fix these first.
`SKIP`	The check did not run because a check it depends on did not pass, or was excluded with `--skip`. The detail line names the dependency, so a single root cause produces one failure rather than a cascade.
`ERR`	The check itself broke (a `rocprofv3 --doctor` bug). This says nothing about your system; the detail asks you to report it with the traceback from `--verbose`.

Below the summary, a How to Fix section repeats every failure and actionable warning together with its remediation command.

Colors are used when the output is a terminal. They are disabled automatically when stdout is redirected, when --no-color is passed, or when the NO_COLOR environment variable is set.

Exit codes#

Code	Meaning
`0`	No check failed. Warnings alone do not produce a non-zero exit.
`1`	At least one check produced `FAIL`.
`2`	`rocprofv3 --doctor` itself could not run (for example, the `rocprofv3` Python package could not be imported), an `--only` pattern matched no check, or a check broke (`ERR`) and none failed. A `FAIL` outranks an `ERR`: findings about the system come first.

Exit code 0 on a warning-only run is intentional: many healthy environments legitimately set ROCR_VISIBLE_DEVICES or omit the optional ATT libraries, and failing CI on those would make the tool useless. If your CI needs to be strict about warnings, parse the JSON output instead.

Machine-readable output#

--format json emits a versioned JSON document suitable for CI assertions and for attaching to bug reports:

rocprofv3 --doctor --format json

{
  "schema_version": 1,
  "tool_version": "1.4.0",
  "rocm_version": "10.0.0",
  "rocm_root": "/opt/rocm",
  "rocm_root_source": "install prefix of rocprofv3",
  "install_kind": "system-package",
  "timestamp_utc": "2026-09-15T12:34:56Z",
  "summary": { "pass": 41, "warn": 6, "fail": 2, "skip": 3, "error": 0, "total": 52 },
  "checks": [
    {
      "id": "install.sdk-library",
      "group": "install",
      "title": "librocprofiler-sdk.so present",
      "severity": "error",
      "probe": "passive",
      "depends": ["install.rocm-root"],
      "status": "pass",
      "detail": "found at /opt/rocm/lib/librocprofiler-sdk.so",
      "remediation": "",
      "data": { "resolved_path": "/opt/rocm/lib/librocprofiler-sdk.so" }
    }
  ]
}

schema_version is an integer that is incremented only on a backward-incompatible change. CI scripts should assert on it:

rocprofv3 --doctor --format json > doctor.json
python3 -c "
import json, sys
d = json.load(open('doctor.json'))
assert d['schema_version'] == 1
sys.exit(1 if d['summary']['fail'] else 0)
"

Filtering checks#

List every available check id and group:

rocprofv3 --doctor --list-checks

Run only part of the catalog with --only, or exclude part of it with --skip. Both flags are repeatable, match against either the check id or the group name, and accept glob patterns. --only also runs the prerequisites of the checks it selects, so a selected check never reports on a foundation nobody verified; --skip takes precedence, and a check whose prerequisite was skipped reports SKIP. An --only pattern that matches nothing is an error (exit code 2), so a typo cannot pass as a clean run:

rocprofv3 --doctor --only driver              # only the driver group
rocprofv3 --doctor --only driver.kfd-*        # only the KFD checks
rocprofv3 --doctor --only driver --only python
rocprofv3 --doctor --skip counters.avail-enumeration

Skipping a specific check is the right response when you know a check does not apply to your environment – for example counters.avail-enumeration on a deliberately restricted system where enumerating counters is expected to fail.

Verbose and quiet modes#

--verbose shows the detail line and the structured data for every check, including passing ones. This is the most useful form to attach to a bug report:

rocprofv3 --doctor --verbose

--quiet prints only failing and warning checks:

rocprofv3 --doctor --quiet

--output FILE writes the report to a file in addition to stdout. The file copy is always written without color codes:

rocprofv3 --doctor --output rocprofv3-doctor-report.txt

Smoke testing#

By default the tool does not run rocprofv3 itself. Pass --run-smoke-test, or name the checks with --only, to also run the smoke checks:

rocprofv3 --doctor --run-smoke-test
rocprofv3 --doctor --only smoke

They can take 30 seconds or more, which is why they are opt-in. What they establish is deliberately narrow:

smoke.avail-info shows that the HSA runtime initializes and that rocprofiler-sdk enumerates at least one GPU agent.
smoke.rocprofv3-launcher shows that rocprofv3 starts, injects its tool library, and runs a program (/bin/true) to completion. That program launches no GPU work, so a pass does not verify kernel execution, callback delivery, or trace output. Its output goes to a private temporary directory that is removed on success and kept (and reported) on failure.

How failures are diagnosed#

A library being present on disk does not mean it works. runtime.libraries-load loads librocprofiler-sdk.so, the rocprofv3 tool library, the HSA runtime, AQLProfile, HIP, and ROCTx, each in a separate child process with every symbol resolved immediately. This catches the failures that otherwise appear only when rocprofv3 starts: a missing dependency, libraries mixed from two ROCm versions, an old libstdc++ from a conda environment, or a corrupt file. When a dependency is missing, every missing dependency is listed (via ldd), not only the first. The child loads the libraries with the library search path rocprofv3 gives a profiled application (LD_LIBRARY_PATH plus <rocm-root>/lib), and with LD_PRELOAD cleared and the SDK’s load-time initialization disabled, so loading has no side effects.

When a check runs a command – rocprofv3-avail, or rocprofv3 in the smoke tests – and it fails or crashes, its output is matched against known error messages, and the report says what the output shows and what most likely caused it. A matched message is treated as evidence, not as a conclusion: where one message has several possible causes, the tool checks facts on the machine before naming one, and labels the cause likely only when those facts support it, possible otherwise. For example, ROCr reports HSA_STATUS_ERROR_OUT_OF_RESOURCES both when /dev/kfd cannot be opened and for genuine allocation failures, so the tool looks at /dev/kfd (/dev/dxg under WSL2) first:

[ WARN ]  counters.avail-enumeration          rocprofv3-avail enumerates counters
           rocprofv3-avail info --pmc exited 1: the HSA runtime reported
           HSA_STATUS_ERROR_OUT_OF_RESOURCES (likely cause: the current user cannot
           open /dev/kfd read-write)

A fix is suggested only for the mechanism actually implicated. When the output matches nothing known, the report keeps the last line of output as evidence and suggests re-running with ROCPROFILER_LOG_LEVEL=info.

In JSON output, each diagnosis is under data.diagnoses with its id, the observation, the candidate cause and its confidence (high or possible), the evidence line that matched, and the related_checks worth reading alongside it. The recognised messages are:

Message	Diagnosis
`cannot open shared object file`	A dependency is missing: a broken ROCm install if it is a ROCm library, otherwise a missing system package.
`undefined symbol`	Libraries from different ROCm versions are being mixed. `likely` when `LD_LIBRARY_PATH` or `LD_PRELOAD` points at another ROCm installation.
version `GLIBCXX_...' not found	An older `libstdc++`/glibc is being loaded, often from a conda environment.
`wrong ELF class`, `invalid ELF header`	A 32-bit library shadows a 64-bit one, or a library file is corrupt.
`HSA_STATUS_ERROR_OUT_OF_RESOURCES`	If `/dev/kfd` (`/dev/dxg` under WSL2) is missing or not accessible, the GPU could not be opened. Otherwise, possibly a genuine resource limit such as exhausted GPU memory, or under WSL2 a runtime that does not take the DXG path.
`no ROCm-capable device is detected`	No GPU is visible to HIP. `likely` caused by a `*_VISIBLE_DEVICES` mask that hides every GPU (see `environ.gpu-visibility`); any other mask is only a `possible` cause, since it may select a GPU that exists.
`could not be locked for profiling due to lack of permissions`, `Required permission (CAP_PERFMON) is not set`	The GPU could not be locked for exclusive profiling without `CAP_PERFMON`. Dispatch counters still work but may be inaccurate if another process profiles the same GPU; device-wide counter collection is degraded. Running as root grants the capability. `perf_event_paranoid` does not govern this, and file capabilities on the `rocprofv3` script have no effect.
`has a profiler attached to it`	Another process holds the GPU’s profiler lock.
`depends on a newer version of KFD`, `does not support locking device`	The amdgpu kernel driver is too old for the requested feature.
`Context has a conflict with another context`	Another profiling tool is already active in the process. `likely` when `LD_PRELOAD`, `HSA_TOOLS_LIB`, or `ROCP_TOOL_LIBRARIES` injects one.
`No module named '...'`	A Python module is missing.

A crash with none of these messages is still reported, with the signal that killed the process and a possible cause.

ROCm installation layouts#

ROCm is not always under /opt/rocm. rocprofv3 --doctor inspects the first of these that contains a ROCm installation:

The prefix rocprofv3 itself is installed under (symlinks resolved).
ROCM_PATH, ROCM_HOME, or ROCM_DIR.
The prefix of the rocprofv3 found on PATH.
A TheRock Python package (_rocm_sdk_core) importable by the current interpreter.
/opt/rocm, then the newest /opt/rocm/core-X.Y.

A virtual-environment prefix is searched for TheRock Python packages too, so running the tool from an activated venv finds the ROCm inside it.

The report header and install.rocm-root show which root was chosen, how it was found, and how it appears to have been installed (install_kind in JSON output: system-package, therock-package, python-wheel, distro-package, or standalone). Remediation commands follow that layout – for example, a missing library in a Python-package installation suggests pip, not apt.

When no location contains a ROCm installation, install.rocm-root fails and the checks that depend on it report SKIP, so the report shows that one cause rather than a missing file for every library.

Override the root explicitly when several installations exist side by side:

rocprofv3 --doctor --rocm-root /opt/rocm-6.2.0
rocprofv3 --doctor --rocm-root /opt/rocm/core-10.0
rocprofv3 --doctor --rocm-root "$(rocm-sdk path --root)"
rocprofv3 --doctor --rocm-root ~/therock/install

Common findings#

Check	What it means
`driver.kfd-device`	Neither `/dev/kfd` nor, under WSL2, `/dev/dxg` exists, so the amdgpu/KFD driver is not loaded or no AMD GPU is present. Every GPU-dependent check downstream reports `SKIP`.
`driver.kfd-readable`	The device node exists but your account cannot open it. Add yourself to the `render` and `video` groups and start a new login session. Under WSL2, `/dev/dxg` is normally world-accessible; check its mode instead.
`driver.render-group`	When your account is already listed in the group but the session is not, the remediation is to re-login or run `newgrp render` – not to run `usermod` again.
`runtime.libraries-load`	A ROCm library is present but cannot be loaded. The detail names the cause – usually a missing dependency or libraries mixed from two ROCm installations.
`install.kokkosp-library`	`librocprofiler-sdk-tool-kokkosp.so` is missing. `rocprofv3` resolves it on every launch, not only with `--kokkos-trace`, so every run fails.
`install.metrics-path`	The counter database `config.yaml` is not where the SDK looks. Each `:`-separated entry of `ROCPROFILER_METRICS_PATH` is searched first, then the installed copy; when the variable names no usable entry, the SDK falls back to the installed copy and the check warns.
`counters.cap-perfmon`	Informational. Without `CAP_PERFMON`, dispatch counter collection works, but the GPU cannot be locked for exclusive profiling and device-wide counter collection is degraded. `kernel.perf_event_paranoid` is deliberately not checked: rocprofiler-sdk does not use perf events.
`driver.amdgpu-module`	Passes whenever `/sys/module/amdgpu` exists. The driver version is shown only when the driver publishes one – the DKMS driver does, the in-tree driver of a distribution kernel does not.
`driver.gpu-topology` (WSL2)	Under WSL2 there is no KFD topology. The check instead locates `libdxcore.so` (provided by WSL under `/usr/lib/wsl/lib`) and `librocdxg.so`, and fails when `HSA_ENABLE_DXG_DETECTION=0`. `driver.dri-devices`, `driver.amdgpu-module`, and the group checks report `SKIP`: they do not apply there.
`environ.gpu-visibility`	A `*_VISIBLE_DEVICES` variable restricts the GPUs `rocprofv3` sees. `-1` hides all of them, as does an empty `ROCR_VISIBLE_DEVICES`; an empty `HIP_VISIBLE_DEVICES` or `CUDA_VISIBLE_DEVICES` restricts nothing, and a non-empty `HIP_VISIBLE_DEVICES` overrides `CUDA_VISIBLE_DEVICES`. The remediation unsets every such variable that is set.
`python.sqlite3`	Warning only: the `rocpd` command needs Python `sqlite3` to convert, merge, or query databases, but `rocprofv3 --output-format rocpd` writes them without it.
`environ.run-as-user`	The tool runs as root on behalf of another account (through `sudo`, `su`, or similar), so its results describe root rather than the user who profiles. Re-run it without `sudo`. If you also run `rocprofv3` with `sudo`, the results apply; use `sudo -E` for both.
`environ.ld-preload-conflict`	`LD_PRELOAD` loads a v1/v2 profiler library. Only one profiler may intercept the ROCm runtime at a time; `unset LD_PRELOAD`.
`install.no-mixed-rocm`	`LD_LIBRARY_PATH`, `ROCM_PATH`, or `ROCM_HOME` points at a different ROCm installation than the one being inspected, which risks an ABI mismatch. Paths that are links into the same installation, such as `/opt/rocm` and `/opt/rocm/core-X.Y`, are not reported.
`container.device-passthrough`	You are in a container and the GPU device nodes were not mounted into it. The remediation prints the required `docker run` flags.