What is ROCm Systems Profiler?

What is ROCm Systems Profiler?#

ROCm Systems Profiler (rocprofiler-systems) is a system-level profiler for applications running on the CPU or the CPU and GPU. Using dynamic binary instrumentation, call-stack sampling, and other techniques, it captures HIP/HSA APIs, kernel dispatches, memory copies, RCCL, and GPU telemetry (temperature, power, utilization, interconnect) on one timeline, down to the function and line number currently executing. It also correlates this GPU-side data with host call stacks, MPI, OpenMP, and Python activity, so you can see why the device is busy, stalled, or waiting.

For example, in a distributed training job, the GPU can appear idle not because its kernels are slow, but because a data loader is starved, an MPI collective is blocking, or Python’s GIL is stalling the dispatch queue.

This topic orients you to how ROCm Systems Profiler is put together and how to invoke it. For the full, categorized feature catalog and use cases, see ROCm Systems Profiler features and use cases.

Capabilities#

The feature set is spread across several how-to and conceptual pages; the following tables are a quick index into them:

Tracing#

Feature	Description	See also
ROCm API tracing	Traces ROCm API and runtime domains via ROCprofiler-SDK, including HIP, HSA, ROCTx, RCCL, kernel dispatches, memory events, rocDecode, and rocJPEG. List available domains with `rocprof-sys-avail --list-domains`.	GPU metrics
MPI and communication tracing	Records collective and point-to-point operation timing with per-rank attribution across MPI (via standard PMPI wrappers), RCCL, UCX, and OpenSHMEM/rocSHMEM.	Communication runtime profiling
OpenMP tracing	Captures thread team creation, parallel regions, task execution, and synchronization via the OMPT callback interface.	OpenMP performance profiling
Unified memory	Tracks page faults and host/device migrations for HIP managed memory (KFD).	Unified memory profiling

Metrics#

Feature	Description	See also
GPU telemetry	Samples GPU temperature, power, utilization, clocks, memory, XGMI/PCIe bandwidth, and VCN/JPEG engine activity via `amd-smi`.	GPU metrics, XGMI, PCIe, and SDMA metrics sampling and monitoring, VCN and JPEG activity sampling and tracing
GPU hardware counters	Collects GPU hardware counters (occupancy, VALU utilization, wavefront counts, etc.) via `--gpu-events`. See `rocprof-sys-avail -H -c GPU` for the device-specific list.	GPU metrics
CPU hardware counters	Reads IPC, cache misses, branch mispredictions, and other CPU PMU events via the Linux `perf` subsystem. See `rocprof-sys-avail -H -c CPU` for the device-specific list.	CPU metrics
Network (NIC) metrics	Samples NIC throughput and errors (PAPI); AINIC where supported.	Network performance profiling

Collection mechanism (host-side correlation)#

Feature	Description	See also
Binary instrumentation	Inserts probes at selected function entry/exit with no source changes (runtime instrumentation or binary rewrite). Complete, deterministic host call-stack data.	Binary instrumentation vs. call-stack sampling
Call-stack sampling	Periodically interrupts the application and records the host call stack (and related metrics). Lower-overhead statistical data to correlate with the GPU timeline.	Binary instrumentation vs. call-stack sampling
Python hooks	Instruments Python interpreter call frames for mixed Python/C++/HIP workload analysis.	Profiling Python scripts
Process attachment	Attaches to an already-running process without restarting it.	Attaching to a running process

Analysis#

Feature	Description	See also
Trace	Event timeline of GPU APIs, kernels, copies, and correlated host activity. Default analysis mode.	Data collection modes, Understanding the Systems Profiler output
Profile	High-level summary profiles with statistical aggregations (mean, min, max, stddev) per function. Lower overhead than a full timeline; JSON and text outputs are generated.	Data collection modes
Causal profiling	Estimates the end-to-end speedup from optimizing a given function or line by selectively slowing other code regions.	Performing causal profiling

How it works#

ROCm Systems Profiler couples GPU kernel dispatches, memory copies, and device telemetry with the host-side activity around them. GPU data is captured via ROCprofiler-SDK callbacks and amd-smi polling; host call stacks, MPI, OpenMP, and Python frames are captured via binary instrumentation, statistical sampling, callback APIs, or symbol interception. Everything is merged into a single trace/profile output, correlated on the GPU timeline.

For the full explanation of each collection mode, including overhead trade-offs and a worked instrumentation-vs-sampling example, see Data collection modes.

Binary instrumentation vs. call-stack sampling#

Binary instrumentation and call-stack sampling can be used independently or together, trading overhead against completeness:

Mode	Overhead	Best for
Binary instrumentation	Higher	Function-level profiling of selected functions when complete, deterministic call data is needed and the added overhead is acceptable.
Call-stack sampling	Low	Whole application context at low overhead. For example, long-running jobs, MPI workloads, etc.
Combined (both enabled)	Higher than sampling alone	Enable statistical sampling after binary instrumentation to help “fill in the gaps” between instrumented regions.

To use ROCm Systems Profiler for instrumentation, follow these two configuration steps:

Indicate the functions and modules to instrument in the target binaries, including the executable and any libraries.
Specify the instrumentation parameters to use when the instrumented binaries are launched.

Command-line tools#

ROCm Systems Profiler ships as a set of standalone executables, each oriented toward a different profiling workflow:

Tool	Purpose	Example	See also
`rocprof-sys-avail`	Lists what is available to configure and collect on your system (settings, domains, counters, components).	`rocprof-sys-avail -d` or `rocprof-sys-avail -H -c GPU`	General tips for using ROCm Systems Profiler
`rocprof-sys-instrument`	Performs dynamic binary instrumentation, including binary rewriting.	`rocprof-sys-instrument -o ./app.inst -- ./app`	Instrumenting and rewriting a binary application
`rocprof-sys-run`	Launches a binary-rewritten executable, or profiles an application directly using preset/domain flags.	`rocprof-sys-run -- ./app.inst`	Instrumenting and rewriting a binary application, Quick start
`rocprof-sys-sample`	Performs call-stack sampling without instrumentation.	`rocprof-sys-sample -f 1000 -- ./app`	Sampling the call stack
`rocprof-sys-attach`	Attaches to an already-running process.	`rocprof-sys-attach -p $(pidof my_app)`	Attaching to a running process
`rocprof-sys-causal`	Performs causal profiling to estimate optimization impact.	`rocprof-sys-causal -l foo -- ./app`	Performing causal profiling

Preset profiles#

Presets replace manually setting numerous environment variables with a single --preset flag available on rocprof-sys-run and rocprof-sys-sample. Presets were introduced in ROCm 7.12 and expanded into an extensible JSON-based system in ROCm 7.13.

Discover, inspect, and apply a preset as follows:

List available presets:
```
rocprof-sys-run --list-presets
```
See what a preset configures:
```
rocprof-sys-run --explain=balanced
```
Run with the preset (balanced is a good starting point):
```
rocprof-sys-run --preset=balanced -- ./my_app
```

Optionally, combine the preset with domain flags for finer control:

rocprof-sys-run --preset=balanced --gpu=temp,power -- ./my_app

Each built-in preset enables a different combination of tracing, profiling, sampling, and domains, tuned for a specific workload scenario:

Category	Presets	Best for	See also
General	`balanced`, `profile-only`, `detailed`	Most profiling scenarios; minimal-overhead production profiling; in-depth full-system analysis	General purpose presets
GPU and workload	`trace-gpu`, `workload-trace`, `trace-hw-counters`	GPU device activity; AI/ML, HPC, and GPU-accelerated training; hardware counter collection	GPU and workload presets
HPC	`trace-hpc`, `trace-openmp`, `profile-mpi`	MPI/OpenMP/Kokkos applications; OpenMP target offload; MPI communication latency analysis	HPC presets
API tracing	`sys-trace`, `runtime-trace`	Full system API visibility; runtime-only API tracing	API tracing presets

For the exact configuration behind a preset, run rocprof-sys-run --explain=<name>. For domain flags, configuration export, and custom presets, see Using preset profiles and domain flags.

Output formats#

ROCm Systems Profiler supports several output formats, each suited to a different analysis or visualization workflow. rocpd is the default output format.

Format	File extension	Description	Viewer
ROCm Profiling Data (rocpd)	`.db`	Detailed trace and counter data stored as a SQLite3 database; queryable with SQL or convertible to other formats via `rocpd convert`	ROCm Optiq
Perfetto (pftrace)	`.pftrace`	Detailed trace stored as a protocol buffer for interactive timeline visualization	ui.perfetto.dev
Text	`.txt`	Aggregated results (mean, min, max, stddev per function) as human-readable text	Text editor
JSON	`.json`	Aggregated high-level results for programmatic analysis; hatchet-compatible	Any JSON viewer, hatchet, custom scripts

Output-format selection differs by tool:

When no --output-format is specified, rocprof-sys-run and rocprof-sys-sample produce a rocpd (SQLite3 database) trace:
```
rocprof-sys-run -- ./my_app
```
--output-format (introduced in ROCm 7.14) selects one or more formats explicitly:
```
rocprof-sys-run --output-format pftrace rocpd json text -- ./my_app
```
rocprof-sys-attach uses its own -F flag with different token names for the same formats (perfetto instead of proto):
```
rocprof-sys-attach -p 12345 -F perfetto,rocpd
```

For the legacy flags and the environment variables each --output-format token maps to, see Selecting output formats. For output path conventions, metadata, and per-format details, see Understanding the Systems Profiler output.