Profile mode#

The following chapter walks you through ROCm Compute Profiler’s core profiling features by example.

Learn about analysis with ROCm Compute Profiler in Analyze mode. For an overview of ROCm Compute Profiler’s other modes, see Modes.

Profiling#

Use the rocprof-compute executable to acquire all necessary performance monitoring data through analysis of compute workloads.

Profiling with ROCm Compute Profiler provides the following benefits:

Run rocprof-compute profile -h for more details. See Basic usage.

Warning

Kernel dispatches are serialized across HIP streams on the same GPU during profiling.

ROCm Compute Profiler collects GPU performance counters with kernel dispatch association which requires serializing kernel dispatches. Kernels launched on separate HIP streams on the same GPU will not execute concurrently during profiling. Streams on different GPUs are not serialized. This means:

  • Kernel duration and throughput metrics reflect serialized execution, not the concurrent behavior that may occur during normal execution.

  • Some metrics may show reduced utilization compared to normal execution due to the lack of concurrent kernel execution.

Profiling example#

The ROCm/rocm-systems repository includes source code for a sample GPU compute workload, vcopy.cpp. A copy of this file is available in the share/sample subdirectory after a normal ROCm Compute Profiler installation, or via the $ROCPROFCOMPUTE_SHARE/sample directory when using the supplied modulefile.

The examples in this section use a compiled version of the vcopy workload to demonstrate the use of ROCm Compute Profiler in MI accelerator performance analysis. Unless otherwise noted, the performance analysis is done on the MI325X platform.

Workload compilation#

The following example demonstrates compilation of vcopy.

$ hipcc vcopy.cpp -o vcopy
$ ls
vcopy   vcopy.cpp
$ ./vcopy -n 1048576 -b 256
vcopy testing on GCD 0
Finished allocating vectors on the CPU
Finished allocating vectors on the GPU
Finished copying vectors to the GPU
sw thinks it moved 1.000000 KB per wave
Total threads: 1048576, Grid Size: 4096 block Size:256, Wavefronts:16384:
Launching the  kernel on the GPU
Finished executing kernel
Finished copying the output vector from the GPU to the CPU
Releasing GPU memory
Releasing CPU memory

The following sample command profiles the vcopy workload.

$ rocprof-compute profile --name vcopy -- ./vcopy -n 1048576 -b 256
   INFO 6a57288d55

                                 __                                       _
_ __ ___   ___ _ __  _ __ ___  / _|       ___ ___  _ __ ___  _ __  _   _| |_ ___
| '__/ _ \ / __| '_ \| '__/ _ \| |_ _____ / __/ _ \| '_ ` _ \| '_ \| | | | __/ _ \
| | | (_) | (__| |_) | | | (_) |  _|_____| (_| (_) | | | | | | |_) | |_| | ||  __/
|_|  \___/ \___| .__/|_|  \___/|_|        \___\___/|_| |_| |_| .__/ \__,_|\__\___|
               |_|                                           |_|

   INFO Rocprofiler-Compute version: 3.5.0
   INFO Profiler choice: rocprofiler-sdk
   INFO Path: /home/auser/rocm-systems/projects/rocprofiler-compute/workloads/vcopy/MI325X
   INFO Target: MI325X
   INFO Command: ./sample/vcopy -n 1048576 -b 256
   INFO Kernel Selection: None
   INFO Kernel Iteration Range: None
   INFO Filtered sections: All
   INFO
   INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   INFO Collecting Performance Counters
   INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   INFO
   INFO Using native counter collection tool: /tmp/rocprofiler-compute-tool-xxxxx/librocprofiler-compute-tool.so
   INFO [profiling] Iteration multiplexing: Disabled
   INFO [Run 1/14][Approximate profiling time left: pending first measurement...]
   INFO [profiling] Current input file: /home/auser/rocm-systems/projects/rocprofiler-compute/workloads/vcopy/MI325X/perfmon/pmc_perf_0.yaml
   INFO    |-> [rocprofiler-sdk] [rocprofiler-compute] [rocprofiler_configure] (priority=1) is using rocprofiler-sdk v1.1.0 (1.1.0)
   INFO    |-> [rocprofiler-sdk] W20260323 16:43:44.337323 139842239868672 simple_timer.cpp:55] [rocprofv3] tool initialization ::     0.250706 sec
   INFO    |-> [rocprofiler-sdk] [rocprofiler-compute] In tool init
   INFO    |-> [rocprofiler-sdk] W20260323 16:43:44.337534 139842239868672 simple_timer.cpp:55] [rocprofv3] './sample/vcopy -n 1048576 -b 256' ::     0.000000 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:43:44.511316 139842239868672 tool.cpp:2422] HSA version 8.21.0 initialized (instance=0)
   INFO    |-> [rocprofiler-sdk] W20260323 16:43:44.611214 139842239868672 simple_timer.cpp:55] [rocprofv3] './sample/vcopy -n 1048576 -b 256' ::     0.273680 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:43:44.626377 139842239868672 generateRocpd.cpp:582] writing SQL database for process 113640 on node 1574819130
   INFO    |-> [rocprofiler-sdk] E20260323 16:43:44.626821 139842239868672 generateRocpd.cpp:605] Opened result file: /home/auser/rocm-systems/projects/rocprofiler-compute/workloads/vcopy/MI325X/out/pmc_1/banff-ccs-aus-g05-05/113640_results.db (UUID=0000001e-7561-7561-8a76-754447167346)
   INFO    |-> [rocprofiler-sdk] W20260323 16:43:44.662283 139842239868672 simple_timer.cpp:55] SQLite3 generation :: rocpd_string             ::     0.008452 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:43:44.662683 139842239868672 simple_timer.cpp:55] SQLite3 generation :: rocpd_info_node          ::     0.000389 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:43:44.663665 139842239868672 simple_timer.cpp:55] SQLite3 generation :: rocpd_info_process       ::     0.000977 sec

...

   INFO    |-> [rocprofiler-sdk] Finished executing kernel
   INFO    |-> [rocprofiler-sdk] Finished copying the output vector from the GPU to the CPU
   INFO    |-> [rocprofiler-sdk] Releasing GPU memory
   INFO    |-> [rocprofiler-sdk] Releasing CPU memory
   INFO [Run 14/14][Approximate profiling time left: 0 seconds]...
   INFO [profiling] Current input file: /home/auser/rocm-systems/projects/rocprofiler-compute/workloads/vcopy/MI325X/perfmon/pmc_perf_9.yaml
   INFO    |-> [rocprofiler-sdk] [rocprofiler-compute] [rocprofiler_configure] (priority=1) is using rocprofiler-sdk v1.1.0 (1.1.0)
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:43.871622 140315166887680 simple_timer.cpp:55] [rocprofv3] tool initialization ::     0.224905 sec
   INFO    |-> [rocprofiler-sdk] [rocprofiler-compute] In tool init
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:43.871808 140315166887680 simple_timer.cpp:55] [rocprofv3] './sample/vcopy -n 1048576 -b 256' ::     0.000000 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.049909 140315166887680 tool.cpp:2422] HSA version 8.21.0 initialized (instance=0)
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.147303 140315166887680 simple_timer.cpp:55] [rocprofv3] './sample/vcopy -n 1048576 -b 256' ::     0.275496 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.162131 140315166887680 generateRocpd.cpp:582] writing SQL database for process 116379 on node 1574819130
   INFO    |-> [rocprofiler-sdk] E20260323 16:44:44.162543 140315166887680 generateRocpd.cpp:605] Opened result file: /home/auser/rocm-systems/projects/rocprofiler-compute/workloads/vcopy/MI325X/out/pmc_1/banff-ccs-aus-g05-05/116379_results.db (UUID=0000001f-5e13-7e13-9cab-4e867beb6f39)
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.195657 140315166887680 simple_timer.cpp:55] SQLite3 generation :: rocpd_string             ::     0.006754 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.196516 140315166887680 simple_timer.cpp:55] SQLite3 generation :: rocpd_info_node          ::     0.000842 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.197639 140315166887680 simple_timer.cpp:55] SQLite3 generation :: rocpd_info_process       ::     0.001077 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.214007 140315166887680 simple_timer.cpp:55] SQLite3 generation :: rocpd_info_agent         ::     0.015730 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.248623 140315166887680 simple_timer.cpp:55] SQLite3 generation :: rocpd_info_pmc           ::     0.034606 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.249465 140315166887680 simple_timer.cpp:55] SQLite3 generation :: rocpd kernel info        ::     0.000832 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.249471 140315166887680 simple_timer.cpp:55] SQLite3 generation :: rocpd_region             ::     0.000002 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.255495 140315166887680 simple_timer.cpp:55] SQLite3 generation :: rocpd_kernel_dispatch    ::     0.006021 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.255503 140315166887680 simple_timer.cpp:55] SQLite3 generation :: rocpd_pmc_event          ::     0.000000 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.255505 140315166887680 simple_timer.cpp:55] SQLite3 generation :: rocpd_memory_copy        ::     0.000000 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.255507 140315166887680 simple_timer.cpp:55] SQLite3 generation :: rocpd_memory_allocate    ::     0.000001 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.255584 140315166887680 simple_timer.cpp:55] SQLite3 generation :: SQL indexing             ::     0.000076 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.255888 140315166887680 simple_timer.cpp:55] SQLite3 generation :: total                    ::     0.093758 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.257594 140315166887680 simple_timer.cpp:55] [rocprofv3] output generation ::     0.106959 sec
   INFO    |-> [rocprofiler-sdk] W20260323 16:44:44.257624 140315166887680 simple_timer.cpp:55] [rocprofv3] tool finalization ::     0.110297 sec
   INFO    |-> [rocprofiler-sdk] [rocprofiler-compute] In tool fini
   INFO    |-> [rocprofiler-sdk] [rocprofiler-compute] [generate_output] Counter collection data has been written to: /home/auser/rocm-systems/projects/rocprofiler-compute/workloads/vcopy/MI325X/out/pmc_1/116379_native_counter_collection.csv.gz
   INFO    |-> [rocprofiler-sdk] vcopy testing on GCD 0
   INFO    |-> [rocprofiler-sdk] Finished allocating vectors on the CPU
WARNING PC sampling data collection skipped as --pc-sampling is not specified.
   INFO [roofline] Checking for roofline.csv in /home/auser/rocm-systems/projects/rocprofiler-compute/workloads/vcopy/MI325X
GPU Device 0 (gfx942) with 304 CUs: Profiling...
100% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
HBM BW, GPU ID: 0, workgroupSize:256, workgroups:6225920, experiments:100, traffic:25501368320 bytes, duration:5.4 ms, mean:4705.4 GB/sec, stdev:22.1 GB/sec
100% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
MALL BW, GPU ID: 0, workgroupSize:256, workgroups:38912, experiments:100, traffic:2611340115968 bytes, duration:403.9 ms, mean:6466.1 GB/sec, stdev:26.892396 GB/sec
100% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
L2 BW, GPU ID: 0, workgroupSize:256, workgroups:38912, experiments:100, traffic:1632087572480 bytes, duration:57.3 ms, mean:28505.3 GB/sec, stdev:27.182463 GB/sec
100% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
L1 BW, GPU ID: 0, workgroupSize:256, workgroups:38912, experiments:100, traffic:127506841600 bytes, duration:3.4 ms, mean:37782.9 GB/sec, stdev:688.291117 GB/sec

...

100% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
Peak MFMA FLOPs (F32), GPU ID: 0, workgroupSize:256, workgroups:38912, experiments:100, FLOP:1275068416000, duration:8.23 ms, mean:154854.6 GFLOPS, stdev:1782.6 GFLOPS
100% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
Peak MFMA FLOPs (F64), GPU ID: 0, workgroupSize:256, workgroups:38912, experiments:100, FLOP:637534208000, duration:4.09 ms, mean:155983.3 GFLOPS, stdev:1811.2 GFLOPS
100% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||]
Peak MFMA IOPs (I8), GPU ID: 0, workgroupSize:256, workgroups:38912, experiments:100, IOP:10200547328000, duration:4.52 ms, mean:2258954.1 GOPS, stdev:41800.0 GFLOPS
   INFO [roofline] Roofline data saved to /home/auser/rocm-systems/projects/rocprofiler-compute/workloads/vcopy/MI325X/roofline.csv
Run 'rocprof-compute analyze -p /home/auser/rocm-systems/projects/rocprofiler-compute/workloads/vcopy/MI325X' for charts

Tip

To reduce verbosity of profiling output try the --quiet flag. This hides rocprofiler-sdk output and activates a progress bar.

Notice the two main stages in ROCm Compute Profiler’s default profiling routine.

  1. The first stage collects all the counters needed for ROCm Compute Profiler analysis (omitting any filters you have provided).

  2. The second stage runs the roofline benchmark to measure the roofline ceilings (skipped when -b or --set selects blocks that do not include block 4).

At the end of profiling, you can find all resulting csv files in a SoC-specific target directory; for example:

  • “MI300A” or “MI300X” for the AMD Instinct MI300 Series GPUs

  • “MI200” for the AMD Instinct MI200 Series GPUs

  • “MI100” for the AMD Instinct MI100 Series GPUs

The SoC names are generated as a part of ROCm Compute Profiler, and do not always distinguish between different GPUs in the same family; for instance, an Instinct MI210 vs an Instinct MI250.

Note

Additionally, you will notice a few extra files. An SoC parameters file, sysinfo.csv, is created to reflect the target device settings. All profiling output is stored in log.txt. Roofline-specific benchmark results are stored in roofline.csv. To generate roofline HTML plots, run rocprof-compute analyze on the profiling output directory (see Analyze mode).

$ ls workloads/vcopy/MI325X/
total 408
-rw-r--r-- 1 auser agroup   55771 Mar 21 23:49 log.txt
drwxr-xr-x 1 auser agroup    4096 Mar 21 23:47 perfmon
-rw-r--r-- 1 auser agroup   15017 Mar 21 23:48 pmc_perf.csv.gz
-rw-r--r-- 1 auser agroup    1119 Mar 21 23:47 profiling_config.yaml
-rw-r--r-- 1 auser agroup    1684 Mar 21 23:49 roofline.csv
-rw-r--r-- 1 auser agroup     899 Mar 21 23:47 sysinfo.csv

Output directory configuration#

Profile mode writes results into a workload directory. By default, the output directory is derived from --name and the target system information:

  • Without MPI rank detection, the default is ./workloads/<name>/<gpu_model>.

  • With MPI rank detection, the default is ./workloads/<name>/<rank>.

You can override the output directory with -d, --output-directory. When --output-directory is explicitly provided, --name is ignored.

The output directory can be parameterized with the following keywords:

  • %hostname%: Host name

  • %gpumodel%: GPU model

  • %rank%: MPI process rank (ignored with a warning if no rank is detected)

  • %env{NAME}%: Environment variable NAME (empty string if unset)

If MPI rank is detected and the output directory does not include %rank%, ROCm Compute Profiler appends /<rank> to avoid collisions across ranks.

Each profiling run should use a fresh output directory. A workload directory holds the output of exactly one run, so profiling into a non-empty directory fails with an error rather than silently mixing runs. To re-profile in place, add --overwrite, which clears the directory first:

$ rocprof-compute profile --name vcopy --overwrite -- ./vcopy -n 1048576 -b 256

Warning

--overwrite deletes the target directory’s existing contents. Nothing is removed without it.

Note

Automated runs should never reuse a workload directory: profile into a fresh directory each time, or pass --overwrite.

Examples:

  • Profiling without MPI:

$ rocprof-compute profile --name vcopy -- ./vcopy -n 1048576 -b 256

$ tree workloads/vcopy

└── MI325X
 ├── log.txt
 ├── perfmon
 │   ├── counter_def_0.yaml
 │   ├── counter_def_1.yaml
 │   ├── counter_def_2.yaml
 │   ├── counter_def_3.yaml
 │   ├── pmc_perf_0.yaml
 │   ├── pmc_perf_1.yaml
 │   ├── pmc_perf_10.yaml
 │   ├── pmc_perf_11.yaml
 │   ├── pmc_perf_12.yaml
 │   ├── pmc_perf_13.yaml
 │   ├── pmc_perf_2.yaml
 │   ├── pmc_perf_3.yaml
 │   ├── pmc_perf_4.yaml
 │   ├── pmc_perf_5.yaml
 │   ├── pmc_perf_6.yaml
 │   ├── pmc_perf_7.yaml
 │   ├── pmc_perf_8.yaml
 │   └── pmc_perf_9.yaml
 ├── profiling_config.yaml
 ├── results_pmc_perf_0.csv.gz
 ├── results_pmc_perf_1.csv.gz
 ├── results_pmc_perf_2.csv.gz
 ├── roofline.csv
 └── sysinfo.csv

The output files use rocpd format. See Profiling output format for details on when the final pmc_perf.csv.gz is created.

  • Profiling with MPI at host amd-ryzen:

$ mpirun -n 4 rocprof-compute profile --output-directory /tmp/profiles/%hostname%/%rank% -- ./vcopy -n 1048576 -b 256

$ tree /tmp/profiles/amd-ryzen/0

└── MI325X
 ├── log.txt
 ├── perfmon
 │   ├── counter_def_0.yaml
 │   ├── counter_def_1.yaml
 │   ├── counter_def_2.yaml
 │   ├── counter_def_3.yaml
 │   ├── pmc_perf_0.yaml
 │   ├── pmc_perf_1.yaml
 │   ├── pmc_perf_10.yaml
 │   ├── pmc_perf_11.yaml
 │   ├── pmc_perf_12.yaml
 │   ├── pmc_perf_13.yaml
 │   ├── pmc_perf_2.yaml
 │   ├── pmc_perf_3.yaml
 │   ├── pmc_perf_4.yaml
 │   ├── pmc_perf_5.yaml
 │   ├── pmc_perf_6.yaml
 │   ├── pmc_perf_7.yaml
 │   ├── pmc_perf_8.yaml
 │   └── pmc_perf_9.yaml
 ├── profiling_config.yaml
 ├── results_pmc_perf_0.csv.gz
 ├── results_pmc_perf_1.csv.gz
 ├── results_pmc_perf_2.csv.gz
 ├── roofline.csv
 └── sysinfo.csv

Profiling output format#

Raw performance counter data produced by the underlying ROCprofiler-SDK backend is written in rocpd (SQLite) format:

  • The rocpd database files are converted to gzip-compressed CSV files (results_pmc_perf_0.csv.gz, results_pmc_perf_1.csv.gz, etc.) for each profiling run, after which the database files are removed.

  • These files are merged into a single gzip-compressed pmc_perf.csv.gz file when running rocprof-compute analyze.

  • Use --retain-rocpd-output to preserve the rocpd database(s) in the workload folder for custom analysis.

Note

Intermediate CSV generation (results_*.csv) in rocpd mode and --retain-rocpd-output are deprecated and will be removed in a future release.

Filtering#

To reduce profiling time and the counters collected, use profiling filters. The filters described below apply to the default rocprofiler-sdk backend. See Configuring the environment for profiling for details on backend selection.

Choosing what to profile#

A profile run collects one of the following. Options in the same group can be combined; options from different groups can’t be used together, and rocprof-compute exits with an error:

  • Blocks: -b and its aliases --speed-of-light, --memory-chart, and --roofline, in any combination.

  • A metric set: --set. See Metric sets filtering.

  • Only the roofline benchmark: --roofline-bench-only. It also can’t be used with the deprecated --no-roof. See Benchmark only.

Without any of these options, profile mode collects every block.

Filtering options#

-b, --block

Allows system profiling on one or more selected analysis report blocks to speed up the profiling process. See Analysis report block filtering.

--speed-of-light, --memory-chart, --roofline

Aliases of -b 2, -b 3, and -b 4. They profile the Speed-of-Light, Memory Chart, and Roofline blocks.

-k, --kernel

Allows for kernel filtering. See Kernel filtering.

--kernel-iteration-range

Allows for kernel iteration filtering. See Kernel iteration filtering.

--set

Allows for single pass counter collection of sets of metrics with minimized profiling overhead. See Metric sets filtering and Choosing what to profile.

Tip

Be cautious when combining different profiling filters in the same call. Conflicting filters may result in error.

For example, filtering a kernel iteration, but that iteration doesn’t match your kernel name filter.

Analysis report block filtering#

You can profile specific hardware report blocks to speed up the profiling process. In ROCm Compute Profiler, the term analysis report block refers to a section of the analysis report which focuses on metrics associated with a hardware component or a group of hardware components. All profiling results are accumulated in the same target directory without overwriting those for other hardware components. This enables incremental profiling and analysis.

The following example only gathers hardware counters used to calculate metrics for Compute Unit - Instruction Mix (block 10) and Wavefront Launch Statistics (block 7) sections of the analysis report, while skipping over all other hardware counters.

$ rocprof-compute profile --name vcopy -b 10 7 -- ./vcopy -n 1048576 -b 256

                                 __                                       _
 _ __ ___   ___ _ __  _ __ ___  / _|       ___ ___  _ __ ___  _ __  _   _| |_ ___
| '__/ _ \ / __| '_ \| '__/ _ \| |_ _____ / __/ _ \| '_ ` _ \| '_ \| | | | __/ _ \
| | | (_) | (__| |_) | | | (_) |  _|_____| (_| (_) | | | | | | |_) | |_| | ||  __/
|_|  \___/ \___| .__/|_|  \___/|_|        \___\___/|_| |_| |_| .__/ \__,_|\__\___|
               |_|                                           |_|

INFO Rocprofiler-Compute version: 3.5.0
INFO Profiler choice: rocprofiler-sdk
INFO Path: /home/auser/rocprofiler-compute/workloads/vcopy/MI325X
INFO Target: MI325X
INFO Command: ./vcopy -n 1048576 -b 256
INFO Kernel Selection: None
INFO Kernel Iteration Range: None
INFO Filtered sections: ['10', '7']
INFO
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO Collecting Performance Counters
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

It is also possible to collect individual metrics from the analysis report by providing metric ids. The following example only collects the counters required to calculate Total VALU FLOPs (metric id 11.1.0) and LDS Utilization (metric id 12.1.0).

$ rocprof-compute profile --name vcopy -b 11.1.1 12.1.1 -- ./vcopy -n 1048576 -b 256

                                 __                                       _
 _ __ ___   ___ _ __  _ __ ___  / _|       ___ ___  _ __ ___  _ __  _   _| |_ ___
| '__/ _ \ / __| '_ \| '__/ _ \| |_ _____ / __/ _ \| '_ ` _ \| '_ \| | | | __/ _ \
| | | (_) | (__| |_) | | | (_) |  _|_____| (_| (_) | | | | | | |_) | |_| | ||  __/
|_|  \___/ \___| .__/|_|  \___/|_|        \___\___/|_| |_| |_| .__/ \__,_|\__\___|
               |_|                                           |_|

INFO Rocprofiler-Compute version: 3.5.0
INFO Profiler choice: rocprofiler-sdk
INFO Path: /home/auser/rocprofiler-compute/workloads/vcopy/MI325X
INFO Target: MI325X
INFO Command: ./vcopy -n 1048576 -b 256
INFO Kernel Selection: None
INFO Kernel Iteration Range: None
INFO Filtered sections: ['11.1.1', '12.1.1']
INFO
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO Collecting Performance Counters
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

To see the available report blocks and their metrics, use --list-metrics. See List blocks and metrics.

Kernel filtering#

Kernel filtering is based on the name of the kernels you want to isolate. Use a kernel name substring list to isolate desired kernels.

Important

Kernel filtering is strongly recommended when profiling with --iteration-multiplexing. Without it, the tool collects counters for all dispatched kernels, including incidental ones such as __amd_rocclr_fillBufferAligned and __amd_rocclr_copyBuffer. These helper kernels are typically dispatched only a few times and do not have enough dispatches to fill all counter sets, producing warnings such as:

WARNING Insufficient number of kernel calls for kernels:
__amd_rocclr_fillBufferAligned

To avoid this, use -k to profile only the kernels you care about:

$ rocprof-compute profile -k <your_kernel> --iteration-multiplexing -- <app>

If you are unsure which kernel names your application dispatches, first run a profile without --iteration-multiplexing and inspect the output to identify the kernel names of interest.

The following example demonstrates profiling isolating the kernel matching substring vecCopy.

$ rocprof-compute profile --name vcopy -k vecCopy -- ./vcopy -n 1048576 -b 256

                                 __                                       _
 _ __ ___   ___ _ __  _ __ ___  / _|       ___ ___  _ __ ___  _ __  _   _| |_ ___
| '__/ _ \ / __| '_ \| '__/ _ \| |_ _____ / __/ _ \| '_ ` _ \| '_ \| | | | __/ _ \
| | | (_) | (__| |_) | | | (_) |  _|_____| (_| (_) | | | | | | |_) | |_| | ||  __/
|_|  \___/ \___| .__/|_|  \___/|_|        \___\___/|_| |_| |_| .__/ \__,_|\__\___|
               |_|                                           |_|

INFO Rocprofiler-Compute version: 3.5.0
INFO Profiler choice: rocprofiler-sdk
INFO Path: /home/auser/rocprofiler-compute/workloads/vcopy/MI325X
INFO Target: MI325X
INFO Command: ./vcopy -n 1048576 -b 256
INFO Kernel Selection: ['vecCopy']
INFO Kernel Iteration Range: None
INFO Filtered sections: All
INFO
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO Collecting Performance Counters
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

Kernel iteration filtering#

Kernel iteration filtering selects which iterations of each kernel to profile. Indices are 1-based, so the first iteration of a kernel is 1. Each value is a positive integer or a range with start <= end, written as either start:end or start-end (for example, 1, 3:5, or 3-5).

The following example profiles the first iteration of each kernel in the application.

$ rocprof-compute profile --name vcopy --kernel-iteration-range 1 -- ./vcopy -n 1048576 -b 256

                                 __                                       _
 _ __ ___   ___ _ __  _ __ ___  / _|       ___ ___  _ __ ___  _ __  _   _| |_ ___
| '__/ _ \ / __| '_ \| '__/ _ \| |_ _____ / __/ _ \| '_ ` _ \| '_ \| | | | __/ _ \
| | | (_) | (__| |_) | | | (_) |  _|_____| (_| (_) | | | | | | |_) | |_| | ||  __/
|_|  \___/ \___| .__/|_|  \___/|_|        \___\___/|_| |_| |_| .__/ \__,_|\__\___|
               |_|                                           |_|

INFO Rocprofiler-Compute version: 3.5.0
INFO Profiler choice: rocprofiler-sdk
INFO Path: /home/auser/rocprofiler-compute/workloads/vcopy/MI325X
INFO Target: MI325X
INFO Command: ./vcopy -n 1048576 -b 256
INFO Kernel Selection: None
INFO Kernel Iteration Range: ['1']
INFO Filtered sections: All
INFO
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO Collecting Performance Counters
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

Metric sets filtering#

A metrics set contains a subset of metrics that can be collected in a single pass. This filtering option minimizes profiling overhead by only collecting counters of interest. The –set filter option provides a convenient way to group related metrics for common profiling scenarios, eliminating the need to manually specify individual metrics for typical analysis workflows. See Choosing what to profile for the options it can be combined with. --set can’t be used with --pc-sampling.

$ rocprof-compute profile --name vcopy --set compute_thruput_util -- ./vcopy -n 1048576 -b 256

                                 __                                       _
 _ __ ___   ___ _ __  _ __ ___  / _|       ___ ___  _ __ ___  _ __  _   _| |_ ___
| '__/ _ \ / __| '_ \| '__/ _ \| |_ _____ / __/ _ \| '_ ` _ \| '_ \| | | | __/ _ \
| | | (_) | (__| |_) | | | (_) |  _|_____| (_| (_) | | | | | | |_) | |_| | ||  __/
|_|  \___/ \___| .__/|_|  \___/|_|        \___\___/|_| |_| |_| .__/ \__,_|\__\___|
               |_|                                           |_|

INFO Rocprofiler-Compute version: 3.5.0
INFO Profiler choice: rocprofiler-sdk
INFO Path: /home/auser/rocprofiler-compute/workloads/vcopy/MI325X
INFO Target: MI325X
INFO Command: ./vcopy -n 1048576 -b 256
INFO Kernel Selection: None
INFO Kernel Iteration Range: None
INFO Filtered sections: ['11.2.2', '11.2.3', '11.2.4', '11.2.5']
INFO
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO Collecting Performance Counters
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

To see a list of available sets, use the --list-sets option.

$ rocprof-compute profile --list-sets

                                 __                                       _
 _ __ ___   ___ _ __  _ __ ___  / _|       ___ ___  _ __ ___  _ __  _   _| |_ ___
| '__/ _ \ / __| '_ \| '__/ _ \| |_ _____ / __/ _ \| '_ ` _ \| '_ \| | | | __/ _ \
| | | (_) | (__| |_) | | | (_) |  _|_____| (_| (_) | | | | | | |_) | |_| | ||  __/
|_|  \___/ \___| .__/|_|  \___/|_|        \___\___/|_| |_| |_| .__/ \__,_|\__\___|
               |_|                                           |_|

Available Sets:
===================================================================================================================
Set Option                          Set Title                           Metric Name                    Metric ID
-------------------------------------------------------------------------------------------------------------------
compute_thruput_util                Compute Throughput Utilization      SALU Utilization               11.2.3
                                                                        VALU Utilization               11.2.4
                                                                        VMEM Utilization               11.2.6
                                                                        Branch Utilization             11.2.7

...

launch_stats                        Launch Stats                        Grid Size                      7.1.0
                                                                        Workgroup Size                 7.1.1
                                                                        Total Wavefronts               7.1.2
                                                                        VGPRs                          7.1.5
                                                                        AGPRs                          7.1.6
                                                                        SGPRs                          7.1.7
                                                                        LDS Allocation                 7.1.8
                                                                        Scratch Allocation             7.1.9

Usage Examples:
rocprof-compute profile --set compute_thruput_util  # Profile this set
rocprof-compute profile --list-sets        # Show this help

Standalone roofline#

Roofline analysis occurs on any profile mode run, unless -b or --set selects blocks that do not include block 4. You don’t need to include any additional roofline-specific options for roofline analysis. If you want to focus only on roofline-specific performance data and reduce the time it takes to profile, you can use the --roofline option. This option checks if there is existing roofline benchmark data in the workload directory (roofline.csv):

  1. If found, skips microbenchmark execution;

  2. Otherwise, profile mode runs microbenchmarks and collects roofline performance counters.

See Choosing what to profile for the options --roofline can be combined with.

Profile mode generates roofline.csv containing microbenchmark data. To generate roofline HTML plots, use rocprof-compute analyze on a profiling output directory that contains both roofline.csv and application performance counters (see Analyze mode).

Note

Matrix multiplication benchmarking and counter collection will vary depending on which architecture is profiled:

  • gfx9 (CDNA1/2/3/4) supports Matrix Fused MultiplyAdd (MFMA).

  • gfx10+ (RDNA3+) supports Wave Matrix Multiply Accumulate (WMMA).

CDNA4+ also supports Microscaling formats, which can be identified by the “MX” prefix in datatypes. More details and other resources about MX per CDNA architecture can be found at AMD CDNA Architecture.

The cache levels that are benchmarked are dependent on the memory hierarchy levels of the architecture. See the CDNA Performance Model or RDNA Performance Model pages to view more information about the hardware blocks and cache levels supported in each architecture.

Roofline options (profile)#

--roofline-device

Allows you to specify a device ID to collect performance data from when running a roofline benchmark on your system.

-k, --kernel

Allows for kernel filtering. See Kernel filtering.

Note

For more information on data types supported based on the GPU architecture, see AMD CDNA architecture (CDNA-CDNA4)

Roofline only#

The following example demonstrates profiling roofline data only:

$ rocprof-compute profile --name occupancy --roofline -- ./tests/occupancy -n 1048576 -b 256
                                 __                                       _
 _ __ ___   ___ _ __  _ __ ___  / _|       ___ ___  _ __ ___  _ __  _   _| |_ ___
| '__/ _ \ / __| '_ \| '__/ _ \| |_ _____ / __/ _ \| '_ ` _ \| '_ \| | | | __/ _ \
| | | (_) | (__| |_) | | | (_) |  _|_____| (_| (_) | | | | | | |_) | |_| | ||  __/
|_|  \___/ \___| .__/|_|  \___/|_|        \___\___/|_| |_| |_| .__/ \__,_|\__\___|
               |_|                                           |_|
...
INFO [roofline] Generating pmc_perf.csv (roofline counters only).
INFO Rocprofiler-Compute version: 3.5.0
INFO Profiler choice: rocprofiler-sdk
INFO Path: /home/auser/rocprofiler-compute/workloads/occupancy/MI325X
INFO Target: MI325X
INFO Command: ./tests/occupancy -n 1048576 -b 256
INFO Kernel Selection: None
INFO Kernel Iteration Range: None
INFO Filtered sections: ['4']
INFO
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO Collecting Performance Counters (Roofline Only)
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO
INFO [Run 1/3][Approximate profiling time left: pending first measurement...]
INFO [profiling] Current input file: /home/auser/rocprofiler-compute/workloads/occupancy/MI325X/perfmon/pmc_perf_0.yaml
...
INFO [roofline] Checking for roofline.csv in /home/auser/rocprofiler-compute/workloads/occupancy/MI325X
INFO [roofline] No roofline data found. Generating...
Empirical Roofline Calculation
Copyright © 2026  Advanced Micro Devices, Inc. All rights reserved.
Total detected GPU devices: 8
GPU Device 0 (gfx942) with 304 CUs: Profiling...
99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
...

An inspection of our workload output folder shows roofline.csv was generated successfully.

$ ls workloads/occupancy/MI325X
total 48
drwxr-xr-x 1 auser agroup     0 Mar 21 23:49 perfmon
-rw-r--r-- 1 auser agroup   412 Mar 21 23:49 pmc_perf.csv.gz
-rw-r--r-- 1 auser agroup  1715 Mar 21 23:49 roofline.csv
-rw-r--r-- 1 auser agroup   650 Mar 21 23:49 sysinfo.csv
-rw-r--r-- 1 auser agroup   399 Mar 21 23:49 timestamps.csv

To generate roofline HTML plots from this data, see Analyze mode.

Benchmark only#

If you only want to run the roofline microbenchmark without profiling an application or collecting any performance counters, use the --roofline-bench-only option. No application run is required.

This is useful for:

  • Re-generating roofline.csv in an existing workload directory without re-profiling.

  • Running the microbenchmark on a system where only HIP is available (no rocprofiler-sdk needed).

--roofline-bench-only can’t be used with -b, its aliases, --set or --no-roof.

$ rocprof-compute profile --name my_bench --roofline-bench-only

To target a specific GPU device, use --roofline-device:

$ rocprof-compute profile --name my_bench --roofline-bench-only --roofline-device 2

To regenerate benchmark data in an existing workload directory, point --output-directory at the workload path. This replaces the existing roofline.csv, so it requires --overwrite. Unlike a full profile, --roofline-bench-only replaces only roofline.csv and leaves the rest of the workload (counter data, traces) untouched:

$ rocprof-compute profile --roofline-bench-only --overwrite --output-directory workloads/vcopy/MI300X_A1

Note

--roofline-bench-only writes roofline.csv only; rendering a roofline chart additionally requires application performance counters from a --roofline (or regular profile) run. The intended workflow is to profile first, then re-run --roofline-bench-only --overwrite against that workload later to refresh stale peak values before analyzing:

$ rocprof-compute profile --name vcopy --roofline -- ./vcopy
$ rocprof-compute profile --roofline-bench-only --overwrite --output-directory workloads/vcopy/MI300X_A1
$ rocprof-compute analyze --path workloads/vcopy/MI300X_A1

Torch trace#

ROCm Compute Profiler offers Torch operator mapping functionality to analyze the performance metrics at the PyTorch operator level. This feature maps the performance counters to specific PyTorch operators, enabling detailed performance analysis of the PyTorch workloads at the operator granularity.

When enabled, this feature instruments your PyTorch application to correlate GPU kernel executions with their originating PyTorch operators, providing insights into which operators contribute to specific performance counter values.

Warning

Torch trace is currently an experimental feature. You must pass --experimental to both profile and analyze command when using the Torch trace related options (--trace-ml-api torch for profile; --list-operators torch and --operator for analyze).

Note

Mapping PyTorch operators to GPU kernels: PyTorch operators (such as conv2d, linear, and relu) are high-level API functions. When executed on GPU, these operators may launch multiple low-level GPU kernels (such as implicit_convolve_sgemm) that perform the actual computation on the hardware. The --trace-ml-api torch option provides operator-level attribution by injecting markers that map the collected kernel performance counters to their originating PyTorch operators.

Requirements#

  • PyTorch in the profiling environment. PyTorch 2.13 or 2.14 gets the full operator trace; other versions get the reduced trace described below.

  • PyTorch application must be run as a Python script or a Python command.

--trace-ml-api torch records PyTorch operators in one of two ways:

  • The collector is a small shared library that registers a callback inside PyTorch itself. The callback runs on every thread, so the trace includes the operators that autograd runs on its own worker threads during the backward pass. This is used for PyTorch 2.13 and 2.14.

  • TorchDispatchMode is a PyTorch hook that calls back into Python around each operator. It works on any PyTorch version, but it only sees the thread it was entered on, so backward-pass operators are missed. This is used for every other PyTorch version, and whenever the collector cannot be loaded.

Profiling never stops because of this. When the collector is unavailable, --trace-ml-api torch prints a warning and continues with TorchDispatchMode.

Important

PyTorch must be installed together with ROCm from the TheRock package index. Loading two different ROCm installations in one process aborts at startup.

Install rocm[profiler] and torch from the same index, each with the device-* extra for your GPU. See Installing multi-arch PyTorch Python packages for the index URL, the supported device-* extras, and the PyTorch version compatibility matrix.

Usage#

To enable Torch operator mapping, use --experimental with the --trace-ml-api torch option when profiling a PyTorch workload:

$ rocprof-compute profile --experimental --trace-ml-api torch --name simple_net -- python simple_net.py

                                 __                                       _
 _ __ ___   ___ _ __  _ __ ___  / _|       ___ ___  _ __ ___  _ __  _   _| |_ ___
| '__/ _ \ / __| '_ \| '__/ _ \| |_ _____ / __/ _ \| '_ ` _ \| '_ \| | | | __/ _ \
| | | (_) | (__| |_) | | | (_) |  _|_____| (_| (_) | | | | | | |_) | |_| | ||  __/
|_|  \___/ \___| .__/|_|  \___/|_|        \___\___/|_| |_| |_| .__/ \__,_|\__\___|
               |_|                                           |_|

INFO Rocprofiler-Compute version: 3.5.0
INFO Profiler choice: rocprofiler-sdk
INFO Path: /home/auser/workloads/simple_net/MI325X
INFO Target: MI325X
INFO Command: python simple_net.py
INFO Kernel Selection: None
INFO Kernel Iteration Range: None
INFO Hardware Blocks: All
INFO
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO Collecting Performance Counters
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

Reducing instrumentation overhead#

By default, --trace-ml-api torch also wraps the device- and layout-related torch.Tensor methods to, cpu, cuda, and contiguous, which attribute tensor movement and layout changes to the originating PyTorch call site. These methods are called frequently, so wrapping them adds overhead and increases the trace size.

Set the ROCPROFCOMPUTE_ROCTX_DEEP_TENSOR_WRAPS environment variable to 0 (or false, no, off) in the workload environment to disable these wraps. ROCPROFCOMPUTE_ROCTX_DEEP_TENSOR_WRAPS is enabled by default.

$ ROCPROFCOMPUTE_ROCTX_DEEP_TENSOR_WRAPS=0 rocprof-compute profile --experimental --trace-ml-api torch --name simple_net -- python simple_net.py

Torch trace collector#

rocprofiler-compute ships one collector library, torch_trace_collector.so. It does not need PyTorch development packages and works with any Python version.

The collector depends on how a specific PyTorch version stores its data internally, so it runs only with PyTorch 2.13 and 2.14. Other PyTorch versions use TorchDispatchMode instead, which misses backward-pass operators.

Output#

When Torch operator mapping is enabled, profiling writes additional CSV files in the workload directory: marker_api_trace and counter_collection files with the ml_api_trace prefix. The source marker and counter files are retained in the workload directory.

Analyze reads those files and prints an operator call tree and a per-operator summary (for example with --list-operators torch or --operator).

Performance counter data#

The ml_api_trace_*_counter_collection.csv.gz files hold the same performance counters as a profile without Torch trace. In analyze mode, --operator shows the usual metric tables for the kernels that the matching operators launched.

Limitations#

The Torch trace feature currently has the following limitations:

  • Torch trace is experimental. Pass --experimental to both the profile and analyze commands.

  • The --trace-ml-api torch option requires the application to be a Python command or Python script.

  • The workload’s Python version must match the Python version used by roctx.

  • This feature adds instrumentation overhead to track operator boundaries. For performance-critical measurements, consider profiling without this option first.

If PyTorch and ROCm come from different installations, the workload aborts while loading ROCm libraries and profiling fails with output such as:

: CommandLine Error: Option 'spirv-expand-step' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
ERROR The workload and the profiler loaded two different ROCm installations in the same
process. Duplicate ROCm libraries abort at startup. Install PyTorch and rocm[profiler]
from the same package index: <link to the requirements above>

This means the install requirement above was not met.

Hierarchical operator names#

Analyze shows each operator under the operator that called it. The operator summary writes this call path with / between operators:

nn.Module.Net.forward/nn.Module.Conv2d.forward/aten::convolution
nn.Module.MyModel.forward/nn.Module.Linear.forward
aten::relu

This hierarchical information enables:

  • Context preservation: See exactly which model layer triggered each kernel.

  • Debugging: Identify performance issues in specific model components.

  • Optimization: Focus tuning efforts on bottleneck operators.

Example:

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Linear(512, 1024)
        self.decoder = nn.Linear(1024, 512)

    def forward(self, x):
        x = self.encoder(x)  # Linear.forward appears under MyModel.forward
        x = self.decoder(x)  # A second Linear.forward under MyModel.forward
        return x

Analyzing captured operators: After profiling, use the analyze CLI (see CLI analysis) to list and filter operators. --operator takes shell-style glob patterns, such as --operator 'torch:*relu*' or --operator '*/aten::addmm'. A pattern can match an operator name or its call path. A torch: or triton: prefix limits a pattern to that framework. To select all operators, pass no arguments or **.

Combining Torch operator with other options#

Torch operator mapping can be combined with other profiling options. Use --experimental with --trace-ml-api torch in all cases:

# Combine with block filtering for targeted counter collection
$ rocprof-compute profile -b 11 12 --experimental --trace-ml-api torch --name simple_net -- python simple_net.py

# Combine with iteration multiplexing
$ rocprof-compute profile --experimental --trace-ml-api torch --name simple_net --iteration-multiplexing -- python simple_net.py

# Combine with kernel filtering (filters by GPU kernel name)
$ rocprof-compute profile --experimental --trace-ml-api torch --name simple_net -k elementwise -- python simple_net.py

Triton trace#

In addition to PyTorch, ROCm Compute Profiler can map performance counters to the Triton kernels that your code launches, such as @triton.jit functions. Enable it with the --trace-ml-api triton option. It writes the same ml_api_trace output as Torch trace and is analyzed the same way.

--trace-ml-api triton does not capture the Triton kernels that torch.compile generates, because PyTorch launches those kernels itself. Use --trace-ml-api torch for them. They appear under the torch.compile call in the operator tree.

Warning

Triton trace is currently an experimental feature. You must pass --experimental to both profile and analyze commands when using the Triton trace related options (--trace-ml-api triton for profile; --list-operators triton and --operator 'triton:PATTERN' for analyze).

Requirements#

Triton trace has the same requirements and limitations as Torch trace (see Requirements and Limitations), with a valid Triton installation required in place of PyTorch.

Usage#

To enable Triton kernel mapping, use --experimental with the --trace-ml-api triton option:

$ rocprof-compute profile --experimental --trace-ml-api triton --name triton_ffn -- python triton_ffn.py

--trace-ml-api torch triton instruments both frameworks in a single run:

$ rocprof-compute profile --experimental --trace-ml-api torch triton --name torch_triton_net -- python torch_triton_net.py

Each captured operator records which framework it came from, so you can analyze each framework separately. To trace every supported framework at once, pass –trace-ml-api with no values.

To analyze the captured Triton kernels, use --list-operators triton and --operator 'triton:PATTERN' in analyze mode (see CLI analysis).

ML API trace#

--trace-ml-api without values traces every supported ML framework. Every --trace-ml-api run also records user ROCTx regions.

Warning

ML API trace is currently an experimental feature. You must pass --experimental when using it.

$ rocprof-compute profile --experimental --trace-ml-api --name torch_triton_net -- python torch_triton_net.py

The output is the same as naming every framework, as in --trace-ml-api torch triton. Each kernel is attributed to the framework that launched it. Analyze the results with --list-operators and --operator (see CLI analysis). To select a user ROCTx region and the GPU work inside it, use --list-operators user or --operator 'user:PATTERN'.

Profile vLLM workloads#

vLLM V1 runs GPU kernels in a worker process separate from the vLLM entry process. ROCm Compute Profiler profiles that worker process and records its kernel dispatches, but vLLM terminates the worker with a signal on shutdown, and counter data is only written when a process exits normally. The profiling run therefore reports success while leaving no GPU kernel dispatch or performance counter data.

To profile vLLM V1 workloads on a single GPU, set VLLM_ENABLE_V1_MULTIPROCESSING=0 before launching ROCm Compute Profiler. The vLLM workload then inherits the setting and runs the model in the profiled process, which exits normally and writes its counter data. For example:

$ VLLM_ENABLE_V1_MULTIPROCESSING=0 \
    rocprof-compute profile \
    --iteration-multiplexing \
    --speed-of-light \
    --name vllm-offline -- \
    python offline_inference.py

Important

This workaround applies to single-GPU runs. When the model is split across several GPUs, vLLM starts a separate worker process for each GPU, and a profiling run still completes without performance counter data.

Note

Disabling vLLM V1 multiprocessing does not change the model or the kernels it runs, so the collected counter data remains representative. It does affect end-to-end throughput, because work that normally overlaps across two processes is serialized into one.

Iteration multiplexing#

To reduce profiling overhead when collecting a large number of performance counters, ROCm Compute Profiler supports iteration multiplexing. This technique divides the total set of requested performance counters into smaller subsets that can be collected over multiple iterations of the kernel execution, thereby preventing the need for application replay. Each iteration collects a different subset of counters, and the results are later combined to provide a comprehensive view of the performance metrics.

Note

Iteration multiplexing is most beneficial for large workloads that take a long time to run, as it helps reduce profiling overhead by eliminating the need for application replay while spreading counter collection across iterations. For small workloads with few kernel dispatches, iteration multiplexing may result in incomplete metric calculations due to insufficient kernel dispatch counts to cover all counter subsets.

Usage#

To enable iteration multiplexing in ROCm Compute Profiler, add the --iteration-multiplexing option to your profiling command. Each dispatch of a kernel with the same launch parameters collects a different subset of counters. Launch parameters are the grid size, workgroup size, and LDS size.

Note

  • Iteration multiplexing requires ROCprofiler-SDK from ROCm 7.0.0 or later and is only supported with the rocprofiler-sdk backend.

  • Iteration multiplexing depends on the native counter collection tool. Do not use --no-native-tool with --iteration-multiplexing.

  • Do not use --attach-pid with --iteration-multiplexing. Ensure that --attach-pid is not used in your profiling command.

  • Ensure that your workload runs for enough iterations to cover all counter subsets. When using iteration multiplexing, the total number of iterations for each unique kernel and launch parameters combination specified in the workload should be sufficient to cover all subsets of counters. If the number of iterations is too low, some counters may not be collected.

The following example demonstrates how to use iteration multiplexing with the vcopy workload:

$ rocprof-compute profile --name vcopy --iteration-multiplexing -- ./vcopy -i 20 -n 1048576 -b 256

                                 __                                       _
 _ __ ___   ___ _ __  _ __ ___  / _|       ___ ___  _ __ ___  _ __  _   _| |_ ___
| '__/ _ \ / __| '_ \| '__/ _ \| |_ _____ / __/ _ \| '_ ` _ \| '_ \| | | | __/ _ \
| | | (_) | (__| |_) | | | (_) |  _|_____| (_| (_) | | | | | | |_) | |_| | ||  __/
|_|  \___/ \___| .__/|_|  \___/|_|        \___\___/|_| |_| |_| .__/ \__,_|\__\___|
               |_|                                           |_|

INFO Rocprofiler-Compute version: 3.5.0
INFO Profiler choice: rocprofiler-sdk
INFO Path: /home/auser/rocprofiler-compute/workloads/vcopy_kernel/MI325X
INFO Target: MI325X
INFO Command: ./vcopy -i 20 -n 1048576 -b 256
INFO Kernel Selection: None
INFO Kernel Iteration Range: None
INFO Filtered sections: All
INFO
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO Collecting Performance Counters
INFO ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
INFO
INFO Using native counter collection tool: /tmp/rocprofiler-compute-tool-xxxxx/librocprofiler-compute-tool.so
INFO [profiling] Iteration multiplexing: Enabled
INFO [profiling] Current input files: .../perfmon/pmc_perf_0.yaml, .../perfmon/pmc_perf_1.yaml, ...
INFO    |-> [rocprofiler-sdk] [rocprofiler-compute] [rocprofiler_configure] (priority=1) is using rocprofiler-sdk v1.1.0 (1.1.0)
INFO    |-> [rocprofiler-sdk] [rocprofiler-compute] In tool init
INFO    |-> [rocprofiler-sdk] vcopy testing on GCD 0
INFO    |-> [rocprofiler-sdk] Finished allocating vectors on the CPU
INFO    |-> [rocprofiler-sdk] Finished allocating vectors on the GPU
INFO    |-> [rocprofiler-sdk] ...
INFO    |-> [rocprofiler-sdk] [rocprofiler-compute] In tool fini
...

Caveats#

Iteration multiplexing feature comes with some caveats to be considered when profiling any workload:

  • Accuracy vs speed trade-off

    Iteration multiplexing provides a trade-off with decreased profiling time by eliminating application replay while sacrificing accuracy since only a handful of counters can be collected per kernel dispatch; while we test for closeness in metric values with and without iteration multiplexing in our automatic test suite, more accurate results can be obtained by not using iteration multiplexing.

  • Minimum number of kernel dispatches required

    When using iteration multiplexing it is recommended to filter by kernel(s) of interest using -k (see Kernel filtering) and make sure these kernels are dispatched enough times (50 recommended) to cover all counter subsets (currently around 15).

  • Kernels with missing counter data are excluded from metrics

    If a kernel does not have enough dispatches to cover all counter sets, its counter data cannot be fully imputed and is excluded from metrics calculations. A warning at analysis time lists any affected kernels. Their execution times remain visible in the Top Stats section so their runtime impact is still apparent. To get more complete kernel coverage for metrics calculations, you may consider disabling iteration multiplexing to use application replay, or increasing the number of iterations for these kernels in the workload.

  • Non-deterministic workloads

    Workloads which dispatch kernels with non-deterministic names and launch parameters may trigger warnings for insufficient dispatch counts because iteration multiplexing identifies unique kernels by their names and launch parameters; this is especially true of large AI workloads that dispatch kernels non-deterministically based on the model layers being used for the current input, and in such cases kernel filtering (-k, see Kernel filtering) of common kernels is recommended.

Multi-rank profiling#

When profiling MPI workloads, ROCm Compute Profiler can isolate outputs by rank. If a rank is detected and no rank placeholder is provided, each rank writes to a subdirectory named by its rank to avoid output collisions.

Example usage#

Some examples of using multi-rank profiling are:

  • With --output-directory option:

$ mpirun -n 4 rocprof-compute profile --output-directory /tmp/mpi_profile -- ./laplace_eqn -n 1048576 -b 256

The example above produces:

$ ls /tmp/mpi_profile
0  1  2  3

$ tree /tmp/mpi_profile/0

└── MI325X
 ├── log.txt
 ├── perfmon
 │   ├── counter_def_0.yaml
 │   ├── counter_def_1.yaml
 │   ├── counter_def_2.yaml
 │   ├── counter_def_3.yaml
 │   ├── pmc_perf_0.yaml
 │   ├── pmc_perf_1.yaml
 │   ├── pmc_perf_10.yaml
 │   ├── pmc_perf_11.yaml
 │   ├── pmc_perf_12.yaml
 │   ├── pmc_perf_13.yaml
 │   ├── pmc_perf_2.yaml
 │   ├── pmc_perf_3.yaml
 │   ├── pmc_perf_4.yaml
 │   ├── pmc_perf_5.yaml
 │   ├── pmc_perf_6.yaml
 │   ├── pmc_perf_7.yaml
 │   ├── pmc_perf_8.yaml
 │   └── pmc_perf_9.yaml
 ├── pmc_perf_0.csv
 ├── pmc_perf_1.csv
 ├── pmc_perf_2.csv
 ├── profiling_config.yaml
 ├── roofline.csv
 └── sysinfo.csv
  • With --name option:

$ mpirun -n 4 rocprof-compute profile --name laplace -- ./laplace_eqn -n 1048576 -b 256

The example above produces:

$ ls ./workloads/laplace_eqn
0  1  2  3

$ tree ./workloads/laplace_eqn/0

└── MI325X
 ├── log.txt
 ├── perfmon
 │   ├── counter_def_0.yaml
 │   ├── counter_def_1.yaml
 │   ├── counter_def_2.yaml
 │   ├── counter_def_3.yaml
 │   ├── pmc_perf_0.yaml
 │   ├── pmc_perf_1.yaml
 │   ├── pmc_perf_10.yaml
 │   ├── pmc_perf_11.yaml
 │   ├── pmc_perf_12.yaml
 │   ├── pmc_perf_13.yaml
 │   ├── pmc_perf_2.yaml
 │   ├── pmc_perf_3.yaml
 │   ├── pmc_perf_4.yaml
 │   ├── pmc_perf_5.yaml
 │   ├── pmc_perf_6.yaml
 │   ├── pmc_perf_7.yaml
 │   ├── pmc_perf_8.yaml
 │   └── pmc_perf_9.yaml
 ├── pmc_perf_0.csv
 ├── pmc_perf_1.csv
 ├── pmc_perf_2.csv
 ├── profiling_config.yaml
 ├── roofline.csv
 └── sysinfo.csv

To control output placement explicitly, add %rank% (and other placeholders) to your output directory. The following example is run on the host amd-ryzen:

$ mpirun -n 4 rocprof-compute profile --output-directory /tmp/mpi_profile/%hostname%/%rank% -- ./laplace_eqn -n 1048576 -b 256

$ ls /tmp/mpi_profile/amd-ryzen/
0  1  2  3

$ tree /tmp/mpi_profile/amd-ryzen/0

└── MI325X
 ├── log.txt
 ├── perfmon
 │   ├── counter_def_0.yaml
 │   ├── counter_def_1.yaml
 │   ├── counter_def_2.yaml
 │   ├── counter_def_3.yaml
 │   ├── pmc_perf_0.yaml
 │   ├── pmc_perf_1.yaml
 │   ├── pmc_perf_10.yaml
 │   ├── pmc_perf_11.yaml
 │   ├── pmc_perf_12.yaml
 │   ├── pmc_perf_13.yaml
 │   ├── pmc_perf_2.yaml
 │   ├── pmc_perf_3.yaml
 │   ├── pmc_perf_4.yaml
 │   ├── pmc_perf_5.yaml
 │   ├── pmc_perf_6.yaml
 │   ├── pmc_perf_7.yaml
 │   ├── pmc_perf_8.yaml
 │   └── pmc_perf_9.yaml
 ├── pmc_perf_0.csv
 ├── pmc_perf_1.csv
 ├── pmc_perf_2.csv
 ├── profiling_config.yaml
 ├── roofline.csv
 └── sysinfo.csv

ROCm Compute Profiler supports the following libraries, APIs, and job schedulers:

  • OpenMPI

  • MPICH

  • MVAPICH2

  • Slurm

  • Flux Core

  • PMI

  • PMIx

  • PALS

For other MPI implementations or job schedulers, use the %env{NAME}% placeholder to include environment variables that identify the rank. For example, if your MPI implementation sets the MY_MPI_RANK environment variable, you can specify the output directory as:

$ mpirun -n 4 rocprof-compute profile --output-directory /tmp/mpi_profile/%env{MY_MPI_RANK}% -- ./my_mpi_application

Caveats#

When profiling multi-rank applications, be aware of the following caveats:

MPI launcher placement:

MPI launchers (mpirun, mpiexec, srun, and orterun) must wrap the rocprof-compute command, not appear after --. The following is incorrect:

$ rocprof-compute profile --name my_app -- mpirun -n 4 ./my_application   # WRONG

Instead, use the correct form where the MPI launcher wraps rocprof-compute:

$ mpirun -n 4 rocprof-compute profile --name my_app -- ./my_application   # CORRECT

If you use an MPI launcher after --, an error will be raised with guidance on the correct usage.

Application replay mode (default):

By default, ROCm Compute Profiler uses application replay mode, which runs the workload multiple times to collect all performance counters. This mode fails for MPI applications because running the application multiple times results in multiple MPI_Init and MPI_Finalize calls, which is not permitted by the MPI specification.

Recommended single-pass modes:

For multi-rank applications with MPI communication, use one of these single-pass profiling modes:

  • --iteration-multiplexing: Collects all counters in a single application run by distributing counter collection across kernel dispatches. Recommended for applications with sufficient kernel dispatch counts.

$ mpirun -n 4 rocprof-compute profile --name my_mpi_app --iteration-multiplexing -- ./my_mpi_app
  • --set <name>: Profiles a predefined counter set that fits in a single pass.

$ mpirun -n 4 rocprof-compute profile --name my_mpi_app --set compute_thruput_util -- ./my_mpi_app

Multi-node profiling:

When profiling across multiple nodes, ensure that:

  • Output directories are accessible from all nodes (shared filesystem), or

  • Use node-specific output directories with %hostname% placeholder.

$ mpirun -n 8 --hostfile hosts.txt rocprof-compute profile \
    --output-directory /shared/profiles/%hostname%/%rank% -- ./my_mpi_app