Dynamic process attachment using rocprofv3#
For profiling long-running applications or services where restarting the application is not feasible, rocprofv3 provides dynamic process attachment using the --attach option. This feature facilitates attaching the profiler to a running application without the need to restart it. The attachment is performed using the ptrace system call, which enables the profiler to monitor and collect performance data from the target process.
Prerequisites#
Complete the following steps before using rocprofv3 --attach.
Enable attachment support in the target process
Start the target process with
ROCP_TOOL_ATTACH=1set in its environment. For other ways to enable attachment, see enabling attachment support.Obtain permission to trace the target process
Run
rocprofv3as the user owning the target process or as root. On distributions that enable the Yama security module, including recent Ubuntu releases, also set the ptrace scope to0. For Docker requirements and other restrictions, see trace permissions.$ sudo sysctl kernel.yama.ptrace_scope=0
Warning
kernel.yama.ptrace_scopeis a system-wide setting. Setting it to0allows any process toptraceany other process running under the same user, not justrocprofv3.
Basic usage#
Here is an example syntax for dynamic process attachment:
rocprofv3 --attach <PID> [--hip-trace] [--output-format <format>]
Here are the options used in the preceding example:
<PID>: Process ID of the target application.--hip-trace: This optional flag enables HIP API tracing.--output-format: The desired output format such as rocpd, csv, or json.
Note
In process-attachment mode, rocprofv3 may generate output files asynchronously during detachment. As a result, output files might not be fully written immediately when rocprofv3 returns. If your workflow needs output files to be complete before continuing, for example, if a script processes or removes the output directory right after detach, use --attach-sync-output. This makes detach wait for output generation to finish, which can increase detach time.
Basic attachment syntax:
rocprofv3 -p <PID> <tracing_options>
# or
rocprofv3 --pid <PID> <tracing_options>
# or
rocprofv3 --attach <PID> <tracing_options>
Basic dynamic process attachment#
Follow these steps to attach the profiler to a running process and profile:
Start the target application in the background:
./myapp -n 1 &
Get the process ID (PID) of the running application:
echo $(pgrep myapp) OR ps aux | grep myapp
Attach
rocprofv3to the running application:rocprofv3 --attach <PID> --hip-trace --output-format rocpd
Detach the profiler when done:
To detach the profiler from the target application, press “Enter” in the terminal where
rocprofv3is running. You can also send SIGINT (Ctrl+C) torocprofv3to detach from the target.The profiling data will be saved in the format specified using
output-format.
Duration-specific dynamic process attachment#
Follow these steps to attach the profiler to a running process and profile for a specific duration such as 5 seconds:
Start the target application in the background:
./myapp -n 1 &
Get the process ID (PID) of the running application:
echo $(pgrep myapp) OR ps aux | grep myapp
Attach
rocprofv3to the running application:rocprofv3 --attach <PID> --attach-duration-msec 5000 --sys-trace --output-format csv
The profiler will automatically detach after the specified duration (5 seconds in this case). Note that the duration is to be specified in milliseconds (ms).
The profiling data will be saved in the format specified using
output-format. For example, if you specify--output-format csv, the data will be saved as a CSV file.
Dynamic process attachment with counter collection#
The dynamic process attachment functionality works with all tracing and profiling options available in rocprofv3, providing the same comprehensive analysis capabilities as standard application launching.
The following example attaches the profiler to a process with PID “12345”, collects counters SQ_WAVES and GRBM_COUNT, and saves profiling data in a CSV file:
rocprofv3 --pid 12345 --pmc SQ_WAVES GRBM_COUNT --output-format csv
Reattaching to the same process#
The dynamic process attachment functionality supports reattachment, allowing attaching multiple times to the same PID over a process’s lifetime. You need to provide the same PID to rocprofv3 to reattach.
There are some restrictions on what the options are allowed to change when reattaching. Typically, tracing, PC sampling, ATT, counter collection, and other options that decide the data to be collected can’t be changed. rocprofv3 throws a RuntimeError if it detects a configuration change that isn’t supported.
By default, the output file generation runs asynchronously after detachment, allowing for faster tool detachment. This implies that the output files might not be immediately available when rocprofv3 exits. If the output file generation from the previous attachment is still in progress, rocprofv3 blocks reattachment until the ongoing output generation completes.
- class details#
Full list of options that mustn’t change:
ALL options ending with
traceALL options starting with
pc_samplingALL options starting with
attpmcpmc_groupsoutput_configextra_countersselected_regionsselected_regions_ref_countkernel_include_regexkernel_exclude_regexkernel_iteration_range
Synchronous output generation for scripts#
For use cases requiring output files to be fully written before detachment completes, such as scripts that process or delete output directories immediately after detachment, you can enable synchronous output generation using the --attach-sync-output flag. This causes tool_detach to wait for all output files to be written before returning, ensuring output files are complete when the rocprofv3 process exits.
For example, consider a script that attaches for a fixed duration and then terminates the workload as soon as rocprofv3 returns:
# Start the workload in the background
./myapp &
WL_PID=$!
# Attach for 5 seconds, then detach
rocprofv3 --pid "$WL_PID" --attach-duration-msec 5000 \
--output-format csv -o profile -d "$PWD"
# Workload is killed immediately after rocprofv3 returns
kill "$WL_PID"
With the default asynchronous output generation, the output thread runs inside the target process. Killing the workload right after rocprofv3 returns can terminate that thread before it finishes writing, producing truncated or incomplete output files. Add --attach-sync-output so that detachment waits for output generation to finish before returning:
./myapp &
WL_PID=$!
rocprofv3 --pid "$WL_PID" --attach-duration-msec 5000 \
--attach-sync-output \
--output-format csv -o profile -d "$PWD"
# Safe: output files are fully written before rocprofv3 returns
kill "$WL_PID"
Attaching to a process tree#
By default, rocprofv3 --attach attaches to the target process and all of its descendant processes. This is useful for profiling applications that spawn child processes, such as multiprocess MPI jobs or launchers that fork workers.
To attach to a PID and all its children (default behavior), use:
rocprofv3 --attach 1234 --hip-trace
To attach only to the specified PID and skip its descendants, use --attach-children=false:
rocprofv3 --attach 1234 --attach-children=false --hip-trace
The child process tree is enumerated once at attach time using /proc. Processes that are spawned after the attachment has begun are not automatically profiled.
Every process in the tree is profiled with the tool library selected for the target PID. For more information, see Attaching to a process from a different ROCm installation.
Attaching to a process from a different ROCm installation#
The profiling tool library is loaded into the target process, where it runs with the rocprofiler-sdk library that the target already uses. By default, rocprofv3 --attach (and rocprof-attach) selects the tool library as follows:
It reads
/proc/<PID>/mapsto find the directory of the rocprofiler-sdk libraries that the target process has loaded, and uses the tool library of that installation,<libdir>/rocprofiler-sdk/librocprofiler-sdk-tool.so. The file is looked up in the target’s filesystem, through/proc/<PID>/root, so this also works for processes running in a container.If the target has not loaded rocprofiler-sdk, or its installation’s tool library cannot be found, the tool library of the
rocprofv3installation is used instead. A warning is printed when the target has loaded rocprofiler-sdk but its tool library was not found.
When attaching to a process tree, the selection is made once for the target PID and used for every process in the tree.
To attach a specific tool library instead, set ROCPROF_ATTACH_TOOL_LIBRARY, or pass -t/--attach-tool-library to rocprof-attach. The value is a colon-delimited list, is used exactly as given, and is interpreted in the target’s filesystem. Attachment fails if any library in the list cannot be found or loaded, and the target keeps running without profiling. ROCP_TOOL_LIBRARIES is ignored in attach mode, with a warning. A tool library that does not implement rocprofiler_configure_attach is configured by the first attach, but it cannot be detached, and later attaches to the same process fail.
Option support across versions
When the target runs a different rocprofiler-sdk installation than rocprofv3, a warning is printed, because the requested options might not all be supported by the target’s version:
With the target’s own tool library, options that library does not know are ignored. Aggregate options such as
--sys-trace,--runtime-trace,--hip-trace,--hsa-trace, and--kfd-traceonly enable the tracing that the target’s version supports.With the
rocprofv3installation’s tool library, tracing kinds that the target’s rocprofiler-sdk does not support are skipped, with a warning in the target process’s log. Thread trace (--att) is disabled with a warning when the target’s rocprofiler-sdk is older than 1.5.0. Prefer individual tracing options over aggregate options, and request only options that the target’s version supports.
To use every option that the target’s version supports, run the rocprofv3 from the target’s own installation.
Attachment support in the target’s version
Attachment requires the rocp-bg-attach thread, which the rocprofiler-sdk attach library starts in the target process when attachment is enabled. If the target’s attach library is too old to start this thread, rocprofv3 --attach-children=false reports an error before touching the process. Use the rocprofv3 from the target’s own installation to profile such a process. When attaching to a process tree, such processes are skipped and their descendants are still attached.
Key considerations#
Here are some important points to be noted while using dynamic process attachment:
The target process must be running and actively using GPU resources for meaningful profiling data.
The profiler collects data for the entire remaining lifetime of the process or until the configured collection period expires. To learn how to configure the collection period, see Duration-specific dynamic process attachment.
When attaching to a process tree, if attachment to an individual child process fails (for example, because it exited between enumeration and attach), the error is logged and attachment continues with the remaining processes.