Recording and replaying HIP workloads#

HIP Record & Replay (HRR) records the HIP API calls an application makes, together with the host buffers and code objects it needs, into an archive that can be replayed on a GPU later. The replay does not need the original application, its source, or its input data. The archive holds the workload’s code and data instead, so it is confidential: share one only as agreed, after reading the NOTICE.txt that capture writes into it (see Sharing an archive).

HRR has two halves:

  • Capture is part of the HIP runtime library. It is off unless you ask for it with an environment variable, and records only once you have consented to it.

  • Replay is a separate developer tool, hrr-playback, installed into the ROCm binary directory by a ROCm build that includes the HRR project.

This page takes you from an installed ROCm to a completed replay. For the archive format, the full flag list and the known limitations, see the HRR README and DESIGN.md next to it in the ROCm sources.

Getting the replay tool#

Look in the ROCm binary directory, and ask the tool what it is:

"${ROCM_PATH:-/opt/rocm}/bin/hrr-playback" --version

--version needs neither an archive nor a GPU. It reports the archive format version this build reads, the revision it was built from, and the HIP runtime it loaded. Those three facts identify a replay, so quote them in any report about one.

A ROCm container image that carries the tool is the shortest route when you have been handed an archive and want to open it without touching the host:

docker run --rm --device=/dev/kfd --device=/dev/dri \
  -v "$PWD/my_capture.hrr:/capture:ro" \
  <rocm-image> hrr-playback /capture --info

Note

The tool loads the HIP runtime, the compiler libraries and the ROCm system dependency libraries at run time. Installing ROCm through its packages pulls all of them in. Unpacking the runtime package on its own is not enough, and fails at start-up on a missing library.

If your installation does not carry the tool, the HRR README covers building it from the ROCm sources, and the rule that comes with it: a tool built from the sources reads archives recorded by the HIP runtime built from the same commit.

Recording a workload#

Set HIP_HRR_CAPTURE_OUTPUT to the archive you want written, and run the application as usual:

HIP_HRR_CAPTURE_OUTPUT=./my_capture.hrr ./my_hip_app

Capture is off whenever the variable is unset, so nothing changes for an ordinary run. It also needs your consent, given in one of two ways:

  • Once per user, with the line yes in the file ${XDG_CONFIG_HOME:-$HOME/.config}/hrr/capture-consent. The file covers that user on every host that shares the home directory.

  • With HIP_HRR_CAPTURE_CONSENT=yes, where no such file can be kept, as in CI or a container. The variable applies to every process that inherits it, child processes and MPI ranks included. Any other value, such as no, withholds consent even when the file says yes. An empty value counts as unset, so the file still decides: to turn capture off, unset HIP_HRR_CAPTURE_OUTPUT or set HIP_HRR_CAPTURE_CONSENT=no.

Without either, the application runs unrecorded and says why in one line on stderr. Whoever controls your environment or configuration directory can give consent for you. The file counts only if it is a regular file you own, not reached through a symbolic link and with no other hard link.

d="${XDG_CONFIG_HOME:-$HOME/.config}/hrr" && mkdir -p "$d" && echo yes > "$d/capture-consent"

The archive is a directory, not a single file, and it holds one pid-<pid>/ subdirectory per recorded process:

my_capture.hrr/
  manifest.json       which processes the archive holds
  pid-<pid>/
    manifest.json     what this process recorded, and whether it ended cleanly
    events.bin        the recorded call sequence
    blobs/            host buffers, addressed by content hash
    code_objects/     the GPU code the run used
    regions/          optional, and written by tooling outside the runtime:
                      memory a framework allocated without going through HIP

Expect the result to be large. It contains the buffers the workload moved, so a short run of a big model produces gigabytes.

Sharing an archive#

Warning

Capture writes the notice below to NOTICE.txt at the archive root. Read it before an archive leaves the machine.

HIP Record & Replay capture archive: confidential, read before sharing

This archive is a record of a process's GPU work. It can contain:
- host buffers copied to or from the device, and host memory registered with
  hipHostRegister: model weights, prompts, inputs and outputs, passwords, keys
  and session tokens, and any personal, customer or third-party data among
  them;
- code objects, fat binary bundles and linker input buffers (the
  application's GPU binary code) and kernel names, which can disclose
  proprietary software;
- kernel argument bytes, other structures passed into HIP calls, and strings
  passed to HIP such as file paths, device names and process ids.
HRR does not intentionally collect user names, host names, command-line
arguments or environment variables as dedicated metadata. However, these
values may still appear in captured memory or other recorded content. HRR does
not encrypt, redact or anonymise anything. File permissions protect the
archive only on the machine that wrote it.

Treat this archive like the most sensitive code and data the process handled.
- Do not post it publicly: not to public repositories, public issues (GitHub
  issues are public), public storage, or CI artefacts readable beyond the
  authorised recipients. Store it only in approved systems with appropriate
  access controls and encryption.
- Share it only with recipients specifically authorised to receive that code
  and data, through an approved secure transfer mechanism, encrypted in
  transit and at rest, and on terms agreed in advance, such as a
  non-disclosure agreement, plus a data processing agreement if it holds
  personal data.
- Before sending, check which calls and kernels it holds with hrr-playback
  --info, and encrypt it for the recipient. If they need a key or passphrase
  to decrypt it, send that separately, through another approved secure
  channel.
- Delete copies you no longer need. If an archive leaks, change every
  password, key and token the process may have held.

You decide what to record and whom to share it with. You are responsible for
having the right to share everything in it, including third-party model
weights and code under their own licences (some require a licence copy and
notice with every transfer), and for data-protection, export and sanctions
rules when it crosses borders.

Replay runs the recorded GPU code on your GPU, with your user's rights: replay
an archive only if you trust whoever produced it.

What HRR is responsible for, and what it leaves to you, is listed under Scope
in projects/hrr/PRIVACY.md of the ROCm source:
https://github.com/ROCm/rocm-systems/blob/develop/projects/hrr/PRIVACY.md

HRR is provided "as is", without warranty of any kind, as stated in its
licence. You, not AMD, are responsible for the content of archives you create
and for how they are shared. Report security issues in HRR to psirt@amd.com.

The HRR sources explain what the notice covers in Sharing an archive.

Your first replay#

Start with a summary of what the archive holds. This reads the archive only, and needs no GPU:

hrr-playback ./my_capture.hrr --info

Pointed at the archive directory, --info lists the recorded processes:

HRR Archive Root: ./my_capture.hrr
========================================
Capture Mode: in-tree
Owner PID:    1180
Processes:    2

  PID          Parent PID   Complete   Events       Blobs      Path
  ---          ----------   --------   ------       -----      ----
  1180         0            yes        412          6          ./my_capture.hrr/pid-1180
  1204         1180         NO         918274       1503       ./my_capture.hrr/pid-1204

Complete reports whether that process shut down cleanly. An archive from a run that crashed is still usable: the reader recovers every complete record. To leave it in a tidy state, hrr-playback ./my_capture.hrr --repair rewrites every process capture under the archive with a clean trailer and rebuilds the index.

Point the tool at one pid-<pid>/ directory for the detail: the event and kernel breakdown, and with --events the whole recorded sequence.

hrr-playback ./my_capture.hrr/pid-1204/ --info

Then replay it on a GPU:

hrr-playback ./my_capture.hrr/pid-1204/

An archive holding a single process can be replayed by naming the archive directory itself; the tool resolves it. With more than one recorded process it lists them and asks you to choose, because there is no sensible default.

Replay reissues the recorded calls in order. Where the original run copied data back from the device, HRR compares what the GPU produces now against what was recorded, and finishes with a summary:

[HRR] Archive : 918274 events, 1180 kernels, 1503 blobs, 3 code objects
[HRR] Threads : 4 captured
[HRR] Device  : AMD Instinct MI350X (gfx950:sramecc+:xnack-)
[HRR] Runtime : /opt/rocm/lib/libamdhip64.so
[HRR] Mode    : single-threaded
[HRR] -- Replay summary ------------------------------
[HRR]   Wall time      : 8421.3 ms
[HRR]   Threads used   : 1
[HRR]   Kernels launched: 1180
[HRR]   Graphs launched : 0
[HRR]   D2H checks     : 14 pass (12 exact, 2 within tol), 0 fail, 0 skipped
[HRR] PASS

The exit code is zero when every comparison passed.

When a replay faults or hangs, --sync-after-launch makes the tool wait for the GPU after every kernel, so the failure stops replay on the kernel that caused it and that kernel is named. --sync-watchdog-ms gives up on a hang after a time limit instead of blocking. The README covers the rest of the flags.

Matching the archive to the tool#

The archive format carries a version number, and the reader accepts only its own version, which --version reports. An archive and a hrr-playback from different ROCm releases therefore do not necessarily work together, in either direction. The mismatch is reported with both numbers:

[HRR] Version mismatch in ./my_capture.hrr/pid-1204/events.bin: file=4 reader=5

Read this as: the archive is written in format version 4, and this tool reads version 5. The fix is to replay with a hrr-playback from the same ROCm release that produced the archive, rather than to convert the archive.

The number changes when the recorded layout changes, which since version 5 no longer happens merely because ROCm gained an API. So when you ask somebody for an archive, ask which ROCm version produced it.

Recording and replaying on the same machine and the same ROCm installation always matches, because both halves ship together.