ROCm Core SDK 10.2.0 release notes#
2026-11-18
These release notes describe notable changes since the previous ROCm release.
Note
Since ROCm 7.14, ROCm uses TheRock as its build and release system. For more information, see the transition guide.
Release highlights#
This release adds platform, framework, runtime, profiling, and library enhancements across AMD Instinct™, Radeon™, and Ryzen™ platforms. It expands support for AMD Ryzen processors with integrated Radeon graphics and selected GPU virtualization configurations, and updates support for PyTorch, JAX, and vLLM. Profiling tools add device-wide Streaming Performance Monitor (SPM) counter sampling, a unified rocsys command for ROCm Systems Profiler, and improved memory analysis in ROCm Compute Profiler. HIP adds new module, library, and kernel APIs. Library updates include complex GEMM support in hipBLASLt and new and improved sparse routines in rocSPARSE.
Platform and hardware support#
This release adds support for additional AMD Ryzen APUs and new GPU virtualization configurations.
Expanded AMD GPU support#
ROCm 10.2.0 adds support for the following AMD APUs and GPUs:
For the complete list of supported AMD hardware, see AMD hardware support.
Operating system support update#
ROCm 10.2.0 adds support for:
RHEL 10.3 for AMD Instinct and Radeon GPUs.
RHEL 10.3 replaces RHEL 10.2 as the validated RHEL 10 release for AMD Radeon GPUs.
RHEL 9.9 for AMD Instinct and Radeon GPUs.
RHEL 9.9 replaces RHEL 9.8 as the validated RHEL 9 release for AMD Radeon GPUs.
SLES 16.1 for AMD Instinct GPUs.
For the full list of supported Linux distributions, see Operating system support.
Installation updates#
ROCm 10.2.0 introduces several improvements to the Runfile Installer:
Added support for HPC SDK (hipTensor, rocALUTION) component installation.
Added support for new Linux distributions: SLES 16.1, RHEL 9.9, and RHEL 10.3.
Improved installation of installer-specific dependencies.
Fixed additional dependency installation issues for SLES.
Added support for various Ryzen APUs.
Improved multi-GPU detection.
Minor bug fixes.
Expanded GPU virtualization support for Instinct and Radeon GPUs#
ROCm 10.2.0 adds support for the following virtualization configurations on AMD Instinct and Radeon GPUs:
Instinct
On AMD Instinct MI350P:
KVM SR-IOV with Ubuntu 24.04 host OS and Ubuntu 24.04 guest OS.
Radeon
On AMD Radeon PRO W7800, W7900, W7900D, and W7900 Dual Slot:
KVM Passthrough with Ubuntu 24.04 or Ubuntu 26.04 host OS and Ubuntu 24.04 or Ubuntu 26.04 guest OS.
Supported Single Root I/O Virtualization (SR-IOV) configurations require the AMD GPU Virtualization Driver (GIM) 9.4.0.K. For details, see GPU virtualization support.
GPU partitioning support update#
GPU partitioning support remains unchanged in this release. For details, see GPU partitioning support.
AI inference and frameworks#
This release enables support for the following frameworks:
PyTorch 2.15.0
JAX 0.11.2
vLLM 0.32.0
SGLang 0.5.21
The updated framework support replaces the previous PyTorch 2.12.0, JAX 0.11.0, and vLLM 0.29.0 support.
For details, see AI ecosystem support.
Runtimes and compilers#
HIP feature highlights#
The following are notable enhancements to HIP. For the full list, see the HIP changelog.
Module, library, and kernel APIs added#
HIP adds new APIs to give you more control over modules, libraries, and kernels:
hipModuleEnumerateFunctionsreturns the function handles defined in a loaded module, so applications no longer need to know kernel names ahead of time.hipLibraryGetModulereturns the module handle associated with a library.hipKernelSetAttributeForDevicesets a device-wide attribute for a kernel on a specific device.
Profiling and debugging tools#
ROCprofiler-SDK feature highlights#
The following are notable changes to ROCprofiler-SDK and rocprofv3. For the full list, see the ROCprofiler-SDK changelog.
Device-wide counter sampling added#
Device-wide sampling lets you observe GPU counter behavior throughout a run, even when no kernels are running. The new device counting mode for the Streaming Performance Monitor (SPM) samples counters on a GPU for as long as profiling is active. SPM now supports gfx1200, gfx1201, and gfx1250, and on those GPUs you can set the sample interval in reference-clock cycles.
Thread trace reliability and timestamps improved#
Thread trace is more reliable for applications that use many GPUs or processes. It now creates queues only on GPUs that load a code object, and its timestamps line up with API traces. SQTT markers and the rocprof-trace-decoder Python API now ship with ROCm, so you no longer need to build them from source.
Other enhancements#
Improved PC sampling accuracy so fewer samples are dropped or invalid. PC sampling now uses a larger sample buffer on gfx94x, gfx95x, and gfx125x, and
rocprofv3filters out invalid stochastic PC samples.Added support for running your own OpenMP Tools (OMPT) tool alongside ROCprofiler-SDK.
ROCm Compute Viewer updates#
ROCm Compute Viewer (rocprof-compute-viewer) is developed in its own repository and versioned separately from ROCprofiler-SDK. This release adds SPM data support for gfx1200, gfx1201, and gfx1250 traces, including derived counters. The ISA view gains color coding by instruction category, resizable and hideable columns, and optional line numbers. For details about the ISA view, see Views and controls.
ROCm Compute Profiler feature highlights#
The following are notable changes to ROCm Compute Profiler (rocprofiler-compute). This release includes a breaking change to command-line options. For the full list, see the ROCm Compute Profiler changelog.
Command-line options aligned with rocprofv3 (breaking change)#
The dispatch filter and output directory options in profile mode now match rocprofv3. If your scripts select dispatches with -d or --dispatch, update them to use --kernel-iteration-range. -d now sets the output directory, and dispatch IDs now start at 1. For details, see Kernel iteration filtering.
Memory chart redesigned and bandwidth analysis added#
The memory chart shows more traffic detail on AMD Instinct GPUs and AMD Ryzen AI APUs (gfx115x), and MI350 Series GPUs get experimental bandwidth bottleneck analysis.
Redesigned the memory chart for AMD Instinct GPUs. It breaks down read, write, and atomic requests and shows L2-to-fabric bandwidth. On MI300 Series and earlier GPUs, it also shows the percentage of traffic to high-bandwidth memory (HBM) and remote memory. On MI350 Series GPUs, it shows Local Data Share (LDS) activity and per-channel HBM, xGMI, and PCIe bandwidth.
Improved the memory chart for AMD Ryzen AI APUs (gfx115x) with per-interface traffic, LDS utilization, renamed metrics, and updated arrows, labels, and legend.
Added experimental memory bandwidth analysis on MI350 Series GPUs. It marks bottlenecks at the L1 cache, L2 cache, or memory fabric on the chart and explains each one. Collect the data with
--experimental --membw-analysis. For details, see Using memory bandwidth analysis.
Roofline supports MXFP8#
Roofline analysis now covers MXFP8 and lets you compare precisions and runs side by side.
Added MXFP8 roofline support on AMD Instinct MI350 Series GPUs, for both peak measurement and kernel analysis.
Improved the standalone roofline HTML report to include every datatype the GPU supports in one report, with fixed axes so you can compare runs side by side.
For details, see Standalone roofline and Roofline HTML generation.
Other enhancements#
Added a single install command for analyze mode dependencies,
pip install "rocm-profiler[compute-analyze]". Userocprof-compute analyze --verify-depsto check that nothing is missing. For details, see Install the analyze mode dependencies.Added two PC sampling metrics for each instruction: active thread percent, which points to branch divergence, and wave occupancy, which requires stochastic sampling. For details, see PC sampling.
Deprecations#
Deprecated the analyze mode
--guiand--tuioptions. They will be removed in a future release. Use command-line analysis instead, which supports text, CSV, and analysis database output. For details, see Analysis output formats.
ROCm Systems Profiler feature highlights#
The following are notable changes to ROCm Systems Profiler. For the full list, see the ROCm Systems Profiler changelog.
rocsys unified command added#
rocsys is a single entry point for all ROCm Systems Profiler tools, so you no longer need to call each rocprof-sys-* binary separately. The existing binaries are unchanged.
Run
rocsys -- ./apporrocsys profile -- ./appfor a full trace profile.Run
rocsys rewriteto instrument a binary in binary-rewrite mode.Use the
instrument,causal,avail,python, andattachsubcommands to run the existing tools.
For details, see Using rocsys.
Other enhancements#
Added AMD Infinity Storage technology I/O telemetry. Per-GPU I/O counters are sampled into the profiler output. Enable it with
ROCPROFSYS_USE_HIPFILE. For details, see hipFile telemetry.Added support for
rocprof-sys-attachto attach to processes running in containers. For details, see Attaching to a running process.
Control and monitoring tools#
AMD SMI feature highlights#
The following are notable changes to AMD SMI. For the full list, see the AMD SMI changelog.
AMD SMI Python library on PyPI#
The AMD SMI Python library is now on PyPI as amdsmi 27.2.0; you can install it
using pip install amdsmi. This PyPI package bundles the AMD SMI library, so using
the Python API doesn’t require a ROCm installation. It provides the Python library only;
the amd-smi command-line tool ships with ROCm. amdsmi releases on PyPI follow the
AMD SMI library version (27.2.0 as of ROCm 10.2), not the ROCm SDK version.
Requirements:
Linux x86_64 with glibc 2.27 or later
Python 3.6 or later and pip 20.3 or later (older pip versions install the unrelated legacy 7.0.2 release instead)
libdrm-amdgpu1(Ubuntu/Debian),libdrm(RHEL), orlibdrm_amdgpu1(SLES)
See https://pypi.org/project/amdsmi/ for more information.
Power profile management added#
You can now discover, activate, and configure the driver’s predefined power and performance profiles from both the C API and the CLI. AMD SMI calls them AMPP (AMD SMI power profile) and uses ampp in API and CLI names, for example amd-smi static --ampp. Profiles and fields are enumerated at runtime, so your tools pick up whatever the driver exposes without hard-coded profile lists.
Link topology API added for bare metal#
The new amdsmi_get_link_topology() API lets you query the link between two GPUs on bare-metal systems with one call, matching the existing host interface. Python and Rust bindings are included.
Fabric link events added to error reports#
amd-smi ras --cper now reports fabric link events alongside GPU errors in the same Common Platform Error Record (CPER) output, so you can see fabric faults and GPU errors together. The new amdsmi_get_fabric_cper_entries() API retrieves the fabric reliability, availability, and serviceability (RAS) CPER records programmatically.
Energy counter API supports AMD Radeon GPUs#
amdsmi_get_energy_count now returns valid data on supported AMD Radeon GPUs in bare-metal and pass-through environments, so you can track energy use and calculate average power draw. Previously, the API was available only on AMD Instinct GPUs.
CLI exit codes and output updated (breaking changes)#
The amd-smi CLI now returns more specific exit codes, a structured error format, and less N/A output, which can affect scripts that parse its output or check $?:
Exit codes: Exit codes now identify the failure type more precisely and match the
codeprinted in the error message.amd-smi rasalso exits non-zero when a CPER file fails to decode, where it previously exited0.setandresetcommands continue past a per-device failure instead of stopping.Error messages: Errors are now structured, and
--jsonand--csverrors gain anerror_typefield.--csvrows gain a third column, so parsers that expect two fields per row need updating.Metric output:
amd-smi metricomits fields that the GPU’s metrics table version can’t report, so human-readable output shows fewer N/A values. On AMD Instinct MI300X, MI325X, and MI355X GPUs,--powerno longer printsTHROTTLE_STATUS, and--usageno longer printsMM_ACTIVITY,VCN_ACTIVITY, orJPEG_ACTIVITY. Use--show-unsupportedto restore omitted fields. JSON and CSV output are unchanged.
For API names, CLI syntax, and exit code values, see the AMD SMI changelog.
Libraries#
hipBLASLt feature highlights#
The following are notable enhancements to hipBLASLt. For the full list, see the hipBLASLt changelog.
Complex GEMM support added#
Scientific and engineering applications can now run single-precision (CGEMM) and double-precision (ZGEMM) complex matrix multiplications on AMD Instinct MI350 Series and MI300 Series GPUs (gfx950 and gfx942). Support covers the HIP_C_32F and HIP_C_64F data types and transpose and conjugate-transpose combinations of the A and B matrices. Complex GEMM solutions are also available through rocBLAS.
Solution catalogs stay compressed on disk#
hipBLASLt solution catalogs now stay compressed on disk as .dat.zlib files, instead of being compressed only in the package. For example, the gfx942 catalogs use about 194 MB on disk instead of about 3.8 GB. A catalog is decompressed on first use, which can add initial overhead. Subsequent calls are unaffected. hipSPARSELt also loads catalogs in this format.
Sub-matrix offsets supported for batched GEMM#
General batched GEMM now accepts 64-bit element offsets, so you can run GEMM on a smaller sub-matrix of a larger pre-allocated matrix. This also improves performance because the device pointer array no longer needs to be copied back and forth between the host and the device. This is available on gfx942, gfx950, and gfx1250.
Matrix layout attribute: To apply an offset, set
HIPBLASLT_MATRIX_LAYOUT_OFFSETon the matrix layout.Requirements: Nonzero offsets require
HIPBLASLT_BATCH_MODE_POINTER_ARRAY. For non-batched and strided batched GEMM, apply the offset directly through the strided batch offset instead. A nonzero offset with the sub-byte MX typesHIP_R_6F_E2M3,HIP_R_6F_E3M2, orHIP_R_4F_E2M1returnsHIPBLAS_STATUS_NOT_SUPPORTED.Benchmarking:
hipblaslt-benchsupports the newbatch_offset_a,batch_offset_b,batch_offset_c, andbatch_offset_darguments.
hipFile feature highlights#
The following are notable enhancements to hipFile. For the full list, see the hipFile changelog.
Asynchronous I/O failover and new documentation#
hipFile adds failover for asynchronous fastpath I/O, a batch I/O example, and Stats API documentation.
Async fastpath failover: Asynchronous fastpath I/O now switches to the fallback I/O backend when the fastpath reports a device error (
ENODEVorEREMOTEIO), instead of failing the request. Synchronous and batch I/O already behaved this way.Batch round-trip example: A new
batch-roundtripexample shows how to use the batch I/O API.Stats API documentation: The new Statistics API reference covers how to read I/O statistics.
PyTorch adds AMD Infinity Storage hipFile support#
PyTorch for ROCm now supports the torch.cuda.gds file I/O APIs, including GdsFile, gds_register_buffer, and gds_deregister_buffer, through AMD Infinity Storage hipFile. On supported filesystems, data moves directly between storage and GPU memory instead of being staged through host memory. To check whether your PyTorch build includes this support, use torch.cuda.gds.is_available().
hipFile support added to fio and IOR#
hipFile support is now available in fio 3.43 and in the main branch of IOR. Both are open source storage benchmarking tools that you can use to evaluate storage performance.
rocFFT feature highlights#
The following are notable enhancements to rocFFT. For the full list, see the rocFFT changelog.
Just-in-time callbacks supported on multi-GPU transforms#
Just-in-time (JIT) load and store callbacks, set with rocfft_plan_description_set_load_callback and rocfft_plan_description_set_store_callback, can now be used on multi-GPU transforms that use fields.
Function pointer callbacks set with the deprecated rocfft_execution_info_set_load_callback and rocfft_execution_info_set_store_callback APIs can no longer be used on transforms that use fields. For these transforms, rocfft_execute returns rocfft_status_invalid_arg_value. Use the JIT callback APIs instead.
64-bit indexing selected automatically for large transforms#
64-bit index arithmetic is now selected automatically for transforms large enough to overflow 32-bit addressing, so very large transforms, including lengths greater than 2^32, can run. Smaller transforms continue to use 32-bit arithmetic.
rocSPARSE feature highlights#
The following are notable enhancements to rocSPARSE. For the full list, see the rocSPARSE changelog.
Strided batched sparse triangular solve supported#
rocsparse_sptrsm now supports strided batched execution, so you can solve independent sparse triangular systems with a single call instead of looping over the batch on the host.
Formats: Batched solves support CSR, CSC, and COO matrices.
Broadcast inputs: A zero batch stride lets you share the sparse matrix or the dense matrices across the batch.
Diagonal solves: Batched execution also supports diagonal-only solves.
Generic sort routine added#
The new rocsparse_spsort generic API sorts COO, CSR, and CSC matrices through one routine instead of the format-specific rocsparse_csrsort, rocsparse_cscsort, and rocsparse_coosort_by_* routines. You can sort in place or into a separate target matrix. The routine also supports uniform strided batches, including broadcasting one source matrix across the target batch.
Performance improvements#
Improved sparse triangular solve and incomplete factorization performance on AMD Instinct MI300 Series and MI350 Series GPUs (gfx942 and gfx950), in both the analysis and solve phases. This covers routines such as
rocsparse_spsv,rocsparse_spsm,rocsparse_spilu0, androcsparse_spic0. Matrices with long dependency chains see smaller gains.Improved SpGEMM performance for products with long, wide output rows. Workloads such as algebraic multigrid (AMG) setup in rocALUTION can particularly benefit.
HPC SDK highlights#
For more information about the HPC SDK, see AMD ROCm HPC SDK and Install ROCm HPC SDK.
New documentation#
hipTensor and rocALUTION now have standalone documentation sites:
rocALUTION feature highlights#
The following are notable enhancements to rocALUTION:
Ruge-Stueben interpolation and truncation added#
Ruge-Stueben algebraic multigrid (AMG) now supports the matrix-matrix-based interpolation operators MMExtPI and MMExtPE. To select one, use RugeStuebenAMG::SetInterpolationType(). To truncate the interpolation operator, use RugeStuebenAMG::SetInterpolationTruncationFactor() to drop entries that are small relative to the largest entry in their row, and RugeStuebenAMG::SetInterpolationMaxElmts() to cap the number of entries per row.
Depth limit for AMG hierarchies#
The new BaseAMG::SetMaxLevels() limits the number of levels in an AMG hierarchy. Combine it with another AMG solver as the coarse grid solver to build hybrid hierarchies, such as unsmoothed aggregation on the finest levels followed by Ruge-Stueben AMG on the coarser levels. The new cg-uaamg-rsamg sample demonstrates this configuration.
AMD hardware support#
The following table lists supported AMD Instinct GPUs, Radeon GPUs, and Ryzen APUs. Each supported device is listed with its corresponding GPU microarchitecture and LLVM target.
Note
If your GPU is not listed, it might be community-enabled through TheRock nightly builds. For more information, see TheRock supported GPUs. For installation guidance, see TheRock releases.
Device series |
Device |
LLVM target |
Architecture |
|---|---|---|---|
| AMD Instinct MI350 Series | gfx950 | CDNA 4 | |
| AMD Instinct MI300 Series | gfx942 | CDNA 3 | |
| AMD Instinct MI200 Series | gfx90a | CDNA 2 | |
| AMD Instinct MI100 Series | gfx908 | CDNA |
Device series |
Device |
LLVM target |
Architecture |
|---|---|---|---|
| AMD Radeon AI PRO R9000 Series | gfx1201 | RDNA 4 | |
| AMD Radeon RX 9000 Series | gfx1201 | ||
| gfx1200 | |||
| AMD Radeon PRO W7000 Series | gfx1100 | RDNA 3 | |
| gfx1101 | |||
| AMD Radeon RX 7000 Series | gfx1100 | ||
| gfx1101 | |||
| gfx1102 | |||
| AMD Radeon PRO V Series | gfx1101 | ||
| gfx1030 | RDNA 2 | ||
| AMD Radeon PRO W6000 Series | gfx1030 |
Device series |
Device |
LLVM target / codename |
Architecture |
|---|---|---|---|
| AMD Ryzen AI Max PRO 400 Series |
Ryzen AI Max+ PRO 495 (Radeon 8065S) Ryzen AI Max PRO 490 (Radeon 8050S) Ryzen AI Max PRO 485 (Radeon 8050S) |
gfx1151 Gorgon Halo |
RDNA 3.5 |
| AMD Ryzen AI Max PRO 300 Series |
Ryzen AI Max+ PRO 395 (Radeon 8060S) Ryzen AI Max PRO 390 (Radeon 8050S) Ryzen AI Max PRO 385 (Radeon 8050S) Ryzen AI Max PRO 380 (Radeon 8040S) |
gfx1151 Strix Halo |
|
| AMD Ryzen AI Max 300 Series |
Ryzen AI Max+ 395 (Radeon 8060S) Ryzen AI Max+ 392 (Radeon 8060S) Ryzen AI Max+ 388 (Radeon 8060S) Ryzen AI Max 390 (Radeon 8050S) Ryzen AI Max 385 (Radeon 8050S) |
gfx1151 Strix Halo |
|
| AMD Ryzen AI PRO 400 Series |
Ryzen AI 9 HX PRO 475 (Radeon 890M) Ryzen AI 9 HX PRO 470 (Radeon 890M) Ryzen AI 9 PRO 465 (Radeon 880M) |
gfx1150 Gorgon Point |
|
|
Ryzen AI 7 PRO 450 (Radeon 860M) Ryzen AI 5 PRO 440 (Radeon 840M) |
gfx1152 Gorgon Point |
||
|
Ryzen AI 5 PRO 435 (Radeon 840M) |
gfx1153 Gorgon Point |
||
| AMD Ryzen AI 400 Series |
Ryzen AI 9 HX 475 (Radeon 890M) Ryzen AI 9 HX 470 (Radeon 890M) Ryzen AI 9 465 (Radeon 880M) |
gfx1150 Gorgon Point |
|
|
Ryzen AI 7 450 (Radeon 860M) |
gfx1152 Gorgon Point |
||
|
Ryzen AI 5 435 (Radeon 840M) Ryzen AI 5 430 (Radeon 840M) Ryzen AI 7 445 (Radeon 840M) |
gfx1153 Gorgon Point |
||
| AMD Ryzen AI PRO 300 Series |
Ryzen AI 9 HX PRO 375 (Radeon 890M) Ryzen AI 9 HX PRO 370 (Radeon 890M) |
gfx1150 Strix Point |
|
|
Ryzen AI 7 PRO 350 (Radeon 860M) Ryzen AI 5 PRO 340 (Radeon 840M) |
gfx1152 Krackan Point |
||
| AMD Ryzen AI 300 Series |
Ryzen AI 9 HX 375 (Radeon 890M) Ryzen AI 9 HX 370 (Radeon 890M) Ryzen AI 9 365 (Radeon 880M) |
gfx1150 Strix Point |
|
|
Ryzen AI 7 350 (Radeon 860M) Ryzen AI 7 345 (Radeon 840M) Ryzen AI 5 340 (Radeon 840M) Ryzen AI 5 330 (Radeon 820M) |
gfx1152 Krackan Point |
||
| AMD Ryzen PRO 200 Series |
Ryzen 7 PRO 253 (Radeon 780M) Ryzen 7 PRO 250 (Radeon 780M) Ryzen 7 PRO 249 (Radeon 780M) Ryzen 7 PRO 217 (Radeon 740M) Ryzen 5 PRO 230 (Radeon 760M) Ryzen 5 PRO 225 (Radeon 760M) Ryzen 5 PRO 224 (Radeon 760M) Ryzen 5 PRO 220 (Radeon 740M) Ryzen 5 PRO 216 (Radeon 740M) Ryzen 5 PRO 215 (Radeon 740M) Ryzen 3 PRO 210 (Radeon 740M) Ryzen 3 PRO 205 (Radeon 740M) |
gfx1103 Hawk Point |
RDNA 3 |
| AMD Ryzen 200 Series |
Ryzen 9 270 (Radeon 780M) Ryzen 7 260 (Radeon 780M) Ryzen 7 253 (Radeon 780M) Ryzen 7 250 (Radeon 780M) Ryzen 7 249 (Radeon 780M) Ryzen 7 217 (Radeon 740M) Ryzen 5 240 (Radeon 760M) Ryzen 5 230 (Radeon 760M) Ryzen 5 225 (Radeon 760M) Ryzen 5 224 (Radeon 760M) Ryzen 5 220 (Radeon 740M) Ryzen 5 216 (Radeon 740M) Ryzen 3 210 (Radeon 740M) Ryzen 3 205 (Radeon 740M) |
gfx1103 Hawk Point |
|
| AMD Ryzen 100 Series |
Ryzen 9 180 (Radeon 780M) Ryzen 7 165 (Radeon 760M) Ryzen 7 155 (Radeon 740M) Ryzen 5 125 (Radeon 740M) |
gfx1103 Hawk Point |
Operating system support#
ROCm supports the following Linux distributions and Microsoft Windows versions. If you’re running ROCm on Linux, ensure your system is using a supported kernel version.
Important
The following table is a general overview of supported operating systems. Actual support might vary by AMD GPU or APU. Use the Compatibility matrix to verify support for your specific setup before installation.
Linux distribution |
Supported versions |
Linux kernel version |
|---|---|---|
Ubuntu |
26.04.1 |
GA 7.0 |
24.04.5 |
GA 6.8 |
|
22.04.5 |
GA 5.15 |
|
Debian |
13 |
6.12 |
12 |
6.1.0 |
|
Red Hat Enterprise Linux (RHEL) |
10.3 |
6.12.0-264 |
10.2 |
6.12.0-211 |
|
10.0 |
6.12.0-55 |
|
9.9 |
5.14.0-741 |
|
9.8 |
5.14.0-687 |
|
9.6 |
5.14.0-570 |
|
9.4 |
5.14.0-427 |
|
8.10 |
4.18.0-553 |
|
Oracle Linux |
10 |
UEK 8.1 |
9 |
UEK 8 |
|
8 |
UEK 7 |
|
SUSE Linux Enterprise Server (SLES) |
16.1 |
6.12 |
16.0 |
6.12 |
|
15.7 |
6.4.0-150700.51 |
|
Rocky Linux |
9 |
5.14.0-570 |
Operating system |
Supported versions |
Linux kernel version |
|---|---|---|
Ubuntu |
26.04.1 |
GA 7.0 |
24.04.5 |
HWE 7.0 |
|
22.04.5 |
HWE 6.8 |
|
Red Hat Enterprise Linux (RHEL) |
10.3 |
6.12.0-264 |
9.9 |
5.14.0-741 |
|
Windows |
11 25H2 |
— |
Operating system |
Supported versions |
Linux kernel version |
|---|---|---|
Ubuntu |
26.04.1 |
GA 7.0 |
24.04.5 |
HWE 7.0 |
|
Windows |
11 25H2 |
— |
Kernel driver and firmware bundle support#
ROCm requires a coordinated stack of compatible firmware, driver, and user-space components. Maintaining version alignment between these layers ensures correct GPU operation and performance, especially for AMD data center products. While AMD publishes the AMD GPU driver and ROCm user space components, your server OEM (original equipment manufacturer) or infrastructure provider distributes the firmware packages. AMD supplies those firmware images (platform level data model (PLDM) bundles), which the OEM integrates and distributes.
AMD device |
PLDM Bundle (Firmware) |
Linux driver |
|---|---|---|
Instinct MI355X |
01.26.01.06 (or later) |
AMD GPU Driver (amdgpu) |
Instinct MI350X |
||
Instinct MI350P |
BKC13.0 (IFWI PRD1000A) or later |
|
Instinct MI325X |
01.26.01.03 (or later) |
|
Instinct MI300X |
01.26.00.04 (or later) |
|
Instinct MI300A |
PI100E |
|
Instinct MI250X |
Maintenance update (MU) 7 W/IFWI 50 (or later) |
|
Instinct MI250 |
Maintenance update (MU) 5 with IFWI 75 (or later) |
|
Instinct MI210 |
||
Instinct MI100 |
VBIOS D3430401-037 |
[*] The following known issues are observed with amdgpu driver 31.70.0 that pairs with ROCm 10.2.0. They will be fixed in an upcoming amdgpu driver release:
Linux driver |
Windows driver |
|---|---|
|
AMD Software: Adrenalin Edition 26.11.2
Windows CDE CPR driver |
Linux driver |
Windows driver |
|---|---|
|
Ubuntu 26.04.1: 7.0.0 inbox kernel |
AMD Software: Adrenalin Edition 26.11.2
Windows CDE CPR driver |
GPU virtualization support#
AMD Instinct and Radeon GPUs support virtualization in the following configurations. Supported SR-IOV configurations require the AMD GPU Virtualization Driver (GIM) 9.4.0.K. See the AMD Instinct Virtualization Driver documentation for more information.
AMD GPU |
Hypervisor |
Virtualization technology |
Virtualization driver |
Host OS |
Guest OS |
VF per GPU - Compute Mode - Memory Mode |
|---|---|---|---|---|---|---|
Instinct MI355X |
KVM |
Passthrough |
— |
Ubuntu 26.04 |
Ubuntu 26.04 |
1 - SPX - NPS1 |
Ubuntu 24.04 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
||||
Ubuntu 22.04 |
Ubuntu 22.04 |
1 - SPX - NPS1 |
||||
SR-IOV |
GIM 9.4.0.K | Ubuntu 26.04 |
Ubuntu 26.04 |
1 - SPX - NPS1 |
||
Ubuntu 24.04 |
Ubuntu 24.04 |
|||||
RHEL 10.0 |
1 - SPX - NPS1 |
|||||
RHEL 9.6 |
1 - SPX - NPS1 |
|||||
ESXi |
SR-IOV |
— |
VMware ESXi 9.1 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
|
Instinct MI350X |
KVM |
Passthrough |
— |
Ubuntu 26.04 |
Ubuntu 26.04 |
1 - SPX - NPS1 |
Ubuntu 24.04 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
||||
Ubuntu 22.04 |
Ubuntu 22.04 |
1 - SPX - NPS1 |
||||
SR-IOV |
GIM 9.4.0.K | Ubuntu 26.04 |
Ubuntu 26.04 |
1 - SPX - NPS1 |
||
Ubuntu 24.04 |
Ubuntu 24.04 |
|||||
RHEL 10.0 |
1 - SPX - NPS1 |
|||||
RHEL 9.6 |
1 - SPX - NPS1 |
|||||
ESXi |
SR-IOV |
— |
VMware ESXi 9.1 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
|
Instinct MI350P |
KVM |
Passthrough |
— |
Debian 13 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
SR-IOV |
GIM 9.4.0.K | Ubuntu 24.04 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
||
ESXi |
Passthrough |
— |
VMware ESXi 9.1 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
|
SR-IOV |
— |
VMware ESXi 9.1 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
||
Instinct MI325X |
KVM |
Passthrough |
— |
Ubuntu 26.04 |
Ubuntu 26.04 |
1 - SPX - NPS1 |
Ubuntu 24.04 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
||||
RHEL 9.4 |
1 - SPX - NPS1 |
|||||
Ubuntu 22.04 |
Ubuntu 22.04 |
1 - SPX - NPS1 |
||||
RHEL 9.4 |
RHEL 9.4 |
1 - SPX - NPS1 |
||||
SR-IOV |
GIM 9.4.0.K | Ubuntu 26.04 |
Ubuntu 26.04 |
1 - SPX - NPS1 |
||
Ubuntu 22.04 |
Ubuntu 22.04 |
1 - SPX - NPS1 |
||||
RHEL 10.2 |
RHEL 10.2 |
1 - SPX - NPS1 |
||||
RHEL 9.4 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
||||
RHEL 9.4 |
1 - SPX - NPS1 |
|||||
Instinct MI300X |
KVM |
Passthrough |
— |
Ubuntu 26.04 |
Ubuntu 26.04 |
1 - SPX - NPS1 |
Ubuntu 24.04 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
||||
RHEL 9.4 |
1 - SPX - NPS1 |
|||||
Ubuntu 22.04 |
Ubuntu 22.04 |
1 - SPX - NPS1 |
||||
RHEL 9.4 |
RHEL 9.4 |
1 - SPX - NPS1 |
||||
ESXi 8 U3 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
||||
Ubuntu 22.04 |
1 - SPX - NPS1 |
|||||
SR-IOV |
GIM 9.4.0.K | Ubuntu 26.04 |
Ubuntu 26.04 |
1 - SPX - NPS1 |
||
Ubuntu 24.04 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
||||
Ubuntu 22.04 |
Ubuntu 22.04 |
1 - SPX - NPS1 |
||||
RHEL 10.2 |
RHEL 10.2 |
1 - SPX - NPS1 |
||||
RHEL 9.4 |
RHEL 9.4 |
1 - SPX - NPS1 |
||||
RHEL 9.4 |
Ubuntu 24.04 |
1 - SPX - NPS1 |
||||
Instinct MI210 |
KVM |
Passthrough |
— |
Ubuntu 26.04 |
Ubuntu 26.04 |
N/A |
Ubuntu 24.04 |
Ubuntu 24.04 |
N/A |
||||
Ubuntu 22.04 |
Ubuntu 22.04 |
N/A |
||||
RHEL 9.4 |
Ubuntu 22.04 |
N/A |
||||
SR-IOV |
GIM 9.4.0.K | Ubuntu 26.04 |
Ubuntu 26.04 |
N/A |
||
RHEL 9.4 |
Ubuntu 22.04 |
N/A |
||||
RHEL 9.4 |
N/A |
[*] Multi-VF support requires a compatible firmware.
AMD GPU |
Hypervisor |
Virtualization technology |
Virtualization driver |
Host OS |
Guest OS |
VF per GPU - Compute Mode - Memory Mode |
|---|---|---|---|---|---|---|
Radeon AI PRO R9700S |
KVM |
Passthrough |
— |
Ubuntu 24.04 |
Ubuntu 24.04 |
N/A |
Radeon PRO W7900 |
KVM |
Passthrough |
— |
Ubuntu 26.04 |
Ubuntu 26.04 |
N/A |
Ubuntu 24.04 |
N/A |
|||||
Ubuntu 24.04 |
Ubuntu 26.04 |
N/A |
||||
Ubuntu 24.04 |
N/A |
|||||
Radeon PRO W7900D |
KVM |
Passthrough |
— |
Ubuntu 26.04 |
Ubuntu 26.04 |
N/A |
Ubuntu 24.04 |
N/A |
|||||
Ubuntu 24.04 |
Ubuntu 26.04 |
N/A |
||||
Ubuntu 24.04 |
N/A |
|||||
Radeon PRO W7900 Dual Slot |
KVM |
Passthrough |
— |
Ubuntu 26.04 |
Ubuntu 26.04 |
N/A |
Ubuntu 24.04 |
N/A |
|||||
Ubuntu 24.04 |
Ubuntu 26.04 |
N/A |
||||
Ubuntu 24.04 |
N/A |
|||||
Radeon PRO W7800 |
KVM |
Passthrough |
— |
Ubuntu 26.04 |
Ubuntu 26.04 |
N/A |
Ubuntu 24.04 |
N/A |
|||||
Ubuntu 24.04 |
Ubuntu 26.04 |
N/A |
||||
Ubuntu 24.04 |
N/A |
|||||
Radeon PRO V710 |
KVM |
SR-IOV |
GIM 9.4.0.K | Ubuntu 26.04 |
Ubuntu 26.04 |
1 to 12 |
Ubuntu 24.04 |
1 to 12 |
|||||
RHEL 10.2 |
1 to 12 |
|||||
RHEL 9.6 |
1 to 12 |
|||||
Ubuntu 24.04 |
Ubuntu 24.04 |
1 to 12 |
||||
RHEL 10.2 |
1 to 12 |
|||||
RHEL 9.6 |
1 to 12 |
GPU partitioning support#
The following compute partition and NUMA-per-socket (NPS) configurations are available on AMD Instinct GPUs in bare-metal deployments.
Deployment |
Device |
Compute partition mode |
Memory partition mode |
|---|---|---|---|
Bare metal |
Instinct MI355X, Instinct MI350X |
SPX |
NPS1 |
DPX |
NPS2 |
||
CPX |
NPS2 |
||
QPX |
NPS2 |
||
Instinct MI350P |
SPX |
NPS1 |
|
DPX |
NPS1 |
||
CPX |
NPS1 |
||
Instinct MI325X |
SPX |
NPS1 |
|
Instinct MI300X |
SPX |
NPS1 |
|
DPX |
NPS2 |
||
CPX |
NPS4 |
The following configurations are available on AMD Instinct GPUs in passthrough deployments.
Deployment |
Device |
Compute partition mode |
Memory partition mode |
|---|---|---|---|
KVM Passthrough |
Instinct MI355X, Instinct MI350X, Instinct MI325X, Instinct MI300X |
SPX |
NPS1 |
ESXi Passthrough |
Instinct MI350P, Instinct MI300X |
SPX |
NPS1 |
The following configurations are available on AMD Instinct GPUs in SR-IOV deployments. See GPU virtualization support for driver support information.
Deployment |
Device |
VFs per GPU |
Compute partition mode |
Memory partition mode |
|---|---|---|---|---|
KVM SR-IOV |
Instinct MI355X, Instinct MI350X |
1 |
SPX |
NPS1 |
2[*] |
DPX |
NPS2 |
||
8[*] |
CPX |
NPS2 |
||
Instinct MI325X |
1 |
SPX |
NPS1 |
|
Instinct MI300X |
1 |
SPX |
NPS1 |
|
8[*] |
CPX |
NPS4 |
||
ESXi SR-IOV |
Instinct MI355X, Instinct MI350X |
1 |
SPX |
NPS1 |
See the AMD GPU partitioning topic in the AMD GPU Driver documentation to learn more.
AI ecosystem support#
ROCm 10.2.0 provides optimized support for popular deep learning frameworks and AI inference engines. The following table lists supported frameworks and libraries, their compatible operating systems, and validated versions.
Important
The following table is a general overview of supported frameworks and AI inference engines. Actual support might vary by AMD GPU or APU. Use the Compatibility matrix to verify support for your specific setup.
|
Framework |
Supported versions |
Supported OS |
Supported Python versions |
|---|---|---|---|
| PyTorch |
2.15.0 |
Linux |
3.14, 3.13, 3.12, 3.11 |
|
2.15.0 |
Windows |
3.14, 3.13, 3.12, 3.11 |
|
|
2.14.0, 2.13.0 |
Linux |
3.14, 3.13, 3.12, 3.11 |
|
| JAX |
0.11.2 |
Linux |
3.14, 3.13, 3.12 |
|
0.11.1 |
3.14, 3.13, 3.12 |
||
|
0.10.2 |
3.14, 3.13, 3.12, 3.11 |
||
| vLLM |
Linux |
3.14 (requires PyTorch 2.14.0) |
|
| SGLang |
0.5.21 |
Linux |
3.14 (requires PyTorch 2.14.0) |
| TensorFlow |
2.21 |
Linux |
3.12 |
|
2.20 |
|||
| ONNX Runtime |
Linux |
3.12 |
|
| MIGraphX |
Linux |
3.12 |
ROCm Core SDK components#
The following table lists core tools and libraries included in the ROCm 10.2.0 release.
Important
The following table is a general overview of ROCm Core SDK components. Actual support for these libraries and tools can vary by GPU and OS. Use the Compatibility matrix to verify support for your specific setup.
Component group |
Component name |
Version |
Supported platforms |
|
|---|---|---|---|---|
Math and compute libraries |
Composable Kernel | 1.3.0 ⇒ 1.3.0 | Linux/Windows · Instinct/Radeon | |
| hipBLAS | 3.7.0 ⇒ 3.8.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| hipBLASLt | 1.4.1 ⇒ 1.5.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| hipCUB | 4.7.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| hipFFT | 1.0.26 ⇒ 1.0.27 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| hipRAND | 3.4.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| hipSOLVER | 3.7.0 ⇒ 3.8.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| hipSPARSE | 4.8.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| hipSPARSELt | 0.2.9 | Linux/Windows · Instinct (gfx950/ |
||
| libhipcxx | 3.0.2 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| MIOpen | 3.6.1 ⇒ 3.6.2 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| rocBLAS | 5.7.0 ⇒ 5.8.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| rocFFT | 1.0.40 ⇒ 1.0.41 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| rocPRIM | 4.6.0 ⇒ 4.8.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| rocRAND | 5.0.0 ⇒ 5.2.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| rocSOLVER | 3.37.0 ⇒ 3.38.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| rocSPARSE | 5.1.0 ⇒ 5.1.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| rocThrust | 4.7.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| rocWMMA | 2.2.1 ⇒ 2.2.2 | Linux/Windows · Instinct/Radeon/Ryzen | ||
Communication libraries |
RCCL | 2.30.7 ⇒ 2.32.3 | Linux · Instinct/Radeon/Ryzen | |
| rocSHMEM | 3.7.0 ⇒ 3.8.0 | Linux · Instinct/Radeon (gfx950/ |
||
Media and vision libraries |
rocDecode | 1.10.0 ⇒ 1.12.0 | Linux · Instinct/Radeon/Ryzen (gfx1150/ |
|
| rocJPEG | 1.10.0 ⇒ 1.13.0 | Linux · Instinct/Radeon/Ryzen (gfx1150/ |
||
| RPP | 3.2.0 ⇒ 3.2.0 | Linux · Instinct/Radeon/Ryzen (gfx1150/ |
||
Storage libraries |
hipFile | 0.5.0 ⇒ 0.6.0 | Linux · Instinct | |
Threading libraries |
hipThreads | 1.0.0 | Linux/Windows · Radeon/Ryzen | |
Runtimes and compilers |
HIP | 7.16.0 ⇒ 7.17.0 | Linux/Windows · Instinct/Radeon/Ryzen | |
| HIPIFY | 10.1.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| LLVM | 24.0.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
| ROCr Runtime | 1.21.0 | Linux · Instinct/Radeon/Ryzen | ||
| SPIRV-LLVM-Translator | 24.0.0 | Linux/Windows · Instinct/Radeon/Ryzen | ||
Profiling and debugging tools |
ROCdbgapi | 0.80.0 ⇒ 0.82.1 | Linux · Instinct/Radeon | |
| ROCgdb | 16.3 ⇒ 16.3 | Linux · Instinct/Radeon | ||
| ROCm Compute Profiler | 3.9.0 ⇒ 3.10.0 | Linux · Instinct/Ryzen (gfx1150 (Strix Point only)/ |
||
| ROCm Systems Profiler | 1.9.0 ⇒ 1.10.0 | Linux · Instinct/Radeon/Ryzen (gfx1150 (Strix Point only)/ |
||
| ROCprofiler-SDK | 1.4.1 ⇒ 1.4.2 | Linux · Instinct/Radeon/Ryzen (gfx1150/ |
||
| ROCr Debug Agent | 2.2.0 | Linux · Instinct/Radeon | ||
Control and monitoring tools |
AMD SMI | 27.1.0 ⇒ 27.2.0 | Linux · Instinct/Radeon | |
| ROCm Data Center Tool | 1.3.1 ⇒ 1.4.0 | Linux · Instinct | ||
| rocminfo | 1.0.0 | Linux · Instinct/Radeon/Ryzen |
ROCm component changelogs#
The following sections describe key changes to ROCm Core SDK components.
Note
For a historical overview of ROCm component updates, see the ROCm consolidated changelog.
AMD SMI (27.2.0)#
Added#
Added unified
amdsmi_get_link_topology()API on bare metal.Returns link weight, status, type, hop count, and framebuffer-sharing data.
Matches the existing host interface; includes Python and Rust bindings.
Exports
AmdSmiLinkStatusin Python andAmdsmiLinkStatusTin Rust for status interpretation.Self pair (same GPU):
INTERNAL,ENABLED, zero hops and weight,fb_sharing=1.Migration from component APIs on bare metal:
amdsmi_topo_get_link_type(gpu, gpu)reports PCIe and 2 hops instead.amdsmi_topo_get_link_weight(gpu, gpu)reports twice the NUMA node weight (typically 40).
Added IFoE fabric RAS CPER API.
amdsmi_get_fabric_cper_entries(): Retrieves IFoE RAS CPER records from the UALoE library.CLI:
amd-smi ras --cpernow reports fabric link events alongside GPU errors (fabric-linkdown, fabric-linkup, fabric-fatal).
Added AMPP (amdsmi power profile) API and CLI support.
amdsmi_get_ampp_profiles,amdsmi_get_ampp_fields,amdsmi_activate_ampp_profile, andamdsmi_configure_ampp_profileexpose the driver’sapp_modes/sysfs tree of predefined power/performance recipes. Profiles, fields, and units are all dynamically enumerated at runtime, and nothing is hardcoded, since the driver does not guarantee a fixed field set, profile count, or naming across SoC generations.amd-smi static --amppdisplays the tree-wideapp_modes/profile_abiversion plus per-profile active/writable/configured state and per-field name/unit/value/min/max.amd-smi set --ampp-activate <profile_name>activates a profile.amd-smi set --ampp-configure <profile_name> KEY=VALUE...stages one or more fields on a writable profile and commits them.amd-smi set --ampp-configure @<path>restores every writable profile with staged fields from anamd-smi static --ampp --json-shaped file in one call.This is unrelated to the legacy
amdsmi_*_gpu_power_profile*preset-mask API, which is unchanged.
Added
AMDSMI_VRAM_TYPE_HBM4toamdsmi_vram_type_t.Identifies HBM Generation 4 VRAM, reported by
amdsmi_get_gpu_vram_info().Also added the pre-existing
HBM3Evalue to the PythonAmdSmiVramTypeenum, which had been missing it.
Added NPM (Node Power Management) power limit setting.
New
amd-smi set --node-power-limit/-nto set the node-level power limit in watts (node-wide, not per-GPU).New API:
amdsmi_set_npm_limit().New
current_node_powerfield inamdsmi_npm_info_t(returned byamdsmi_get_npm_info()) reports the current (instantaneous) node power in watts, queried once per node rather than once per GPU.amd-smi node -pnow displays it when available.New
max_node_power_limitfield inamdsmi_npm_info_t(returned byamdsmi_get_npm_info()), the platform max bound foramdsmi_set_npm_limit()requests.amdsmi_set_npm_limit()rejects the request withAMDSMI_STATUS_INVALwhen NPM is disabled on the node (amdsmi_npm_info_t::status == AMDSMI_NPM_STATUS_DISABLED), instead of writing a limit that would have no defined effect.amd-smi set --node-power-limitperforms the same check up front to fail fast with a friendlier message.
Added NPM (Node Power Management) balancing mode get/set.
New
amdsmi_npm_balancing_mode_tenum:AMDSMI_NPM_BALANCING_MODE_POWER_BALANCING(PB, the default) orAMDSMI_NPM_BALANCING_MODE_FREQUENCY_BALANCING(FB).New APIs:
amdsmi_get_npm_balancing_mode()andamdsmi_set_npm_balancing_mode(). Getting always reflects the last-selected mode regardless of NPM enablement, returningAMDSMI_STATUS_NOT_SUPPORTEDonly when the underlying value is missing or unreadable.Setting is AMD-SMI-only (not exposed through BMC Redfish/APML) and requires elevated privileges; it returns
AMDSMI_STATUS_NOT_SUPPORTEDwithout touching sysfs when NPM is disabled on the node.CLI:
amd-smi node -pnow showsBALANCING_MODE; newamd-smi set --node-balancing-mode {POWER_BALANCING,FREQUENCY_BALANCING}(node-wide, not per-GPU, requires sudo).New
AMDSMI_NPM_BALANCING_MODE_MAXenum value marks the end of the defined range, and newamdsmi_get_npm_supported_balancing_modes()reports the balancing modes supported by a node’s platform/ASIC as a bitmask (bit N for enum value N), independent of NPM enablement.amdsmi_set_npm_balancing_mode()now returnsAMDSMI_STATUS_SETTING_UNAVAILABLEwhen the requested mode is absent from this bitmask, distinct from theAMDSMI_STATUS_NOT_SUPPORTEDNPM-disabled case. CLI:amd-smi node -pnow also showsSUPPORTED_BALANCING_MODES, andamd-smi set --node-balancing-modereports a distinct “not supported on this platform” message for a mode absent from the bitmask vs. the generic “NPM is disabled” message.
Added a receipt timestamp to
amd-smi eventrecords.AmdSmiEventReader.read()now attaches atimestamp(Unix epoch seconds) to every event record, and theeventsubcommand surfaces it as aTIMESTAMPfield so events can be correlated across streams.Human-readable output renders one indented block per event (including
TIMESTAMP);--csvwrites the header once followed by a row per event, and--jsonemits newline-delimited JSON (one object per line).
Changed#
Use fwupd daemon to read and write UMA carveout information.
amdsmi_get_gpu_uma_carveout_info()/amdsmi_set_gpu_uma_carveout()(and thereforeamd-smi static/set --mem-carveout) now read and write the carveout through the fwupd BIOS-settings interface on integrated GPUs and fall back to the amdgpu.../device/uma/carveoutsysfs node when the fwupd daemon is absent.Implemented in the AMD SMI library over the fwupd daemon’s D-Bus BIOS-settings interface; the fwupd path is used only for the integrated (APU) GPU, and PolicyKit brokers authorization instead of requiring explicit root. Reading needs fwupd >= 1.8.4; writing needs fwupd >= 2.1.1 (Ubuntu 26.04+).
Reworked
amd-smiCLI process exit codes.How exit codes are chosen:
Library failures exit with the underlying
AMDSMI_STATUS_*value (0-56).CLI-only errors use dedicated codes in the reserved 193-253 band.
When more than one failure with different codes is recorded in a single run — across devices, or across sub-steps of one command (e.g.,
reset --clocks) — the process exits204.
Multi-device
set/resetbehavior:A per-device GPU failure is now recorded, and the command continues to the remaining devices.
The exit code is decided once, after every device has been attempted. Previously a single device failure could abort the loop early.
Notable value changes (many cases previously aliased onto
1/2; each now has a distinct code):Case
Was
Now
Import failure
1193Invalid command
1194Invalid parameter
2195Device not found (bad device index, or no devices present)
3/255196Invalid file path
4197Invalid parameter value
5198Missing parameter value
6199Command not supported (CLI, parse-time)
7200(distinct from libraryNOT_SUPPORTED=2)Device interface unavailable (CLI, runtime)
—
201Required target/argument missing
9202Invalid subcommand
10203Permission denied
1110(realAMDSMI_STATUS_NO_PERM)Mixed device/field failures (differing codes)
—
204amdsmi_init()watchdog timeout2205Drivers not loaded
255206Interactive confirmation declined (answering no, or closing stdin)
1207Library (device) failure
(1000 + status)wrapped to a byteunderlying
AMDSMI_STATUS_*(0-56)Unknown/unmapped library error
231255(libraryUNKNOWN_ERRORreported as its low byte)Library status too large to report as an exit code
—
253(not reachable with current library versions)The
Was/Nowvalues are process exit codes ($?). Previously the printedcodefield showed the negative of these (e.g.,-7) while the process exited with the absolute value (7); the rework makes the printedcodeand$?agree (both200).
amd-smi set --power-capnow applies per GPU instead of aborting on the first out-of-range device.Each GPU is validated against its own reported range.
An out-of-range value on one GPU is reported for that GPU, while in-range GPUs still apply.
The process exit code reflects the per-device failure.
Clearer, more consistent
amd-smierror messages.Errors now read
[AMDSMI_STATUS_<name>] <message>, so the underlying status is obvious at a glance. The internal class name is no longer prepended, and--json/--csverrors are now valid JSON/CSV.Before:
amdsmi_cli_exceptions.AmdSmiLibraryErrorException: AMDSMI has returned error '-1002' - 'Command not supported' Error code: -1002After (human):
[AMDSMI_STATUS_NOT_SUPPORTED] Command not supported Error code: 2 [AMDSMI_STATUS_NOT_SUPPORTED]After (
--json):{"error": "[AMDSMI_STATUS_NOT_SUPPORTED] Command not supported", "code": 2, "error_type": "AMDSMI_STATUS_NOT_SUPPORTED"}
--jsonand--csverrors gain anerror_typefield (e.g.,INVALID_PARAMETER_VALUEor anAMDSMI_STATUS_*name) alongside the numericcode.--csvgains a third column (error,code,error_type); a reader that unpacks exactly two fields per row will need updating.amd-smi rasreports a readable error when a CPER file can’t be read or decoded, instead of printing a Python traceback.
amd-smi rasnow exits non-zero when a CPER file fails to decode.Affects
--afid --folder,--afid --cper-file, and--cper, which previously exited0.The exit code is the underlying
AMDSMI_STATUS_*, for example43. When files fail with different codes, the command exits204.The failing file shows
[AMDSMI_STATUS_<name>] <message>in place of its AFIDs.A record that decodes successfully but carries no AFIDs (e.g., a corrected entry) shows
N/Ain place of its AFIDs.--jsonand--csvgainstatus,message, andcodefields. Existing fields are unchanged.--afid --csvnow emits CSV instead of the human-readable table.
amd-smi set --compute-partition/-Cnow attempts each GPU individually.An unsupported GPU reports
NOT_SUPPORTEDon its own, instead of the whole command aborting up front.Input is validated against the static partition type names when profiles cannot be enumerated.
A numeric profile INDEX (e.g.,
-C 0) is also accepted, matched against an estimatedSPX, DPX, TPX, QPX, CPXordering.The
-Chelp now listsSPX, DPX, TPX, QPX, CPXand0-4instead ofN/A.
amdsmi_get_npm_info()andamdsmi_set_npm_limit()now rejectamdsmi_node_handlevalues not vended byamdsmi_get_node_handle().Previously any non-null handle was dereferenced directly; an unregistered/garbage handle now returns
AMDSMI_STATUS_INVALinstead.
NPM sysfs numeric reads (e.g.,
cur_node_power_limit,max_node_power_limit) now reject negative or malformed content.Previously a leading
-(e.g.,"-1") parsed successfully asUINT64_MAX; such content now fails withRSMI_STATUS_UNEXPECTED_DATA.
amd-smi metricno longer prints fields the GPU’sgpu_metricstable version cannot carry (human-readable output only).These fields previously printed as
N/Aand are now omitted from human-readable output. On a metrics v1.3 GPU this removes 88 of the 110N/Avalues inamd-smi metric -g 0(80%), taking the command from 140 lines to 95.Fields that stop appearing on current accelerators. On the metrics v1.6 and later tables, which cover MI300X, MI325X and MI355X, the
POWERsection losesTHROTTLE_STATUSand theUSAGEsection losesMM_ACTIVITY,VCN_ACTIVITYandJPEG_ACTIVITY. Naming the section does not bring them back, soamd-smi metric --powerno longer printsTHROTTLE_STATUSandamd-smi metric --usageno longer prints the three activity rows. On a metrics v1.3 GPU plainamd-smi metricinstead drops the wholeTHROTTLEsection, andTHROTTLE_STATUSis still printed.--show-unsupportedrestores all of them, and--jsonand--csvnever stopped emitting them.--jsonand--csvare unaffected and still emit every field,N/Aones included. They are the machine-readable contract, so their key sets and column sets stay complete and stable: any key or column a script indexed before still resolves. Anything parsing the human-readable output does see fewer fields, and--show-unsupportedgives it back the previous output.Pass
--show-unsupportedto restore the full human-readable output. The flag is accepted with--jsonand--csvand has no effect there, since those formats are never filtered.Scoped to the metrics table version, not to ASIC support: fields the CLI can also source from hwmon or sysfs are always printed, namely the
edge,hotspotandmemtemperature sensors, the fan section, the voltages, and thegfx_N,vclk_N,dclk_N,mem_Nandsocclk_Nclock slots. Entries sourced only from the metrics blobs are eligible for omission: thehbm_stacks,mid,aidandxcdtemperature arrays, and theuclk_aidandsocclks_midclock arrays.A section named on the command line is never emptied. Plain
amd-smi metricdrops a section the version can populate nothing of, which on a metrics v1.3 GPU removes the wholethrottlesection;amd-smi metric --throttleprints all 31 rows instead. A named section that is only partly suppressed is still filtered.--partitionscopes the data rather than naming a section, soamd-smi metric --partitionprotects nothing and filters like plainamd-smi metric.A field the version does carry but the ASIC or driver leaves unpopulated still prints
N/A, a field reporting a real value is never removed, and an unrecognized metrics version or an unreadable header suppresses nothing.
amd-smi staticshows NUMA information on Linux guests.--numawas previously limited to Linux bare metal and passthrough guests.Default
amd-smi staticoutput on guests now includes NUMA, in human-readable, JSON, and CSV output.
Removed#
Removed the internal
amd-smiCLI exception classesAmdSmiParameterNotSupportedExceptionandAmdSmiUnknownErrorException.
These were CLI-internal (not part of the publicamdsmiPython library). Their exit-code behavior is superseded by the reworked CLI process exit codes described above.
Resolved issues#
Fixed
amd-smiprinting a Python traceback when an unknown NIC or switch is selected.amd-smi static --nic 999and--switch 999failed while building the “device not found” error, so the command exited1with a traceback and no readable message.--jsonand--csvproduced no parseable output.Both now report
Can not find a device: NIC '999'(orSWITCH) and exit196, matching--gpu,--cpu, and--core.
Fixed
amd-smi set/reseton a GPU silently exiting0after a per-device failure.A device error during GPU
set/resetis now recorded, so the process exits with a non-zero code instead of reporting success.Example:
set --memory-partitionon a GPU that does not support it.
Fixed the topology APIs reporting uninitialized NUMA data when a GPU has no CPU
io_link.The KFD node left its NUMA node number, weight and type uninitialized, so
amdsmi_topo_get_numa_node_number(),amdsmi_topo_get_link_weight()andamdsmi_topo_get_link_type()could return an unrelated value alongsideAMDSMI_STATUS_SUCCESS. They now returnAMDSMI_STATUS_NOT_SUPPORTEDfor that case.A debug build previously aborted on an assertion in the same path instead of reporting a status.
Fixed
amdsmi_get_gpu_fabric_info()failing when only some UALink sysfs files carry data.The call required every file in scope to have usable content and otherwise returned
AMDSMI_STATUS_UNEXPECTED_DATA, so a legitimately emptyvpod_active_accelsorlocal_accelssuppressed the rest of the fabric data and leftamd-smi fabricwith nothing to print.It now returns
AMDSMI_STATUS_SUCCESSwhen at least one file has usable content, andAMDSMI_STATUS_NO_DATAonly when none do.
Restored
amd-smibash autocompletion.Pressing Tab completes
amd-smisubcommands and their options again, whether completion is registered explicitly or through the globalargcompletehook.
Fixed runtime fatal CPERs reporting no AFIDs.
amd-smi ras --cpershowed an emptylist afidscolumn for fatal records,amd-smi ras --afid --cper-fileprinted-, andamdsmi_get_afids_from_cper()returned no AFIDs. amdgpu writes fatal crashdump sections 32 bytes shorter thansizeof(cper_sec_crashdump), and the section bounds check required the full struct, so every such section was skipped. The check now requires only the dump member the record type uses.
Fixed APU core and skin temperatures showing
0.0 °Cwhen the sensor is not measured.On APUs such as Ryzen AI Max (Strix Halo),
amd-smi metricand the library now report these readings as not available instead of a false 0 °C.
Fixed xGMI read and write data counters reading as unavailable on gfx1250.
On gfx1250 the GPU connects to the CPU over xGMI, and the driver reports that link’s traffic as a single counter. It was dropped instead of being stored as the first link, so
xgmi_read_data_accandxgmi_write_data_accfromamdsmi_get_gpu_metrics_info(), and the linkread/writefromamdsmi_get_link_metrics(), readUINT64_MAX(N/A).
Fixed
amdsmi_get_processor_count_from_handles()crashing on NULL output pointers.It wrote through a NULL
nr_cpusockets,nr_cpucoresornr_gpusand crashed. It now returnsAMDSMI_STATUS_INVAL.
Fixed
amdsmi_get_gpu_cper_entries()failing the page after a large CPER record.A record an earlier call had already returned still had to fit the caller’s buffers on the next call, so that call failed with
AMDSMI_STATUS_OUT_OF_RESOURCES. Records already returned are now skipped before the space check.
Fixed
amd-smi static --clockprinting a Python traceback for an unknown clock name.It now reports the invalid parameter and exits
195(INVALID_PARAMETER).
Fixed
amd-smi static --numa --csvdropping socket affinity.When NUMA-node CPU affinity was unavailable, as on guests, the CSV row printed
N/Afor the socket columns that human-readable and JSON output showed. It also no longer fails when only the socket affinity is unavailable.
Fixed crashes when several threads start or stop AMD SMI at the same time.
Programs that initialize and shut down the library from more than one thread could crash, find no GPUs, or leave the library initialized after every thread had shut it down.
Fixed a one-byte overrun when reading the memory partition into a small buffer.
A buffer too small for the partition name had the byte after its end overwritten. The name is now truncated inside the buffer, and the call still reports that the buffer is too small.
amd-smi set --memory-partitionno longer advises a driver reload when the partition is already current.Requesting the current partition does not cancel a different partition set earlier, so a reload would apply that one.
amd-sminow reports that nothing changed and notes that an earlier request still applies at the next reload.
Fixed
amdsmi_get_gpu_driver_info()reporting0as the driver date.Kernels that no longer use the DRM driver date report it as
0.driver_dateis now empty in the C API andN/Ain Python.
Fixed
amd-smi node --trayprinting an empty section on hardware without UALoE.The tray output was guarded on the retrieved dictionary being non-empty, so the
N/Adefaults next to it could never be reached.NODE:was emitted with no body, and--jsonreturned[{"node": {}}]. All three output formats now reportMAX_ACC_PER_TRAYandTRAY_TYPEasN/A.
Fixed
amdsmi_get_afids_from_cper()raisingTypeErrorwhen passed a list of CPER records.The list branch called
isinstance()with a subscripted generic, which Python rejects at runtime, so the documented list-of-dicts input always failed.bytesinput was unaffected. Input that is neitherbytesnor a list now raisesAmdSmiParameterException.
Fixed a segmentation fault in the DRM example when run without root.
amdsmi_get_gpu_accelerator_partition_profile_config()returnsAMDSMI_STATUS_NO_PERMwithout filling its output, and the example readnum_profilesfrom that uninitialized structure, so the profile loop ran past the end of theprofilesarray on every GPU. The loop now runs only when the query succeeds.
Fixed
amd-smiusing theamdsmiPython library of another ROCm install.When
ROCM_PATHorROCM_HOMEnamed a different ROCm install,amd-smiloaded that install’s library, so it either failed with anAttributeErroror reported the other install’s library and ROCm versions.ROCM_PATHandROCM_HOMEno longer redirect whichamdsmiPython packageamd-smiloads.
Fixed
amdsmi_set_clk_freq()falsely returningAMDSMI_STATUS_INVALon Navi48 (gfx1201).The deep-sleep bitmask validation added previously rejected the entire request when any requested bit fell outside a settable range computed from a prior read, so a supported clock set failed on gfx1201 even though the write is valid. The set is now lenient: out-of-range bits are dropped, and
AMDSMI_STATUS_INVALis only returned when no requested level is settable. The read-only check runs after the write, andTestFrequenciesReadWriteis re-enabled on Aldebaran (gfx90a / MI210).
Composable Kernel (1.3.0)#
Added#
Added a CI path-length check that rejects newly added or renamed files whose repository-relative path exceeds 200 characters, keeping the absolute path under the Windows
MAX_PATHlimit of 260.Added bf16, fp32, fp8, and bf8 data types and the
rrr,crr, andccrlayouts to the CK Tile Tile Engine batched GEMM and its dispatcher bridge.
Optimized#
Improved FMHA backward performance on gfx1250 by moving the operand transfers to TDM and the transposed reads to
ds_load_tr, keeping the dK and dV accumulators in registers, pairing mirrored tiles under a causal mask, and coalescing the dQ atomic onto a single cache line.
HIP (7.17.0)#
Added#
New HIP APIs:
Module Management: matches the corresponding CUDA API.
hipModuleEnumerateFunctionsreturns the function handles defined in a loaded module.
Library Management: matches the corresponding CUDA API.
hipLibraryGetModulereturns the module handle associated with a library.hipKernelSetAttributeForDevicesets a device-wide attribute for a kernel on a specific device.
Support for the
hipMemRangeFlagDmaBufMappingTypePcieflag inhipMemGetHandleForAddressRange, matching the CUDA behavior.Support for recovering allocation properties from imported virtual memory handles.
hipMemGetAllocationPropertiesFromHandlenow correctly reportshipMemLocationTypeHostfor host-backed allocations imported throughhipMemImportFromShareableHandle, instead of always reporting device memory. For device-backed allocations, it reports the owning device rather than the device that was current when the handle was imported.Support in WSL2 to prevent indefinite application hangs and to report errors for device-side faults, including out-of-bounds writes, null pointer dereferences, and device asserts.
Added
HIP_TARGET_API_VERSIONto enable compile-time API version selection, helping maintain source compatibility with applications built against older HIP API versions.
Changed#
Stream priority is now disabled to avoid known queue-priority-related issues. Priority streams are currently not supported.
The
HIPRTC_USE_RUNTIME_UNBUNDLERenvironment variable and its associated functionality have been removed from the HIP runtime. The runtime unbundler path is now always used; setting this variable no longer has any effect.
Optimized#
Enhanced graph execution update performance and scalability by reducing update overhead and eliminating quadratic scaling behavior in certain graph update scenarios.
Optimized memory allocation capacity on Windows by increasing the maximum aperture size to 4 TiB, enabling larger memory allocations and improving support for memory-intensive workloads.
Optimized the performance of
hipMemcpy3DBatchAsyncby reducing SDMA submission overhead. Previously, the runtime issued a separate SDMA submission for each operand, which added overhead for small transfers. The runtime now reduces this submission overhead for batched 3D memory copy operations.
Resolved issues#
Resolved a regression on MI300A and other APUs that could cause the first stream creation,
hipMemcpy,hipMemset, or kernel launch to fail with out-of-memory or invalid-argument errors. The runtime now correctly uses system-memory AQL rings on APUs, which share memory with the host and do not provide a discrete device-memory path.Resolved a memory leak in
VirtualGPU::ManagedBufferthat could occur during stream creation when signal initialization failed. The runtime now properly cleans up partially created resources on error paths.Resolved an issue in
hipLaunchKernelwhere invalid or unloaded kernel handles could cause a segmentation fault. Invalid handles are now properly validated and return an appropriate error.A registered
__device__global that is absent from the loaded code object no longer aborts the process. Symbol lookup now returnshipErrorInvalidSymbolfrom the runtime’s variable materialization path (hipGetSymbolAddress,hipLibraryGetGlobal, and related entry points).hipModuleGetGlobalstill reportshipErrorNotFoundfor a missing name.Fixed
__hip_bfloat162comparisons that ignored or misread the high lane.__hbneu2now returns true only when both lanes are unordered-not-equal,__hgt2and__hisnan2now return the per-lane result in.yinstead of always 1.0, and the<,<=,>,>=operators now compare.ywith.y. Code that relied on the previous results might see different values.Fixed an issue that could cause peer-to-peer (P2P)
hipMemcpyAsync()transfers using thehipMemcpyDeviceToDeviceNoCUmemcpy kind to fail on certain multi-GPU systems. SDMA engine selection has been improved to ensure reliable P2P transfer operation.Resolved a performance issue affecting large host-to-device (H2D) transfers on integrated (unified-memory) GPUs, where HMM-based KFD userptr pinning could reduce transfer throughput and appear to stall workloads. Large transfers are now routed through the existing staging-copy fallback, improving transfer performance.
Fixed a host-blocking issue in
hipMemcpyBatchAsyncthat could occur when the API was repeatedly issued on a stream containing a long-running kernel. The implementation now reserves only the total descriptor space required for each dispatch in the kernel argument pool, preventing pool exhaustion and eliminating unnecessary host waits for kernel completion.Resolved a segmentation fault in
hipStreamEndCaptureduring multi-stream capture caused by a bookkeeping cycle among capture participants. The fix tracks capture participants in a single origin-stream-owned set, eliminating recursive traversal of forked streams during capture finalization and improving stream capture stability.Resolved a potential deadlock between
hipMallocAsyncandhipEventSynchronizeduring concurrent multi-threaded execution.Fixed incorrect calculations when using the type
boolforcooperative_groups::exclusive_scan. The implementation now correctly handles the identity value for the first active lane and prevents unintended type conversions frombooltoint.Resolved an issue where legacy-stream sync operations (
hipMemset,hipMemcpy) incorrectly invalidated ThreadLocal and Relaxed graph captures onhipStreamNonBlockingstreams. These captures now remain active, matching CUDA semantics. Global-mode captures and captures involving blocking participants are still invalidated as expected.
Known issues#
A GPU Timeout Detection and Recovery (TDR) event can occur on ROCm 10.2 running under WSL2 (Windows Subsystem for Linux 2) when executing the
vllm_rocm_gpt-oss-20b_default_inf_fp4vLLM inference workload.
hipBLAS (3.8.0)#
Added#
Added gfx1170 and gfx1171 support to clients.
Changed#
The minimum required CMake version is now 3.17, for the library and the clients.
FindCUDAToolkit, used by the CUDA backend, was added in that release.hipblasSetWorkspacechooses whether the rocBLAS backend can grow its device workspace. A non-null workspace with a nonzero size is a fixed limit and is never resized, which caps device memory for the handle. A later call that needs more, such astrsmBatchedortrsmStridedBatchedwith a largebatchCount, returnsHIPBLAS_STATUS_ALLOC_FAILEDeven when free device memory remains. Passingnullptror a size of 0 returns the handle to rocBLAS-managed memory, which starts from a default size and grows as needed.HIPBLAS_WORKSPACE_CONFIG=:0:0selects that same managed mode for callers that configure the workspace through the environment, such as PyTorch. With the rocBLAS backend, setROCBLAS_VERBOSE_ERROR=1to print the required and available sizes when a fixed workspace is too small fortrsm.
hipBLASLt (1.5.0)#
Added#
HIPBLASLT_CHECK_SYNCHRONIZERenvironment variable: opt-in post-launch dirty-buffer check for the handle’s inter-workgroup flag buffers, which their kernels must leave at zero (1,on, ortrueto enable; anything else disables). It scans the wholeSynchronizerbuffer used by GSUMultipleBufferSingleKernelandamaxD, and the Stream-K flag block bound to the launch’s stream. Coversrocblaslt_matmul_implonly. UnlikeHIPBLASLT_CHECK_NUMERICS, each covered call pays a stream sync and a device-to-host copy, so this is single-threaded debugging use only, not for concurrent-stream workloads.w4a16 mixed-precision GEMM support: matrix A accepts
HIP_R_4I(signed int4, two elements per byte) with per-K-group scales, as used by GPTQ, AWQ, and compressed-tensors checkpoints. Group sizes are 32, 64, and 128, with symmetric scales or asymmetric scales that carry packed zero-points. New API: twelvehipblasLtMatmulMatrixScale_tvalues (HIPBLASLT_MATMUL_MATRIX_SCALE_VEC32_16BF_EXTthroughHIPBLASLT_MATMUL_MATRIX_SCALE_VEC128_16F_EXT, plus the matching_ZP_EXTvariants), theHIPBLASLT_MATMUL_DESC_A_INT4_ENCODING_EXTdescriptor attribute withhipblasLtInt4Encoding_t(HIPBLASLT_INT4_ENCODING_SIGNED_EXTorHIPBLASLT_INT4_ENCODING_UNSIGNED_BIAS8_EXT), andhipblaslt_ext::GemmProblemType::setInt4EncodingAandgetInt4EncodingA. Custom gfx1151 assembly kernels cover bf16 and fp16 activations.hipblaslt-benchadds--int4_encoding.gfx1170 and gfx1171 GPU target support, with tuned Equality and GridBased GEMM libraries for gfx1170.
gfx1250 GEMM support for bf16 and fp32 with bias, GELU activation, and Aux output (
UseE), and for per-tensor FP8 and BF8 inputs with FP8 or BF8 output andOutputAmaxD, each for all four transpose combinations. MXFP8 (TN only) adds mixed e4m3/e5m2 inputs, FP8 inputs with FP16 output, and BF8 inputs.Non-TN (NT, NN, and TT) MXFP4 GEMM support on gfx950.
Custom gfx950 kernels for FP6 and BF6 GEMMs, and for block-scaled mixed-MX GEMMs (FP8, BF8, FP6, BF6, and FP4 inputs), with
Half,BFloat16, andFloatoutputs.Tuned grid-based DGEMM library logic for gfx942 at 228 CUs (MI300A), covering all four transpose combinations.
Gemm-From-Anywhere (GFA) custom kernel framework, which lets hipBLASLt load and dispatch GEMM kernels written outside TensileLite (for example AITER, Composable Kernel, rocRoller, Wave, and Triton) through the existing solution-selection path. Each custom kernel’s
.sfile embeds acustom.configYAML block in its.amdgpu_metadatasection that records its provenance, supported features, and, for external kernels, its Tensile interface. TheTensile.AddCustomConfigtool injects that block, andTensile.ValidateMetadatavalidates it.BF16 TN skinny-GEMM kernels (
nup to 4) for gfx950, adapted from rocBLASwvSpltKand dispatched automatically through the GFA framework, for LLM decode shapes.
Changed#
Persistent launch controls now use
TENSILE_PERSISTENT_*environment names, and Hybrid assignment uses--hybrid_assignment_policyinhipblaslt-benchandHybridAssignmentPolicyin TensileLite YAML. Legacy environment, CLI, and YAML aliases remain supported. Preferred environment names take precedence; conflicting old and new CLI or YAML values are rejected.A tuning file is now trusted one entry at a time instead of all or nothing.
HIPBLASLT_TUNING_OVERRIDE_FILErecords solution indices, which are positions in one build’s kernel library. Previously the C API ignored the whole file when its build-version line did not match the running build, while the C++ API applied it regardless and could run kernels it was never tuned on.hipblaslt-benchnow records akernel_namebeside eachsolution_indexinHIPBLASLT_TUNING_FILE, and on both APIs a row that records a name is checked at replay and dropped only if its index no longer names that kernel. A row without a name is used only when the file’sGit Versionline matches the running build; a build made outside a git checkout has no version, so it uses no such rows. Problems whose rows are dropped fall back to normal kernel selection.Extension API
initializecompares against the size set withsetMaxWorkspaceByteson theGemmorGroupedGemminstance, which defaults to 0. A caller that passes a large enough buffer without setting it is now rejected withHIPBLAS_STATUS_INVALID_VALUEfor solutions with a fixed workspace requirement, such as GSU. Stream-K and zero-workspace solutions are unaffected.Workspace limits and sizes above
UINT32_MAX(4 GiB - 1), such asSIZE_MAXfor no limit, are clamped toUINT32_MAX, because kernels address the workspace with 32-bit offsets. This applies toHIPBLASLT_MATMUL_PREF_MAX_WORKSPACE_BYTES(which then reads back asUINT32_MAX), thehipblasLtMatmulworkspace size, and the extension APIsetMaxWorkspaceBytes.Grouped GEMM calls that select a flag-reading Stream-K solution and contain more than 16 problems now fail with
HIPBLAS_STATUS_INVALID_VALUEinstead of assigning the excess problems another problem’s flags.hipblasLtMatmulDescSetAttributenow returnsHIPBLAS_STATUS_NOT_SUPPORTEDfor a non-NULLHIPBLASLT_MATMUL_DESC_EPILOGUE_AUX_SCALE_POINTER(setting it to NULL still succeeds), andhipblaslt_ext::Gemm::setProblemandhipblaslt_ext::GroupedGemm::setProblemdo the same whenGemmInputs::setScaleAuxis given a non-NULL pointer. No kernel applies an epilogue aux scale, so the pointer was previously accepted and silently ignored, leaving the auxiliary output unscaled.
Removed#
TENSILE_PRECISE_SM_TARGETenvironment variable, which previously disabled the rounding of the SM count target.install.sh --asic-revisionoption andHIPBLASLT_ASIC_REVISIONCMake variable.
Optimized#
Improved DGEMM performance on gfx942 (MI300X) with retuned GridBased library logic.
Improved fp16 and bf16 NN GEMM performance on gfx1100 and fp8, bf8, and int8 TN GEMM performance on gfx1201 with Origami libraries.
Added tuned Origami libraries for CGEMM, ZGEMM, and int8 GEMMs on gfx950.
Resolved issues#
Fixed
rocisamacro expansion ignoring conditions after&&, which could omit required final-loop LDS waits in gfx950 kernels using custom main-loop scheduling.Fixed premature LDS reads when handwritten gfx950 BF16 and FP16 TN kernels skip the second global prefetch at
K=64.Fixed
hipblaslt-benchusing C’s batch stride for D and computing its CPU reference with the wrong layout when C and D have different leading dimensions or batch strides.Fixed output-amax accumulation omitting packed-store values and returning zero when C/D scaling is disabled. Invalid Stream-K or split-reduction combinations with output-amax are rejected during solution validation.
Fixed GEMM output scaling reading C/D scale values before their scalar memory loads completed.
Fixed C++ algorithm support checks rejecting integer and complex scalar types, and preserved complex conjugation when constructing or updating GEMM descriptors.
Fixed gfx1250 output-amax generation to use Wave32 masks, supported atomics, cross-workgroup memory ordering, and target-specific buffer descriptors.
When a tuning file held several entries for one problem and the first could not be used, the later entries were never used either. The replay loop reused a single vector across entries and always read element
[0], so each later entry was checked against the first entry’s solution instead of its own.Solution index
0in a tuning file was rejected outright, despite being a valid index that is present in the shipped logic.A grouped GEMM heuristic query crashed when
HIPBLASLT_TUNING_OVERRIDE_FILEheld any entry, because the override lookup read the grouped problem as a single GEMM. Grouped GEMMs now use default selection.hipblaslt-benchrecorded grouped GEMM winners inHIPBLASLT_TUNING_FILE, where a single-group winner read back as a single GEMM of the same shape. Grouped runs no longer write to the file.An XF32 problem on the C++ extension API could fall back to FP32 kernels. When a tuning file entry failed both the XF32 check and its FP32 fallback check, the problem was left in FP32 mode for the remaining entries and for default kernel selection.
*returnAlgoCountwas read uninitialized when an override satisfied a single-algo request, and then used to scan one element past the end of the caller’s array.A tuning file that yielded no usable rows was re-read and re-parsed on every heuristic query instead of once.
A tuning file row cut short, for example when the disk fills while
hipblaslt-benchappends it, could be replayed with a truncatedsolution_index. A row whose cells do not line up with its header is now ignored.GemmInstance::getSolutionName()inhipblaslt_extcrashed for a RocRoller solution. It looked the encoded RocRoller index up in the Tensile library and dereferenced the null result. It now returns the RocRoller short name, ashipblaslt_ext::getSolutionNameFromAlgo()already did.The logger’s destructor called
close()on the log file, which throws out of a destructor and terminates the process at exit when the log file has failed, for example on a full disk. Reproducible withHIPBLASLT_LOG_LEVEL=4and a failingHIPBLASLT_LOG_FILE.hipblasLtMatmulwith noalgocould return success for a GEMM that never launched. Its status already held the result of choosing the algorithm, and an exception thrown after that returned it unchanged instead of an error.Fixed silently incorrect GEMM results from 32-bit overflow in the scalar division helpers that rocisa emits for tile-index and macrotile arithmetic.
scalarStaticDivideAndRemainderandscalarStaticCeilDividecorrupted the quotient for large dividends, andscalarUInt32DivideAndRemainderproduced wrong workgroup coordinates for problems with more than2^24output tiles, which left parts of D unwritten whilehipblasLtMatmulreturned success.Fixed
hipblasLtMatmulreturning success with a wrong or partly unwritten D for problems that exceed 32-bit limits. Problems whose M, N, K, leading dimensions, or batch strides reach2^32elements are now rejected, andhipblasLtMatmulAlgoGetHeuristicreturns no algorithms for them. Launches whose grid does not fit in 32 bits now returnhipErrorInvalidValueinstead of launching a truncated grid.Fixed
hipblasLtMatmulreturningHSA_STATUS_ERROR_MEMORY_FAULTor silently incorrect results for fp32 NN GEMMs with large leading dimensions on gfx1201, caused by the StaggerU global-read increment overflowing a signed 32-bit register.Fixed incorrect work-group count calculation for GEMMs with matrix dimensions above 262,144 elements on MI300, which caused out-of-bounds memory accesses and wrong results.
Fixed wrong results in Stream-K tree-reduction GEMMs with
LocalSplitU > 1when the partials write spans more than one workspace batch. Peer workgroups summed incorrect values into the output tile.Fixed bf16 GEMM solutions with
LocalSplitU > 1on gfx942 and gfx950 that stored unscaled results in later store batches, producing values about half the expected magnitude foralpha = 2.Fixed a race in persistent direct-to-LDS GEMM kernels on gfx950 where the prefetch of the next tile could overwrite the bias and
scaleAlphavectors while slower waves were still reading them, producing nondeterministic wrong results.Fixed incorrect GEMM results for narrow-input (sub-4-byte) kernels configured with
PrefetchLocalRead=2andClusterLocalRead=0. TensileLite now rejects these solutions.Fixed incorrect results on gfx950 where a bf16-bias query could be served f16-bias solutions cached for an earlier
hipblasLtMatmulAlgoGetHeuristiccall of the same shape, sohipblasLtMatmulreturned success and wrote D without the bias. Solution caches are now keyed on every field the solution predicates read.Fixed wrong or NaN values in the auxiliary output E when an FP16 E store used a 16-byte vector store on gfx940-family GPUs, including gfx950.
Fixed a hang when a Stream-K GEMM was launched on the legacy null stream (
nullptrorhipStreamLegacy). The null stream now claims its own flag block instead of aliasing the block of the next stream that claimed one. Grouped GEMM solution switching also no longer leaves a stale Stream-K flag binding that could send a later GSU reduction into the smaller Stream-K region.Fixed batched TF32 GEMMs on gfx950 that select Stream-K SK3 custom kernels failing with
HIPBLAS_STATUS_INTERNAL_ERRORfor large batch counts.Fixed MI300A (gfx942, 228 CUs) kernel selection, which rejected most generic gfx942 solutions and left SGEMM NT and fp8/bf8 GEMMs with only a few tiles. The full MI300X kernel set is available again. The k-d tree batch search also now skips cells with no entry for the problem’s batch count, which could return an incorrect or missing solution.
Fixed a regression that increased
hipblasLtMatmulAlgoGetHeuristichost time on small GEMMs, most visibly on gfx1200 and gfx1201.Fixed the SM count target being rounded down to a multiple of 32, which applied a target of 100 as 96 and treated any target below 32 as 0 (use all CUs). The target is now used as given.
Fixed gfx1250 hangs and faults: an intermittent hang at compute-loop entry in clustered GEMM kernels, a memory fault and GPU hang in MXFP8 GEMMs with Multicast, TDM, and
ClusterDimon non-power-of-2 macro tiles, and a page fault in bf16 pointer-array batched GEMMs with GL2 prefetch.
hipFFT (1.0.27)#
Changed#
Minor internal changes.
hipFile (0.6.0)#
Added#
hipFileGetStatsL3()now populates theuuidfield of eachhipFilePerGpuStatsentry with the raw 16-byte GPU UUID, zero-filled when the device cannot report one.Added a batch API example,
batch-roundtrip.Added reference documentation for the Stats API.
Added a documentation page listing frameworks and applications that use hipFile, including LMCache, fastsafetensors, rocAL, PyTorch, fio, and IOR.
Changed#
The hipFile CMake options have been renamed from
AIS_*toHIPFILE_*(e.g.,AIS_CXX_STANDARDis nowHIPFILE_CXX_STANDARD). The oldAIS_*names still work but are deprecated and emit a CMake deprecation warning.The
HIPFILE_WARN_UNSAFE_BUFFER_OPSCMake option now defaults toOFF.Documented that Linux kernels built with
CONFIG_PCI_P2PDMA=yare required for fastpath.
Resolved issues#
The
HIPFILE_WARN_UNSAFE_BUFFER_OPSCMake option now applies the clang-Wunsafe-buffer-usageand-fsafe-buffer-usage-suggestionsflags to C++ sources. Previously, the flags were never passed to the compiler.hipFile now returns
hipFileGetNewFDFailedinstead ofhipFileInternalErrorwhen the process or system runs out of file descriptors. API calls that need a new file descriptor, such ashipFileHandleRegister(), can return this error.Asynchronous fastpath operations now fail over to the fallback backend on
ENODEVorEREMOTEIO, matching the synchronous and batch paths.
Known issues#
The bandwidth reported by
ais-statsand the Stats API is derived from the sum of the per-operation latencies instead of the elapsed wall-clock time, so when several threads issue I/O concurrently, the reported bandwidth is understated by roughly the number of concurrent threads. See issue #11919.Asynchronous fastpath I/O, added in hipFile 0.5.0 for ROCm 10.1.0, performs poorly. The ROCm runtime serializes all stream host functions onto a single thread, so asynchronous fastpath operations cannot run concurrently.
hipSOLVER (3.8.0)#
Added#
Added functions:
gelsBatchedhipsolverXXgelsBatched_bufferSizehipsolverXXgelsBatched
geqrfBatchedhipsolverSgeqrfBatched_bufferSize,hipsolverDgeqrfBatched_bufferSize,hipsolverCgeqrfBatched_bufferSize, andhipsolverZgeqrfBatched_bufferSizehipsolverSgeqrfBatched,hipsolverDgeqrfBatched,hipsolverCgeqrfBatched, andhipsolverZgeqrfBatched
getrfBatchedhipsolverSgetrfBatched_bufferSize,hipsolverDgetrfBatched_bufferSize,hipsolverCgetrfBatched_bufferSize, andhipsolverZgetrfBatched_bufferSizehipsolverSgetrfBatched,hipsolverDgetrfBatched,hipsolverCgetrfBatched, andhipsolverZgetrfBatched
getriBatchedhipsolverXgetriBatched_bufferSizehipsolverXgetriBatched
getrsBatchedhipsolverSgetrsBatched_bufferSize,hipsolverDgetrsBatched_bufferSize,hipsolverCgetrsBatched_bufferSize, andhipsolverZgetrsBatched_bufferSizehipsolverSgetrsBatched,hipsolverDgetrsBatched,hipsolverCgetrsBatched, andhipsolverZgetrsBatched
Added compatibility-only functions:
trtrihipsolverDnXtrtri_bufferSizehipsolverDnXtrtri
Changed#
The rocSOLVER backend now allows
devIpivto be passed as a null pointer tohipsolverXgetrsandhipsolverDnXgetrs.
Resolved issues#
Fixed
hipsolverSpSetStreamnot setting the internalrocsparse_handlestream.
MIOpen (3.6.2)#
Added#
[Conv] Added gfx950 depthwise backward-weights (fp16/bf16) and gfx1250 depthwise kernels to the
ConvHipConvsolver (hipconv v0.3.1).[Conv] Added gfx950 patch-embedding kernels (fp16/bf16), gfx950 TF32 kernels for dense and depthwise forward/backward-data and dense backward-weights, and gfx1250 depthwise backward-weights kernels (fp16/bf16) to the
ConvHipConvsolver (hipconv v0.3.2).[Conv] Added a gfx1250 dense backward-weights kernel (fp16/bf16) to the
ConvHipConvsolver (hipconv v0.4.0).[Conv] Added NHWC (channel-last) support to the GEMM convolution solvers, together with the im2col and col2im kernels the layout requires.
[Conv] Added cross-architecture AI heuristics for immediate mode (
MIOPEN_FIND_MODE=2): a LightGBM ranker selects the solver and a per-solver model selects its performance config, extending data-driven selection to the Navi architectures.[Conv] Added the missing 3x3 stride-1 pad-1 tile-28 CK depthwise backward-data instance.
Changed#
[Reduce]
miopenGetReductionWorkspaceSizeandmiopenGetReductionIndicesSizenow reject tensors of rank greater than 6, matchingmiopenReduceTensor.[Reduce] Invalid arguments to the reduction API (mismatched descriptors, unsupported types, undersized workspace or indices buffers) now return
miopenStatusBadParminstead ofmiopenStatusUnknownError.[Conv]
ConvHipConvis held out of the AI performance-config picker and uses its own default config until the model is retrained against the current hipconv config vocabulary.[Conv] Large forward and backward-data convolutions are no longer gated off CK by tensor byte span. The 32-bit byte-offset overflow that motivated the gate is fixed in CK itself, so these problems can again use the faster int32 CK instances.
[Conv] Relaxed the ASM-GTC NHWC large-tensor guard for forward and backward-data. Those kernels address a full 4 GiB window per dispatch slice and advance the base pointer per batch-split, so the input and output bounds were redundant; what remains is a bound on the per-group weight size. Backward-weights keeps the stricter int32 bound, because 292 of its 900 kernels ignore the batch-split dimension (#11805). This restores 8,978 forward and backward-data entries across the six system find-databases that the guard’s first revision removed.
Removed#
[Conv] Removed gfx803 convolution solver
ConvBinWinogradRxSFusedand its kernel sources.[Conv] Removed the gfx803 code paths and kernel sources from
ConvBinWinograd3x3UandConvBinWinogradRxS.[Conv] Removed the gfx803 find-database files.
Optimized#
[Conv] Optimized the
ConvDirectNaiveConvsolver with per-direction kernel files, float accumulators for fp32/fp16/bf16 inputs, spatial tiling and a bfloat16 path, to improve performance on representative model shapes.Optimized the kernel cache by reusing cached database connections instead of opening and closing SQLite on every binary load or store.
Resolved issues#
[Reduce] Fixed
miopenReduceTensorsilently truncating tensor lengths, strides, element counts and byte spans aboveINT32_MAX, and single reductions of about2^30or more fp16, bf16 or int8 elements, which produced incorrect results. Such tensors are now rejected withmiopenStatusBadParm.[Conv] Fixed a
ConvHipConvcrash on gfx1250 for grouped convolutions with more than 65535 group-batch tiles, and an LDS race in its gfx1250 direct kernel (hipconv v0.3.2).[Conv] Fixed
ConvHipConvlosing its tuned config on a perf-db reload, which could fail execution with “not enough workspace for direct kernel” after an exhaustive Find. Records now carry the hipconv minor version, and records from another version are discarded.[Conv] Fixed incorrect backward-weights results from the
ConvHipConvgfx950 direct kernel for configurations withwaves_q == 1, where the LDS staging tile was reused across filter positions without cross-lane ordering.[Conv] Fixed slower-than-expected performance of the
ConvHipConvdirect,direct_l1anddirect_wgradkernels under ROCm 10.1’s full AMDGCN link-time optimization default by building the hipconv device targets with-fno-offload-lto.[Conv] Fixed a multi-node data-parallel training throughput regression on gfx942 by excluding the backward-pass GEMM solvers for single-group 1x1 stride-1 pad-0 NHWC convolutions with a batch size below 4.
[Conv] Fixed invalid Composable Kernel (CK) perf-config entries in the gfx1201 and gfx1100 system databases that MIOpen could not resolve, which caused a fallback to untuned defaults.
[BatchNorm] Fixed just-in-time compilation failures for the batch normalization kernels on non-GFX9 targets by restricting the Welford reduction’s DPP inline assembly to GFX9 instead of a per-architecture denylist.
Fixed an out-of-bounds write in
miopenLoadSolutionwhen a serialized solution carries more than three kernel work dimensions; such solutions are now rejected.Fixed a
uint64_ttypedef conflict betweenmiopen_cstdint.hppand the system<stdint.h>that broke runtime kernel compilation on LP64 Linux.[Conv] Fixed
TransposedConvWinoRageRxS<2,3>being unreachable: it was registered in the solver registry and explicitly instantiated but missing from both Find solver lists, so it could never be enumerated or selected for NHWC convolutions.
RCCL (2.32.3)#
Added#
Compatibility with NCCL 2.32.3.
Support for collective operations across up to eight AMD Instinct MI350P GPUs in a single node.
ncclSetEncryption()andncclEncryptionConfig_t(initialize withNCCL_ENCRYPTION_CONFIG_INITIALIZER) to encrypt NCCL-owned TCP socket traffic with TLS using a pre-shared key of at least 32 bytes. Requires a build configured with-DTLS_BACKEND=OPENSSL3;libssl.so.3is loaded at run time. Encryption is off by default and covers only traffic sent through RCCL’s TCP sockets, not RDMA or GPU-to-GPU transfers.launchCompletionEventin the per-collective config (ncclCollConfig_t, nowncclCollConfig_v23200): a caller-owned event that RCCL records for the collective’s kernel launch. Every rank must pass an event, or every rank must passNULL. HIP has no launch-completion launch attribute, so on ROCm the event is recorded on the stream immediately before the kernel launch.NCCL_WIN_GIN_ONLYwindow registration flag, which registers a window for GIN access only.GIN device-API C entry points for the timeout waits (
ncclGinWaitTimeout,ncclGinWaitSignalTimeout,ncclGinWaitSignalTimeoutVA,ncclGinWaitCounterTimeout,ncclGinFlushTimeout), which returnncclTimeoutaftertimeoutCyclesclock64()cycles, and forncclGinFlushAsync,ncclGinWaitand the VA-signal operations (ncclGinReadSignalVA,ncclGinResetSignalVA,ncclGinWaitSignalVA). The equivalent C++ncclGinmembers were already available.ReduceSum and ReduceCopy device APIs (
ncclLsaReduceSum,ncclLsaCopy,ncclLsaReduceSumCopy,ncclLocalReduceSumCopy) in the LLVM bitcode library (EMIT_LLVM_IR=ON,librccl_device.bc) for 8-, 32- and 64-bit integer, FP16, BF16, FP32 and FP64 types.NCCL_DEBUG_LEVELSand theATTNlog level.ATTNreports non-fatal conditions that need attention, such as configuration fallbacks and plugin initialization failures.NCCL_DEBUG_LEVELSadds individual levels (VERSION,WARN,ATTN,INFO,ABORT,TRACE) to the selection made byNCCL_DEBUG.Device progress counters (
NCCL_PROGRESS_COUNTERS, default0; also requiresNCCL_RAS_ENABLE=1, the default) mirrored to host memory, and a RAS progress monitor that warns when a communicator’s counters stop advancing (NCCL_PROGRESS_COUNTER_MONITOR_POLL_MS,NCCL_PROGRESS_COUNTER_MONITOR_STALE_MS,NCCL_PROGRESS_COUNTER_MONITOR_STALE_WARN_SEC).New RAS diagnostics checks (
NCCL_RUN_RAS_DIAGNOSTICS) for PCI topology (rdma_topo check), the IOMMU mode and groups and the ATS state of GPU/NIC pairs, and the path types each communicator uses.An IB write-bandwidth check in the communicator init diagnostics (
NCCL_RUN_DIAGNOSTICS). It runs perftest’sib_write_bwbetween hosts of multi-node communicators, with a per-run timeout set byNCCL_DIAGNOSTICS_IB_BW_TIMEOUT(default 5 seconds). It measures from GPU memory only when the installedib_write_bwsupports--use_cuda; otherwise it measures from host memory.Communicator initialization reports, at the
ATTNlevel, ranks that run RCCL builds from different git revisions.Inspector plugin ring-buffer drop counters: a single warning per process when events are dropped, a
dump_statsrecord with operation and drop counts in JSON output, andNCCL_INSPECTOR_PROM_DUMP_STATS(default0) to add the same counters to Prometheus output.NCCL_CE_CHUNK_SIZE(default 8 MiB): chunk size for the round-robin chunking that host RMA Copy Engine batches use when per-peer transfer sizes differ.NCCL_CE_INTRA_GPU_MEMCPY_ENABLE(default1): controls whether batched Copy Engine collective copies passhipMemcpyFlagPreferOverlapWithCompute. RCCL previously always set this flag, so the default keeps the previous behavior; set0to omit it.NCCL_HIER_CE_COLL_AG_RAIL_RING_ENABLE(default-1): a positive value selects a ring for the inter-node rail phase of hierarchical Copy EnginencclAllGather. The default keeps the direct path.NCCL_IB_SORT_MERGE_NICS: sorts the sub-devices of a merged IB device by plane ID. RCCL defaults it to0(NCCL defaults to1), so merged-device order and names are unchanged.nccl4py: per-call collective configuration (
NCCLCollConfig,VendorOption) including the launch completion event, communicator properties (NCCLCommProperties), and theGIN_ONLYwindow flag. On ROCm, the HIPEventshim provides only an event handle;record(),sync(), andquery()are not implemented.RCCL_TUNER_EMBEDDED_CONFIG(default1): the built-in CSV tuner configurations intuner/*.csvare now compiled intolibrccl.soand used as the stock tuning defaults, so a cluster no longer has to stage the CSV on every node or on NFS. Set0to ignore the embedded configurations.NCCL_TUNER_CONFIG_FILEand the on-disk tuner directories still take precedence, andNCCL_TUNER_PLUGIN=nonestill disables the tuner entirely.RCCL_CE_AR_MAX_MSG_BYTES(default-1): overrides the 2-shot AllReduce size cap from the arch table. Set to a positive value to overrideceNonRegMax[AR].RCCL_CE_AR_REG_MAX_MSG_BYTES(default-1): overrides the registered CE AllReduce size cap from the arch table.RCCL_CE_AR_STAGING_BYTES(default-1): overrides the CE AllReduce staging buffer allocation size; when unset,NCCL_CE_AR_STAGING_BYTESis used.Per-architecture dispatch table (
rcclArchThresholds) centralizing algo/proto selection for DDA protocol, CE, and symmetric kernel per collective, replacing scattered hardcoded defaults. Tables are defined for gfx1250, gfx950, and gfx942; the lookuprcclGetArchThresholds()maps GCN arch strings to the appropriate entry. ThesymMinR2[func]field sets a per-collective lower bound for registered buffers.RCCL_IGNORE_ARCH_TABLE(default0): when set to1, bypasses thercclArchThresholdsdispatch table and falls back to compile-time constants for all DDA, CE, and symmetric-kernel thresholds. Has no effect on gfx942 and gfx950, which always use the arch table.New
rcclAddonAlgos_tvalues:RCCL_CE_SCRATCH(CE via DDA scratch, distinct fromRCCL_CE_REGISTERED),RCCL_A2A_PIVOT,RCCL_A2A_GDA,RCCL_A2A_GIN_SDMA, andRCCL_DIRECT_ALLTOALL, with correspondingrcclGetAlgoName()labels.rccl-tests: AlltoAll now reports algo/protocol in VERSION output, consistent with other collectives.
Compatibility with NCCL 2.31.2.
net_ib_castrefreshes local GIDs afterIBV_EVENT_GID_CHANGEand applies them to data, flush, probing and port recovery QPs during IB port recovery, matchingnet_ibin NCCL 2.31.2.Per-collective configuration APIs (
ncclCollConfig_t/nccl*Config()entry points) and thencclConfigExt_tvendor extension list. Initialize configs withNCCL_COLLCONFIG_INITIALIZER.Communicator config (
ncclConfig_v23100) fields for implicit launch ordering (launchOrderImplicit), RMA signal count (numRmaSig), eager RMA init (rmaEagerInit), and host collective fault tolerance (hostCftMode).NCCL profiler plugin API v7, with per-call user profiler tags and symmetric-kernel phase events.
RAS diagnostics (
NCCL_RUN_RAS_DIAGNOSTICS) covering GPU inventory, ROCm runtime versions, ECC counters, XGMI link state, andNCCL_*environment consistency (AMDSMI in place of NVML).Communicator init diagnostics (
NCCL_RUN_DIAGNOSTICS) with an active P2P connectivity check.Multiple GIN proxy progress threads via
NCCL_GIN_PROXY_NTHREADS.Enabled hierarchical Copy Engine
ncclAllGather,ncclAlltoAll, andncclAlltoAllvon multi-node communicators: the intra-node phase runs over SDMA/CE across the LSA team and the inter-node phase over one-sided RMA through the CPU (GIN) proxy. The path is off by default and takes two opt-ins:NCCL_CTA_POLICY=2(equivalentlyconfig.CTAPolicy = NCCL_CTA_POLICY_ZERO), and symmetric registration of both the send and the receive buffer throughncclCommWindowRegister. On architectures other than gfx1250 a third is needed,NCCL_CUMEM_ENABLE=1, because symmetric memory depends on cuMem and cuMem auto-detection enables itself only on gfx1250.NCCL_GIN_ENABLE,NCCL_NUM_RMA_CTX,NCCL_WIN_ENABLEandNCCL_DMABUF_ENABLEneed no change, since their defaults already qualify; the inter-node rail draws from the internal RMA context pool (NCCL_NUM_RMA_INT_CTX, default 4) rather than the user-addressable contexts, sonumRmaCtxonly has to stay above zero. Selection also requires an available GIN backend and the same number of ranks on every node; communicators that do not qualify keep using the kernel path.ncclAlltoAllvwalks the gathered per-peer size matrix, skips zero-length peers, and chunks each inter-node put; single-node AlltoAllv stays on the existing CE path. Validated for correctness and path selection from 1 to 32 nodes.Added
ncclGinFenceLevelbarrier semantics on the Anvil SDMA GIN backend (NCCL_GIN_TYPE=7).gin.get,flushAsyncandwaitare implemented instead of trapping, and a barrier’s signal is now ordered behind the payload on the peer’s SDMA queue, soPut,Getand the defaultPut | Getfences drain what they promise.Putcovers a rank’s puts to itself, andncclGinAllContextsfences every GIN context. Single-node validated on gfx950.Added gfx1250 Tensor Data Mover staging to the symmetric AllGather, AllReduce and ReduceScatter kernels. Their deep loops move each tile through the warp’s shared-memory scratch with the DMA engine instead of per-lane vector loads; on ROCm that path previously compiled out entirely, so the
Tmakernels were never generated. Launch tuning was also corrected so these kernels can use up to a quarter of a GPU’s WGPs, where only 14 were reached before. Opt-in viaNCCL_SYM_TMA_ENABLE=1, and reached only when the application opts into symmetric memory.NCCL_SYM_TMA_ENABLE=2additionally forces these kernels wherever the collective has one, skipping both the tuner’s cost comparison and the deep-loop size bar, which is intended for A/B measurement rather than production. The new kernels are generated only when gfx1250 is inGPU_TARGETS, and every other architecture keeps the vector path unchanged.
Changed#
Removed GIN rocSHMEM GDA dependency on rocSHMEM GDA bitcode: GIN rocSHMEM GDA is now header-only. The CMake
roc::rccltarget provides the required include directories in itsINSTALL_INTERFACEproperty.Host-side device API declarations (
ncclDevCommCreate,ncclDevCommDestroy,ncclCommQueryProperties,ncclGetPeerDevicePointer, the*CreateRequirementhelpers, thencclDevCommRequirementsandncclCommPropertiesstructs and their initializers) moved tonccl_device/host.h.nccl_device.hstill includes it; code that includes individualnccl_device/*.hheaders directly must also includenccl_device/host.h.Profiler
ncclProfileKernelPhaseevents are emitted only by symmetric kernels, matching NCCL 2.32. Regular and P2P kernels report onlyncclProfileKernelChstart and stop.Updated the RMA plugin interface to v16, which adds a per-communicator
getRmaPropertiesquery. Plugins built against v13, v14 or v15 still load.NCCL_DEBUG_TIMESTAMP_LEVELSnow timestampsATTNmessages as well asWARNmessages by default.The internal IB RMA proxy chains work requests and rings the doorbell once per batch of up to
NCCL_RMA_IB_WR_BATCHSIZE(default64) requests, which raises the small-message rate.Communicator initialization at large scale is faster because the proxy no longer scans inactive poll descriptors.
Network devices for each GPU are now chosen by rail and plane assignment, replacing the previous start-device scattering.
NCCL_MLOPART_RDMA_ENABLE(default0) is kept. NCCL 2.32 removed it and treats buffers on partitioned GPUs as RDMA-capable on all non-ARM hosts; RCCL keeps network buffer registration for partitioned (CPX/DPX) GPUs opt-in.The CSV tuner configurations are no longer installed to
${ROCM_PATH}/share/rccl/tuner/or copied into the build tree; they ship insidelibrccl.so. A CSV placed in that directory, or in<librccl.so dir>/tuner/, still overrides the built-in defaults. A copy left behind by an earlier build or bycmake --installover an existing prefix also still takes precedence, so delete it to pick up the embedded defaults; package upgrades remove it automatically.Retuned the symmetric AllReduce, ReduceScatter and AllGather kernels for gfx950. The block width is now selected per collective and message size instead of a fixed 256 threads, and the work partitioning and unroll factors in the load-direct kernels and the AllGather store kernel were refitted to the 64-lane wavefront. AllGather also switches from LL to the store kernel from 4 MB instead of 16 MB. The multi-node GIN kernels and all other architectures are unchanged. To fit the wider LL blocks, the shared LL scratch buffer on gfx950 grows from 4 MiB to 8 MiB per 8-rank communicator.
gfx1250 no longer forces unregistered
ncclAllGather,ncclAlltoAll,ncclGatherandncclScatterthrough the Copy Engine scratch path (CE-Scratch), because it was slower than the kernel paths it replaced. Whether CE-Scratch is forced by default now comes from the per-architecture table: gfx942 and gfx950 still force it. A communicator-levelNCCL_CTA_POLICY_ZEROstill forces it on every architecture; an explicitRCCL_FORCE_CE=1still forces it andRCCL_FORCE_CE=0still turns it off. gfx1250ncclAllGathernow uses DDA LL128 up to 128 MiB.gfx1250
ncclAllReduceon single-node communicators keeps the LL128 protocol at larger sizes: Tree at every size, Ring up to nRanks x 32 MiB, and the DDA LL128 tier up to 256 MiB. The DDA LL128 two-shot ceiling now follows that tier ceiling (RCCL_DDA_LL128_TWOSHOT_THRESHOLDstill overrides it); the DDA scratch slot lets two-shot reach 256 MiB from 32 ranks, about 240 MiB at 16 and 128 MiB at 2 to 8. On gfx1250 communicators of 2 to 8 ranks, the DDA fabric scratch buffer grows to about 546 MiB.On gfx1250 single-node communicators of 8 or more ranks, the DDA tier, algorithm and protocol crossovers now scale with the rank count. Communicators below 8 ranks keep the previous thresholds.
ncclAllReduce: DDA LL two-shot up to nRanks x 1 MiB (RCCL_DDA_LL_TWOSHOT_THRESHOLDstill overrides it); from 32 ranks, DDA VMM between 2 MiB and nRanks x 512 KiB for unregistered buffers unlessRCCL_DDA_THRESHOLDorRCCL_DDA_LL_THRESHOLDis set; Tree instead of Ring up to nRanks x 1 MiB unlessNCCL_ALGOorNCCL_PROTOis set, with LL up to nRanks x 32 KiB.ncclAllGather: DDA LL up to nRanks x 16 KiB, DDA LL128 up to 256 MiB from 16 ranks, and Ring LL128 up to 64 MiB per rank.ncclReduceScatter: DDA LL up to nRanks x 256 KiB, DDA LL128 up to nRanks x 512 KiB, then DDA VMM up to nRanks x 2 MiB; Ring LL up to 512 KiB per rank and LL128 up to 8 MiB per rank (32 MiB for 1-byte types); Ring Simple on at most 128 channels between 8 and 32 MiB per rank unlessNCCL_MIN_NCHANNELSorNCCL_MAX_NCHANNELSis set.DDA VMM
ncclAllReduceandncclReduceScatterlaunch 2048 / nRanks blocks, clamped to 32 to 128 and never more than before.
Vendored
{fmt}10.2.1 as a header-only tree underexternal/fmt. RCCL, its tests, and thetopo_expltool now always build against this copy instead of looking for a systemfmtpackage or downloading one throughFetchContentorgit clone.The
rcclSelectAllGathersignature changed. It now takes acudaStream_t streamparameter (for live dispatch) and abool querymode that usesgraphCapturingHintinstead of probing the stream.RCCL_DDA_THRESHOLD,RCCL_DDA_LL_THRESHOLD, andRCCL_DDA_LL128_THRESHOLDnow default to-1(unset), resolved at runtime from the arch dispatch table instead of being hardcoded to 128 MiB and 64 KiB respectively (RCCL_DDA_LL128_THRESHOLDhad no pre-table default; the LL128 tier was off by default). Setting these environment variables explicitly still takes precedence over the table.RCCL_DDA_LL128now defaults to-1(auto): the LL128 tier is enabled only for architectures whose arch-table row has a non-zeroddaLL128Maxfor the collective (currently gfx1250 only).RCCL_CE_ALLREDUCEnow defaults to-1(auto/enabled) instead of0(disabled). CE AllReduce is therefore on by default for gfx1250 communicators that meet all other eligibility criteria.CE-2-Shot AllReduce size cap is now resolved at runtime via
rcclCeAr2ShotMax()(env var wins, then arch tableceNonRegMax[AR], then the 256 MiBNCCL_CE_AR_TMPBUF_DEFAULT_BYTESfallback) rather than from a compile-time constant.RCCL_FORCE_CE_ALLREDUCE=1no longer overrides the CE AllReduce 2-shot selector cap. It bypasses theCTA_POLICY_ZEROcheck only; the selector cap is enforced regardless.Registered
tuning_model_11for gfx1250 single-node and added gfx1250 to the AINIC tuning index map.Extended
ncclFunc_tenum in rccl-testscommon.hto match the librccl internal ABI: renumbered existing values and addedncclFuncAlltoAll(8),ncclFuncScatter(9),ncclFuncGather(10),ncclFuncAllToAllPivot(11),ncclFuncAlltoAllGda(12),ncclFuncAlltoAllvGda(13),ncclFuncAllGatherV(14),ncclFuncPutSignal(15),ncclFuncSignal(16),ncclFuncWaitSignal(17),ncclFuncAlltoAllv(18);ncclNumFuncsupdated to 19.AlltoAll index variables in
alltoall_dda.handalltoall_dda_fabric.hwidened frominttosize_tto prevent overflow on large message sizes.DDA fabric and IPC AlltoAll launchers refactored to derive launch geometry from a single inline helper (
ddaAllToAllFabricGeom,ddaAllToAllFabricLLGeom,ddaAllToAllFabricLL128Geom,ddaAllToAllIpcGeom) shared between the dispatch path and the new block-count query functions.rcclGetCollImplInfonow covers ReduceScatter and AlltoAll (previously both fell back torcclGetAlgoInfo()); rccl-tests reports accurate algo/proto for all four collective types.Algorithm selectors (
rcclSelectAllReduce,rcclSelectAllGather,rcclSelectReduceScatter,rcclSelectAlltoAll) now tag the task withrcclSymkExtract(ALLOW/DENY) sotaskAppenddoes not re-derive the symmetric kernel decision. SettingNCCL_ALGO=Ringnow correctly prevents the symmetric kernel from being extracted even when both buffers are registered.Breaking:
NCCL_GIN_TYPEvalues for AMD backends are not compatible with 2.30.7. NCCL 2.31 inserted EFA GDA at value 5, so rocSHMEM GDA moved 5→6 and Anvil SDMA moved 6→7. The IB proxy remains2. Jobs that still setNCCL_GIN_TYPE=6now select rocSHMEM GDA, not Anvil SDMA. Seesrc/gin/README.md.One-sided RMA supports multiple contexts and signals; the previous restriction to context 0 and signal index 0 has been lifted (
numRmaCtx/numRmaSig).Reduced communicator host memory by allocating topology path link arrays to their actual length.
Widened the LL128 per-thread shared-memory slice from 8 to 32 elements on gfx1250; other architectures are unchanged. Forced Ring/LL128 bandwidth increases at 16 MiB and above. Per-block LDS rises from 36448 to 85472 bytes, and because the scratch region is shared across protocols this applies to every kernel launch, not only LL128 ones. The tuner’s protocol selection is unchanged, so the gain is only visible when LL128 is selected explicitly.
Removed#
NCCL_TOPO_SCATTER_START_NET, which selected how the first network device was scattered across GPUs. Rail and plane assignment replaces it.
Resolved issues#
Fixed profiler overhead when no profiler plugin is loaded or only kernel-channel events are enabled.
Fixed profiler API events reporting rank 0 instead of the originating communicator rank.
Fixed P2P IPC registration reuse producing out-of-bounds remote addresses when a registered allocation spans multiple cuMem segments.
Fixed non-thread-safe token parsing that could corrupt configuration parsed concurrently from multiple threads.
Fixed tuner plugins receiving uninitialized cost-model constants.
Fixed GIN proxy descriptor shared-memory sizing and alignment.
Fixed virtual address space exhaustion when symmetric windows backed by the same physical allocation are registered repeatedly.
Fixed Copy Engine
ncclAlltoAllvhanging on intra-node communicators of more than 8 ranks with uneven send sizes. Each rank decided from its own send sizes whether to add intra-batch barriers, so ranks could issue different barrier counts. The decision now uses the full send-size matrix, which every rank shares.Fixed
NCCL_P2P_DISABLEandNCCL_P2P_LEVELbeing ignored on Intel, ARM and Zhaoxin hosts, where the CPU-based default P2P level overwrote the user setting. On such hostsNCCL_P2P_DISABLE=1now falls back to SHM as documented, andNCCL_P2P_LEVELis honored.Fixed intermittent data corruption in multi-node collectives, seen as out-of-bounds values in 8-node AllToAll, on hosts that use GPU Direct RDMA through peermem. Since RCCL 2.30.4 the GDR flush scratchpad was registered only through dma-buf, so on peermem hosts the flush degraded to a single RDMA read of the receive buffer, which does not order earlier relaxed-ordering writes. The scratchpad is now also registered through peermem, restoring the strictly ordered RDMA write that precedes the flush read.
Fixed FP8 E4M3 and E5M2
ncclAvgreturning wrong values for every element of every reduction collective on gfx942, and on architectures that use the software FP8 fallback. The1/nRanksscale was packed as an FP8 value in the host’s OCP encoding, which those devices decode as FNUZ, so every result came out half as large. The scale is now carried asfloaton all architectures, which also keeps it exact at rank counts FP8 cannot represent: in OCP E4M3,1/384was off by 25% and1/1024rounded to zero. Single-rank FP8 reductions on gfx942 no longer fail to launch theoneRankReducekernel, and rccl-tests no longer skips FP8avgforall_reduce,all_reduce_biasandreduce.Restored topo tuning-model init (
ncclTopoTuneModel) after the 2.31ncclTuningInitswitch so multi-node kernels do not launch withblockDim.x=0.Aligned grouped multi-rank finalize with the v2.31 teardown barrier.
Fixed Copy Engine
ncclAllGather,ncclAlltoAll,ncclScatterandncclGatherreturning incorrect data, and no error, whenNCCL_LSA_TEAM_SIZEwas set below the number of ranks on the node. These routines index both their peer list and their buffer offsets by LSA rank, so a communicator wider than its LSA team exchanged data only within that team and placed it at LSA rather than communicator offsets, leaving the slices owned by every other rank unwritten. Copy Engine selection now requires the LSA team to cover the whole communicator on the scratch (RCCL_FORCE_CE) path as well as the registered-window path, so such communicators use the kernel path instead. Runs that leaveNCCL_LSA_TEAM_SIZEunset are unaffected, because the LSA team then spans every rank on the node.Fixed hierarchical Copy Engine collectives dereferencing a NULL RMA proxy context when
NCCL_RMA_DISABLE=1was combined with the hierarchical opt-ins.NCCL_RMA_DISABLEsuppresses the window registration that creates those contexts, but the hierarchical availability check did not consult it, so a multi-node communicator withNCCL_CTA_POLICY=2and symmetric send and receive windows admitted the path and then crashed at launch. The internal RMA context predicate now defers toncclRmaProxyEnabled, so such a communicator declines the hierarchical path and uses the kernel path instead.Fixed Copy Engine
ncclAllReducehanging when one thread drives several GPUs, as in rccl-tests-g 4. The staging buffer was set up on first launch, and that setup is a collective window registration, so the first rank waited for peers the same thread had not launched yet. The setup is now queued when the collective is enqueued and runs in each communicator’s own group job atncclGroupEnd.Fixed
ncclCommWindowRegisterafterncclDevCommCreateaborting on the GIN Anvil SDMA backend (NCCL_GIN_TYPE=7) withcould not resolve LSA flat addr. Symmetric-window memory is now linked onto the device-runtime list before GIN/RMA registration so plugins that resolve the user VA through that list can see the in-flight window.Fixed the DDA, Copy-Engine 2-shot, GIN SDMA and hierarchical AllGather/ReduceScatter backends not making the communicator’s device current and not ordering a new stream against the previous collective. A collective issued while another device was current failed with
ncclUnhandledCudaError, and consecutive collectives on different streams could overlap.
Known issues#
The FP8 ReduceSum and ReduceCopy device APIs are not exported in the LLVM bitcode library.
On the Anvil SDMA (
NCCL_GIN_TYPE=7) and rocSHMEM GDA (NCCL_GIN_TYPE=6) GIN backends,ncclGinFlushTimeoutignores the timeout and blocks until the flush completes;ncclGinWaitTimeoutdoes the same on Anvil SDMA. rocSHMEM GDA does not supportncclGinFlushAsyncor waiting on a GIN request, and those calls trap on the device. The signal and counter wait timeouts work on all backends.The following NCCL 2.31 features are not available in RCCL: Compute Fabric Transport, the GDAKI and EFA GDA GIN backends, PAT combined with NVLS, NVLink-multicast AllGather, and the CuTeDSL and
nccl4rustbindings.
rocBLAS (5.8.0)#
Added#
Enabled gfx1170 and gfx1171. Level 3
gemm,gemm_ex, and the functions that internally use GEMM default to the hipBLASLt backend on these architectures.ROCBLAS_USE_HIPBLASLTcontinues to force or disable the hipBLASLt backend.
Changed#
On gfx1250, rocBLAS loads the Tensile library catalog that matches the device revision. Strict-revision silicon uses
library/gfx1250-strictwhen that catalog is present in the build, and otherwise falls back tolibrary/gfx1250.On gfx950, hipBLASLt is now the default GEMM backend for every input type except real
f32_r. Real single precision still defaults to Tensile.ROCBLAS_USE_HIPBLASLTcontinues to force or disable the hipBLASLt backend.On gfx942, double-precision Level 3
gemm,gemm_ex, and the functions that internally use GEMM now default to the hipBLASLt backend. gfx942 devices other than the MI300A (gfx942 with 228 compute units) also default to hipBLASLt forf16_r/bf16_rdata, while the remaining data types on gfx942 default to Tensile.ROCBLAS_USE_HIPBLASLTcontinues to force or disable the hipBLASLt backend.Batched GEMM now uses the same hipBLASLt selection as other GEMM while a stream is capturing. Previously, batched APIs were forced to the Tensile backend during stream capture.
If the new
ROCBLAS_VERBOSE_ERRORenvironment variable is set,trsm,trsm_batched, andtrsm_strided_batchedprint the required and available workspace sizes when they returnrocblas_status_memory_errorbecause a fixed-size handle workspace is too small. Previously this failure looked identical to running out of device memory.
Removed#
The
ROCBLAS_USE_HIPBLASLT_BATCHEDenvironment variable, deprecated in rocBLAS 5.6.0, is removed and is now ignored. Batched GEMM is controlled by the same environment variable as all other GEMM, so a batched-only override is no longer required. UseROCBLAS_USE_HIPBLASLT=0to select the Tensile backend for all GEMM, including batched.
Optimized#
Improved the performance of Level 2
gemvnon-transposed (TransA == N) by selecting the n-split reduction from the launch shape rather than a fixed output-length crossover. The split applies when the output grid has at most 8 tiles and the column split produces at least 2 parallel blocks, or when the output length is at or below the crossover.Improved the performance of Level 3
gemmandgemm_exfor transposed operands with a very large summation dimension. k-chunking, which keeps Tensile element offsets inside 32 bits, previously usedmax(lda, ldb)for every transpose combination, so a unit-stride summation was split into many small launches. The bound is now the k stride of each operand: the leading dimension when that operand is k-major, and 1 otherwise. k-major layouts that can overflow are still chunked.
Resolved issues#
Fixed incorrect results and out-of-bounds reads from Level 1 ILP64
dotanddotc, including batched, strided-batched, and_exforms, when a negative increment is wide enough to take the 64-bit increment path andnfits the single-block reduction. That path shifted by(n - 1)before calling the launcher, which applies the same walk, and theyshift testedincxrather thanincy. The offsets are now passed through unshifted.Fixed incorrect results and out-of-bounds indexing from Level 1 ILP64
axpyandaxpy_ex, including batched and strided-batched forms, whenbatch_countis greater than 65535 and an increment is wide enough to take the 64-bit kernel. That kernel was launched with the fullbatch_countrather than the chunk length, so each pass walked past the current batch and appliedalpha * xagain.Fixed incorrect results and out-of-bounds accesses from Level 1 ILP64
rotandrot_ex, including batched and strided-batched forms, whennorbatch_countis large enough to chunk the launch andincxorincyis negative. Chunk offsets weren_base * inc, and the wide-increment path used-inc * n_base, so a negative increment started at the wrong end of the vector. Each chunk now starts atoffset + (inc < 0 ? -inc * (n - n_chunk - n_base) : n_base * inc).Fixed Level 2
trsvand Level 3trsm,trtri,trmm, andgemmtreturningrocblas_status_successwhen an internal GEMM, GEMV, vector copy, or block helper failed. Those statuses are now returned to the caller. On the negative-incxpath, a failedtrsvalso restores the caller’s input vector, which the solve reverses in place.Fixed
rocblas_get_commit_hash_stringwriting past the caller’s buffer. The length check compared against the size of a pointer, so a buffer shorter than the commit hash was accepted, and the copy omitted the terminating null.rocblas_get_version_stringandrocblas_get_commit_hash_stringnow require the size reported by the matching*_sizequery, including the terminating null.
ROCdbgapi (0.82.1)#
Added#
Support for the following architectures:
gfx1170gfx1171gfx1172gfx11-7-generic
ROCm Data Center Tool (1.4.0)#
Resolved issues#
The
RDC_FI_GPU_MEMORY_CUR_BANDWIDTHfix for zero readings under DMA or copy-only memory traffic is now included.rdci dmonandrdci health -sname the missing group again, for exampleCannot find the group 4242. Since ROCm 7.12 they printed onlyGroup not found(orFieldgroup not foundfor a missing field group inrdci dmon).
rocDecode (1.12.0)#
Added#
Initial Windows support: hardware-accelerated video decoding on Windows via the vaon12 backend (Mesa’s VA-API on D3D12 translation layer). Requires HIP and the vaon12 VA-API driver from TheRock for Windows.
On Windows, the TheRock VA-API DLLs (
rocm_sysdeps_va_win32.dll,rocm_sysdeps_va.dll) are loaded at run time, matching the Linuxdlopenbehavior.%ROCM_PATH%\lib\rocm_sysdeps\binno longer needs to be onPATH.HEVC: generation of unavailable reference pictures for a BLA picture or a CRA picture with
NoRaslOutputFlagequal to 1 (H.265 clause 8.3.3).
Resolved issues#
Fixed a stray RASL picture being output from the first CRA picture after an application signals end of sequence with the
ROCDEC_PKT_ENDOFSTREAMpacket flag. The flag now ends the coded video sequence the same way an in-band EOS NAL unit does, so the next IRAP picture getsNoRaslOutputFlagequal to 1.Fixed out-of-bounds memory access issues in the HEVC, AVC, AV1, VP9, and base parsers.
Fixed
rocDecGetDecoderCapsadvertising 16-bit output surface formats for 8-bit content.
rocFFT (1.0.41)#
Added#
Added further support for very large FFTs (length greater than 2^32).
JIT callbacks specified with the
rocfft_plan_description_set_load_callbackandrocfft_plan_description_set_store_callbackAPIs can now be used with transforms that have fields specified on the same plan description.
Changed#
Function pointer callbacks specified with the deprecated
rocfft_execution_info_set_load_callbackandrocfft_execution_info_set_store_callbackAPIs are now disallowed with transforms that have fields specified on the plan description.
Optimized#
Improved performance of unit-strided, interleaved, complex-to-real FFTs on gfx1201, gfx90a, gfx942, and gfx950 for the following lengths:
(100,100,100)
(192,96,96)
(200,96,96)
(128,128,256)
(160,168,168)
(160,168,192)
(168,168,192)
(168,192,192)
(192,192,192)
(192,192,200)
(192,200,200)
(200,200,200)
(216,216,216)
(216,104,104)
(216,104,100)
(224,104,104)
(224,108,104)
(224,108,108)
Resolved issues#
Fixed possible failures of
rocfft_plan_createfor multi-device plans.Fixed out-of-bounds memory access when executing a multi-device complex-to-complex plan whose bricks used non-contiguous data layouts.
Fixed a memory leak when
rocfft_plan_createfails.Fixed
rocfft_plan_createfailures for large prime-length 1D complex transforms when a scale factor is set.Fixed a potential write-after-free if plans are destroyed during process teardown.
Fixed a possible hang in
rocfft_cleanupin builds with RCCL enabled, for single-process multi-device usage.Fixed incorrect results for 1D complex transforms of lengths involving large prime factors with large batch sizes, where 32-bit kernel indexing could overflow.
Fixed
rocfft_plan_createfailures for 1D complex transforms of lengths involving large prime factors when the output data layout uses a batch distance larger than the transform length.Fixed
rocfft_plan_createfailures and GPU memory access faults for 1D complex transforms of lengths involving large prime factors when batches are interleaved (unit batch distance).
rocJPEG (1.13.0)#
Added#
Added the
rocJpegGetDecoderCapsAPI and theRocJpegDecodeCapsstructure to query the decode capabilities of a rocJPEG handle, including the number of JPEG cores, the supported JPEG coding processes, image resolutions, chroma subsamplings, and output formats, the native output surface format of each chroma subsampling as verified with the VA-API driver, and the hardware ROI decode and color conversion support.Added the
RocJpegSurfaceFormatenum to describe the layout of the decoded image produced by the VCN JPEG decoder, and theRocJpegCodingProcessenum to report the supported JPEG coding processes.Added the
jpegDecodeCapssample to demonstrate therocJpegGetDecoderCapsAPI.
ROCm Compute Profiler (3.10.0)#
Added#
Added the
LDS Utilizationmetric to the gfx115x Memory Chart.Added two wave utilization metrics to PC sampling analysis.
active_thread_percentis the percent of a wave’s lanes that were active at an instruction, so a low value points at control flow divergence. Both sampling methods report it.wave_occupancy_percentis the percent of the machine’s wave slots that held a wave. Only stochastic sampling reports it, because a host-trap record carries no wave count.Both appear in the analyze terminal table and in each kernel’s
per_kernel_pc_sampling/CSV.
Added the two wave utilization metrics to the analysis database summary view, so
compute_pc_sampling_summary_viewand thepc_sampling_summary.csvexport carry them alongside the sample counts.Added a profile-mode warning on gfx115x when the
AUTOperformance level can gate the perfmon clock and zero PMC counters such asTCP_REQ, with a link to the ROCprofiler-SDKSTABLE_STDworkaround.Added CLI guidance for viewing the wide memory chart without line wrapping (
less -RSorcode -).Added Python 3.14 support.
Added an analyze-only install path.
pip install "rocm-profiler[compute-analyze]"into a virtual environment installs the analysis dependencies for you.Added
rocprof-compute analyze --verify-deps, which reports any missing analysis dependencies.Added Memory Bandwidth Analysis for gfx950 under
--experimental --membw-analysis. Profile with the flag to collect block 30 counters;analyzedetects the data automatically and annotates active GL1/GL2/EA stall bottlenecks on the memory chart with a guidance panel.Added the rocprof-compute Agent Skills and the documentation they link to to the install under
share/rocprofiler-compute/, with aninstall-skills.shscript that installs or removes them for Claude Code, Codex, or Cursor.rocprof-compute --helpshows where to find them.
Changed#
Dispatch IDs now start at 1 instead of 0.
Renamed the profile-mode dispatch filter to
--kernel-iteration-range, matching the rocprofv3 option it drives. Update any profile command by replacing-d/--dispatchwith--kernel-iteration-rangeto select dispatches.--dispatchis no longer accepted in profile mode.-dis now the short form of--output-directoryin profile mode, also matching rocprofv3.
gfx115x Memory Chart improvements.
Renamed memory chart metric names for more clarity.
Each edge now reports the traffic measured at the interface it represents.
Updated arrows, labels, and the legend in the memory chart to better represent their meaning.
The Torch trace collector now ships prebuilt for PyTorch 2.13 and 2.14, so
--torch-tracetraces operators on every thread, including the autograd threads that run the backward pass. Other PyTorch versions use a reduced trace that misses backward-pass operators.Named each per-kernel PC sampling folder
<short_name>_uuid_<kernel_uuid>instead ofkernel_<kernel_uuid>, and added the matchingshort_namecolumn tokernel.csv. The short name is the demangled identifier captured while profiling. Update any scripts that match the oldkernel_prefix.Redesigned the CDNA (gfx9) Memory Chart with a new Rich-based layout that improves readability in the terminal. Added Non-buffer/Buffer request breakdowns (Read/Write/Atomic wavefronts) and L2-Fabric bandwidth metrics across all CDNA architectures.
gfx908–gfx942: added HBM and remote traffic percentages.
gfx950: added LDS Read/Write/Atomic instruction counts and per-channel bandwidth for HBM, xGMI, and PCIe.
Analyze mode now auto-detects memory bandwidth analysis data from the profiling output. The
--membw-analysisanalyze option has been removed; use--membw-analysisonly at profile time.Renamed gfx1250 metric labels for clarity.
VMEM Atomic RTNtoVMEM Atomic ReturnandVMEM Atomic NORTNtoVMEM Atomic Non-Returnin the WGP VMEM Instruction Mix panel.LDS Atomic RTNtoLDS Atomic ReturnandLDS Atomic NORTNtoLDS Atomic Non-Returnin the WGP LDS Instruction Mix panel.Total Requests - SectorstoTotal GL0 Sectors,Total Requests - Sector ReadstoGL0 Read Sectors, andTotal Requests - Sector WritestoGL0 Write Sectorsin the GL0 Cache and LDS panel.
Removed#
Removed the
--kernel-verboseanalyze option and the kernel name shortener it drove. The option had no effect on any output.Removed the Nuitka standalone binary build (
STANDALONEBINARY,STANDALONEBINARY_EXTRACT_DIR), its RHEL 8 docker recipe, and the--call-binarypytest option that exercised it.Removed the
SKIP_NATIVE_TOOL_BUILDbuild option. The counter collection tool is always built, and its sources are no longer installed for runtime compilation.Removed the deprecated
Active CUsmetric from the System Speed-of-Light panel and the Memory Chart SVG for all CDNA architectures (gfx908, gfx90a, gfx940, gfx941, gfx942, gfx950). UseCU Utilizationinstead.
Optimized#
HBM and remote traffic percentages are now more accurate, with all their counters collected in a single profiling pass.
Analyze mode produces less warning noise. Repeated warnings are de-duplicated, and messages for metrics that evaluate to N/A moved to debug level.
Analyze mode uses less disk space. It reads the profiling output directly instead of writing an extra merged copy of all counter data into the workload directory.
Analyze mode now stops as soon as it finds unreadable profiling data, instead of continuing with an incomplete set of counters.
Improved the profiling failure message when the workload and the profiler load different ROCm installations. The error now points to the PyTorch and
rocm[profiler]install instructions instead of only showing the LLVM abort.
Resolved issues#
Fixed the standalone roofline HTML so it always opens with the same axes for a given GPU, which makes two runs comparable. The axes come from the benchmarked bandwidth and compute ceilings, not from the kernels in the run.
Fixed
L2 Cache (per Channel)labels to use aMetriccolumn and numberedChannelrow labels in CLI, TUI, and analysis database output.Fixed
--setrunning the roofline microbenchmark, which is never part of a metric set.Fixed PC sampling source snapshots to use canonical paths and include source contents and checksums in analysis exports.
Fixed false
0values in the gfx115x Memory Chart; missing counter data now reportsN/A.Fixed
GL2-Fabric Write BWunderstating write bandwidth on gfx115x in the System Speed-of-Light and Memory Chart panels.Fixed
profile -b 3 --experimental --membw-analysisnot collecting memory bandwidth analysis counters.Fixed
analyze -bcrashing when only the gfx950 LDS bandwidth metrics are selected.
Known issues#
On gfx115x,
TCP_REQ*counters and theGL0metrics derived from them can read zero because the perfmon clock is power-gated at theAUTOperformance level.
Upcoming changes#
The analyze-mode
--guiand--tuioptions are deprecated and will be removed in a future release. Use the default CLI analysis instead.
ROCgdb (16.3)#
Added#
Support for workgroup clusters. GDB now includes cluster coordinates in the target ID of a GPU thread when the dispatch is running in cluster mode. The
info dispatchcommand also shows cluster dimensions.Support for the following architectures:
gfx1170gfx1171gfx1172gfx11-7-genericgfx1250-strict
Resolved issues#
Fixed ROCgdb hiding a
SIGSEGVand showing a breakpoint hit instead when a GPU thread caused theSIGSEGV(which is reported imprecisely by default) and also hit a breakpoint. ROCgdb now prioritizes and shows theSIGSEGV.Fixed incorrect architecture selection when calling address dependency queries on generic AMDGPU code objects running on a concrete-architecture wave.
Fixed a broken HIP documentation link in the GDB help text.
Fixed the precise-memory fault location test to account for XNACK mode when determining the expected stop address.
ROCm Systems Profiler (1.10.0)#
Added#
rocsysunified CLI entry point.rocsys -- ./appandrocsys profile -- ./appproduce a full trace profile.rocsys rewriterunsrocprof-sys-instrumentin binary-rewrite mode (-o). Other subcommands (instrument,causal,avail,python,attach) forward to the existing tools. Therocprof-sys-*binaries are unchanged.AMD Infinity Storage I/O telemetry. Per-GPU I/O counters are sampled into the profiler output. Enable collection with
ROCPROFSYS_USE_HIPFILEand select metrics withROCPROFSYS_HIPFILE_METRICS. See hipFile Infinity Storage I/O telemetry.--detach-after <seconds>flag onrocprof-sys-attachfor non-interactive use; the binary sleeps for the given duration after attach instead of waiting forENTERon stdin.rocprof-sys-attachfinds the tool library in the target’s own ROCm installation when the target cannot see this installation’s library (for example, inside a container), soROCPROF_ATTACH_TOOL_LIBRARYis no longer needed there.
Removed#
Legacy direct Perfetto tracing mode. Perfetto traces are now always generated from the trace cache.
ROCPROFSYS_TRACE_LEGACYand--trace-legacy/-Lwere removed (passing-Lis now an unknown-argument error); thetracing.legacykey was removed from JSON configuration export.Timemory component annotations (for example
thread_cpu_clockand PAPI counter values) are no longer attached to Perfetto slices. They will be available through the timemory output formats.
Resolved issues#
Fixed a crash (
SIGSEGV) inrocprof-sys-instrumentwhen instrumenting Fortran+HIP programs compiled withgfortran -gor-g -O0. gfortran encodes assumed-shape and allocatable array bounds as DWARF location expression blocks rather than integer constants at debug optimization levels; DynInst was treating these block-form attributes as errors and dereferencing a null pointer inparseSubrange(). Fixed in ROCm/dyninst#33.Fixed
rocprof-sys-attachaborting the target process when attaching to a process running in a different mount namespace (for example, a container). The tool library path is now validated against the target’s mount namespace before attaching, failing cleanly with a diagnostic instead.
rocPRIM (4.8.0)#
Added#
gfx1250-strict support.
Changed#
Replaced the soon-to-be-deprecated
__hip_atomic_*builtins with the equivalent__scoped_atomic_*builtins.
Optimized#
Updated existing
ordered_block_idandlookback_scanoptimizations to include gfx1250/gfx1250-strict.
Resolved issues#
Fixed incorrect categorization of RDNA4 and CDNA5 GPUs.
ROCprofiler-SDK (1.4.2)#
Added#
Device-wide counter sampling (SPM)
Device counting mode for the Streaming Performance Monitor (SPM) service (experimental). New
rocprofiler_configure_spm_device_counting_servicesamples SPM counters on an agent for as long as the context is active, without tying samples to individual kernel dispatches.Samples are delivered to a buffer as
rocprofiler_spm_counter_record_trecords with no dispatch information.SPM support for gfx12 and gfx1250. gfx1250 SPM samples report the shader array and WGP. Both are encoded in the counter instance ID together with XCC, shader engine, and instance.
New
ROCPROFILER_SPM_PARAMETER_TYPE_SAMPLE_INTERVAL_REFCLK_CYCLESparameter sets the SPM sample interval in reference-clock cycles. Only gfx1200, gfx1201, and gfx1250 support it; other agents returnROCPROFILER_STATUS_ERROR_INVALID_ARGUMENT. Agent configuration queries report shader-clock and reference-clock intervals separately.--spm-sample-interval-unitaccepts the newrefclk_cyclesvalue, which sets--spm-sample-intervalin reference-clock cycles. It works only on gfx1200, gfx1201, and gfx1250.
Thread trace
Thread trace shader data now has timestamps.
rocprofiler_thread_trace_shader_data_thas newstart_timestampandend_timestampfields of typerocprofiler_thread_trace_timestamp_t, each holding a GPU clock value and a system clock value, so thread trace data can be lined up with API traces.New
ROCPROFILER_THREAD_TRACE_PARAMETER_RESOURCE_MODEthread trace parameter androcprofiler_thread_trace_resource_mode_tenum, which control when thread trace resources are allocated:ROCPROFILER_THREAD_TRACE_PARAMETER_RESOURCE_MODE_CODE_OBJECTcreates the thread trace queue for a GPU only when the first code object is loaded on that GPU, so GPUs the application never uses get no queue. This is the default.ROCPROFILER_THREAD_TRACE_PARAMETER_RESOURCE_MODE_HSAallocates resources at HSA initialization, as in previous releases.The
HIP,ALL, andDISPATCHmodes are reserved and returnROCPROFILER_STATUS_ERROR_NOT_IMPLEMENTED.
New
--att-resource-modeoption (default,hsa, orcode-object) sets when thread trace resources are allocated. The default iscode-object.SQTT markers and the
rocprof-trace-decoderPython API are now included in the ROCm installation. Previously, they required building from source.Extended
att-no-detailsupport to gfx1200 and gfx1201, for lower-volume traces that collect occupancy without instruction-level detail.
Memory copy tracing for
hsa_amd_memory_async_batch_copy. Each copy in a batch is reported as its own memory copy record through the existing callback and buffer tracing kinds.New
rocprofiler::sdk::pc_sampling::verify_samplein the header-onlyrocprofiler-sdk/cxx/pc_sampling.hpp. It checks that a stochastic PC sample is consistent with the instruction at its PC.Tech preview of compute and profiling support on WSL2 for RDNA 3.5 (gfx1150, Strix Point) GPUs, including
rocprofv3counter collection and thread trace (--att). Agent topology is read throughlibrocdxg, sorocprofiler_query_available_agentsreturns the same results whether or not HSA has been initialized.CPU agents are now detected on ppc64le (POWER) systems. The
/proc/cpuinfoparser previously handled only x86 fields.HIP API tracing for
hipKernelSetAttributeForDevice,hipLibraryGetModule,hipModuleEnumerateFunctions, andhipDeviceFlushGPUDirectRDMAWrites.HSA API tracing for
hsa_amd_enable_high_precision_timestamps,hsa_amd_agent_set_attribute,hsa_amd_vmem_get_vmem_info, andhsa_amd_svm_discard_and_prefetch_batch_async.Support for the gfx117x, gfx1250-strict, and gfx1250 targets.
Documentation on using
rocprofv3with LLM inference frameworks.
Changed#
Device-wide counter sampling (SPM)
rocPD schema 3.0.5 stores SPM counter location on
rocpd_info_pmc. Each agent and combination of counter, XCC, shader engine, shader array, WGP, and instance has its own pmc id. Location dimensions not provided for a counter are stored asNULLand left blank in CSV output. The instance index is always stored.rocpd_pmc_eventkeeps the sample value and points at that id.Experimental
rocprofiler_spm_dispatch_counting_service_set_agents()scopes dispatch SPM to selected GPU agents; two SPM contexts can be active together when their agent sets are disjoint (#11968).
Thread trace
Thread trace now allocates its memory and signals through KFD and copies data with SDMA. If an SDMA copy isn’t possible, it falls back to HSA and prints a warning. This fixes several crashes and hangs that occurred when thread trace ran after
main()returned.The thread trace decoder now tries the versioned library names (
librocprof-trace-decoder.so.<major>, then.so.<major>.<minor>, then.so.<major>.<minor>.<patch>) before falling back to the unversioned name. This letsrocprofv3 --attwork on installs that ship only the versioned library, such as TheRock runtime wheels.By default, thread trace now creates its queue on each GPU when the first code object is loaded there, not at HSA initialization. To keep the old behavior, use
--att-resource-mode hsaorROCPROFILER_THREAD_TRACE_PARAMETER_RESOURCE_MODE_HSA.
Request ROCr high-precision timestamp correlation during HSA registration when GPU timestamp collection is configured and the runtime provides the tools API. Older runtimes and tools that only trace CPU APIs or markers retain the existing behavior.
Dispatch counter collection, Advanced Thread Trace (ATT), SPM, and PC sampling no longer register with the HSA queue controller’s per-queue callback registry. The write interceptor and async completion path call each service’s dispatch hooks explicitly, with agent-scoped active checks so a service configured on one GPU does not pull other GPUs off the fast path (#11967, #11968, #11969, #11970).
ROCprofiler-SDK takes the OpenMP tools (OMPT) interface only when a registered client subscribes to OMPT tracing. Otherwise it passes the interface to the next OMPT tool, so a user’s own OMPT tool can run alongside ROCprofiler-SDK.
Signal-less KFD dispatch-log mode (
ROCPROFILER_KFD_DISPATCH_LOG_SIGNAL_LESS) now uses version 4 of the KFD dispatch-log stream ABI (profiler ABI version 6).rocprofv3checks each stochastic PC sample against the instruction at its PC. Samples that fail the check are dropped and counted as invalid. Samples from unknown code objects are kept as is.rocprofv3now rejects multi-pass counter collection from an input file when combined with--pidor--collection-period. This already applied to multiple--pmcflags. The error message names what made the run multi-pass.Extra counter YAML files passed with
-E/--extra-countersare now validated before they are loaded. If a definition is malformed,rocprofv3reports the field that is wrong and exits with an error. It previously crashed or hung.Environment variables whose names contain sensitive keywords, such as
password,secret,token, orkey, are no longer written to the rocPD database.When collecting PMC counters on GPUs from gfx1100 up to but not including gfx1250, ROCprofiler-SDK warns if
power_dpm_force_performance_levelis notprofile_standardor can’t be read. Dynamic power and clock scaling can skew counter values. Set the level toprofile_standard(for example, withamd-smi set --perf-level STABLE_STD) for stable results.Queue interposition now issues a memory fence when writing packets to the ring buffer to prevent packets from being read out of order or half-written.
Increased the PC sampling buffer size from 4 MB to 64 MB on gfx94x, gfx95x, and gfx125x GPUs, which allocate a buffer per XCC. Many samples were dropped or invalid at reasonable sampling rates with the smaller buffer.
Renamed the
ROCPROF_ABORT_FLUSH_TIMEOUT_SECONDSenvironment variable toROCPROF_FINALIZE_WAIT_TIMEOUT_SECONDS(default: 10). It sets how long finalization waits for an abort to finish and now also for child processes after a signal. The old name is no longer read, so update any scripts that set it.Tool lifecycle messages are now logged at info level instead of warning. Processes that load a rocprofiler-sdk tool with logging disabled no longer write these messages to stderr.
Resolved issues#
When a context that needs packet interception replaces a queue created by
hsa_amd_queue_createwith a system-memory intercept queue, the device-memory placement flags are now cleared in the caller’s descriptor. HIP reads these flags back and previously assumed a device-memory ring buffer that profiling had silently replaced.Fixed kernel replay dropping later
--pmcgroups when used with--kernel-iteration-range: the range is evaluated once per logical dispatch (pass 0), and later replay passes reuse that decision instead of consuming another iteration each pass (#11335).Fixed a deadlock in queue interposition that occurred when more kernels were linked by barriers than
ROCPROFILER_ASYNC_SIGNAL_HANDLER_THREADSallowed, for example in a HIP graph fork/join or an all-reduce. A single thread now waits for any completion signal. This also works with older ROCr versions.Fixed
rocprofv3hanging at normal exit when the application left a long-running child process. Child processes are now waited for only when exiting on a signal, and that wait has a time limit.Fixed
roctxProfilerPauseandroctxProfilerResumewithrocprofv3process attachment. A pause issued before attaching no longer affects the attached session. With--selected-regions, collection starts at the firstroctxProfilerResumeafter attaching.Fixed two process attachment issues. Reusing a pooled HSA signal leaked the signal’s previous handle. Attach callbacks could also be associated with the wrong tool when
rocprofiler_configureandrocprofiler_configure_attachcame from different shared objects, for example when PyTorch exportsrocprofiler_configure.Fixed output path formatting hanging when a placeholder’s value contains that same placeholder, for example
-o '%argt%'with an application argument of%argt%, or an environment variable that refers to itself. Such placeholders are now left as written and a warning is printed.Fixed
%q{NAME}not expanding in output paths. Names such asresults_%q{MPI_RANK}becameresults_on every rank.Fixed a second copy of rocprofiler-sdk bundled with an application such as PyTorch initializing itself under
rocprofv3, even though all calls go to the globally preloaded library.Fixed the TCP counter instance count on gfx1250 GPUs, where the two SIMD pairs now share one TCP.
Fixed
SQ_WAVESovercounting on gfx1152 when a desktop session is active. On gfx11, SQ counters now count only VMIDs 8 to 15, which excludes graphics work from the desktop.Fixed the gfxip family label for agents whose name includes an ISA variant suffix, such as
gfx1250-strict.Fixed
rocprofv3 --pmcaborting on WSL2, where/dev/kfddoesn’t exist. PC sampling and KFD-only controls are now reported as unavailable instead.Fixed a build failure with newer HSA headers that define
HSA_ACCESS_PERMISSION_EX. Execute permissions are now formatted correctly.Fixed internal singletons being constructed in storage that wasn’t aligned for their type, which is undefined behavior and could cause split-lock accesses on x86.
Fixed
ROCPROFILER_KFD_DISPATCH_LOG_SIGNAL_LESS=truewriting log messages in processes where no tool is registered.Fixed the synchronous device counting sample crashing at exit.
Fixed time-based (microsecond) PC sampling intervals being limited by the maximum meant for cycle-based intervals.
Fixed a possible crash when a traced HIP API call had a string (
const char*) argument that was read after the call returned. String arguments are now copied when the call is traced.Fixed
rocprofv3process attachment failing with framework workloads and Python wheels. Attachment libraries now load on runtime-only installs that lack unversioned library symlinks. Loading the same library through different paths is no longer treated as a conflict. A compatible library next torocprofiler-registeris preferred over an unrelated installation.Fixed
rocprofv3 --attignoringROCPROF_ATT_LIBRARY_PATH. It searched the whole filesystem from/and used the first trace decoder it found.Fixed
rocprofiler-sdk-rocattachfailing to find symbols such asrocprofiler_register_detachin the target process when the library’s memory mappings didn’t line up one-to-one, in address order, with its loadable segments.
Known issues#
Thread trace can produce an incorrect binary (.att) size in gfx115x.
rocRAND (5.2.0)#
Optimized#
Added
__restrict__qualifiers to the device API benchmark, enabling better compiler optimization.Reduced shared memory usage and used the ADDC intrinsic in the Philox generator.
Resolved issues#
Internal kernels are no longer exported from
librocrand.so.Fixed incorrect values generated by the Sobol method when a single generation request crosses the end of the sequence period. Also fixes an out-of-bounds read in host generation.
Fixed an issue with Sobol teardown aborting after
hipDeviceReset.Fixed the host API scrambled Sobol generators using incorrect scramble constants. The output of
ROCRAND_RNG_QUASI_SCRAMBLED_SOBOL32andROCRAND_RNG_QUASI_SCRAMBLED_SOBOL64changes as a result.
rocSHMEM (3.8.0)#
Added#
GDA device code refactor (#7217).
Core GDA device code is now header-only.
Removed indirection and branching when compiled for a single GDA provider.
Reduced the size of device code GDA Queue Pair management objects.
Reduced latency of
rocshmem_quietfor AMD Pensando IONIC provider. This also reduces the latency of blocking communication calls.Communications using the GDA backend can now selectively choose at compilation time to reduce the internal GDA Queue Pair object correctness and safety constraints, improving application performance when such relaxations are known to be safe by the caller. These relaxations can be applied independently at each call site. This is currently an internal mechanism and is not exposed by the public rocSHMEM APIs.
Added standalone signal APIs:
rocshmem_signal_addandrocshmem_ctx_signal_addrocshmem_signal_setandrocshmem_ctx_signal_setrocshmem_signal_wait_until
Deprecated#
Deprecated
rocshmem_signal_fetch_wgandrocshmem_signal_fetch_wave. Userocshmem_signal_fetchinstead.
Resolved issues#
Fixed tile RMA operations (
rocshmem_tile_put,rocshmem_tile_get) to correctly apply non-zerostart_coordvalues to both the source and destination tensors; previously only the remote tensor was offset, causing incorrect data movement whenstart_coord != 0. Fully-packed column-major tiles are now also classified asContiguous, enabling a single flatmemcpyinstead of per-column copies.
rocSOLVER (3.38.0)#
Added#
Hessenberg reduction routines
GEHD2GEHRD
Added support for the
gfx1250-strictarchitecture.
Changed#
Clarified in the API documentation that
rocsolver_rfinfois an input and output parameter because it can be modified within the functions.
Optimized#
Improved performance of expert eigensolvers
SYEVDX/HEEVDXandSYGVDX/HEGVDX.
Resolved issues#
Fixed failure of
GETRSforbatch_count > 64 * 1024.Fixed failure of
GETRFforbatch_count > 64 * 1024for certain sizes such asm = 3andn = 4.Improved kernel launches to detect invalid launch configuration.
Fixed sparse functions potentially not using the stream set in the handle.
rocSPARSE (5.1.0)#
Added#
Added the
rocsparse_spsortandrocsparse_spsort_buffer_sizegeneric routines, which sort a sparse matrix in place or into a separate target matrix. COO matrices are sorted by row or by column, and CSR and CSC matrices have the indices within each row or column sorted. The values are reordered with the indices. Uniform strided batched matrices are supported, including broadcasting a single source matrix to every sample of the target matrix.Added the
rocsparse_spsort_descr_create,rocsparse_spsort_descr_destroy, androcsparse_spsort_set_inputauxiliary routines, together with therocsparse_spsort_descrtype and therocsparse_spsort_alg,rocsparse_spsort_input, androcsparse_spsort_stageenums.Added support for the
gfx1250-strictarchitecture.Added batch support for
rocsparse_sptrsm.Added the
rocsparse_dnmat_transposegeneric routine for dense matrix transposing.Added the
rocsparse_dnmat_copy_datageneric routine for dense matrix data copying.
Changed#
Replaced the soon-to-be-deprecated
__hip_atomic_*builtins with the equivalent__scoped_atomic_*builtins.
Optimized#
Improved
rocsparse_Xgemviperformance, especially whennis greater thanm.Improved the performance of the sparse triangular solves and the incomplete factorizations on gfx942 and gfx950. The kernels no longer write back and invalidate the L2 cache for every solved or factorized row. The row results and their completion flags are now exchanged with device-scope atomic stores and loads instead. This affects
rocsparse_spsvandrocsparse_sptrsv(CSR, CSC, COO, and ELL formats),rocsparse_spsmandrocsparse_sptrsm(CSR, CSC, and COO formats),rocsparse_Xcsrsv_solve,rocsparse_Xcsrsm_solve,rocsparse_Xcsrilu0,rocsparse_Xcsric0,rocsparse_spilu0,rocsparse_spic0, androcsparse_spildlt0. It also affectsrocsparse_Xbsrsv_solvewith block sizes up to 32,rocsparse_Xbsrilu0with block sizes up to 8, androcsparse_Xbsric0with block sizes up to 16 and at most 128 blocks per block row. Larger block sizes androcsparse_Xbsrsm_solvekeep the previous synchronization, because their kernels reread row results too often for device-scope atomics to be worthwhile.Improved the performance of the analysis of the CSR, CSC, COO, and BSR triangular solves and incomplete factorizations on gfx942 and gfx950. This affects the analysis stage of
rocsparse_spsv,rocsparse_sptrsv,rocsparse_spsm, androcsparse_sptrsm, androcsparse_Xcsrsv_analysis,rocsparse_Xcsrsm_analysis,rocsparse_Xbsrsv_analysis,rocsparse_Xbsrsm_analysis,rocsparse_Xcsrilu0_analysis,rocsparse_Xcsric0_analysis,rocsparse_Xbsrilu0_analysis,rocsparse_Xbsric0_analysis, and the analysis stage ofrocsparse_spilu0,rocsparse_spic0, androcsparse_spildlt0. Each row previously updated the maximum row length with its own device-scope atomic on a single address, which serializes on these architectures. On gfx942 and gfx950, each row now updates a per-workgroup maximum with a shared-memory atomic, and the last row of the workgroup to finish issues a single device-scope atomic.Optimized
rocsparse_spgemm(CSR and BSR),rocsparse_csrgemm_nnz,rocsparse_Xcsrgemm,rocsparse_csrgemm_symbolic,rocsparse_Xcsrgemm_numeric,rocsparse_bsrgemm_nnzb, androcsparse_Xbsrgemmfor rows ofCwhose hash table does not fit in shared memory. For BSR matrices with a block dimension greater than 1, onlyrocsparse_bsrgemm_nnzbis affected.
Resolved issues#
Fixed an out-of-bounds write in
rocsparse_coosort_by_rowandrocsparse_coosort_by_columnwhen every row (or column) contains a single non-zero entry. The temporary workspace for the segment computation was one index too small, which could corrupt the segment offsets and produce incorrectly sorted results.Fixed
rocsparse_dense_to_sparse(including the Blocked ELL format),rocsparse_Xnnz,rocsparse_Xdense2csr,rocsparse_Xdense2csc,rocsparse_Xcsr2dense,rocsparse_Xcsc2dense,rocsparse_Xcoo2dense, androcsparse_sparse_to_densefor matrices with a large number of rows, columns, block rows, or non-zeros. The conversion kernels sized the launch grid from these counts without a grid-stride loop, so a large enough matrix exceeded the 32-bit kernel dispatch limit. The launch grids are now clamped to the device limit and the kernels use grid-stride loops. With 32-bit indices,rocsparse_Xcsr2dense,rocsparse_Xcsc2dense, androcsparse_Xcoo2densealso computed the dense element count in 32 bits and skipped zero-filling a dense matrix with2^31or more elements. The count is now 64-bit, so dense matrices with fewer than2^32elements are zero-filled correctly.Fixed an overflow issue in
rocsparse_rotiand the genericrocsparse_rotroutine. When using 64-bit indices andnnz>=2^32, a 32-bit element-index calculation could overflow, leaving some elements unrotated and causing low-index elements to be processed with incorrect data. The kernel now computes element indices in 64-bit arithmetic and uses a grid-stride loop with the launch grid clamped to the 32-bit work-item dispatch limit.Fixed an overflow issue in
rocsparse_Xsctrand the genericrocsparse_scatterroutine. When using 64-bit indices andnnz>=2^32, a 32-bit element-index calculation could overflow and prevent some elements from being scattered. The kernel now computes element indices in 64-bit arithmetic and uses a grid-stride loop with the launch grid clamped to the 32-bit work-item dispatch limit.Fixed an integer overflow in the ELL SpMV kernels used by
rocsparse_spmv(ELL format) and the legacyrocsparse_Xellmvroutines when operating on matrices with more than2^32rows and 64-bit index types. The overflow caused rows beyond the2^32boundary to be mapped to incorrect row indices, leaving some output values uncomputed. Row indices are now always computed using the matrix index type.Fixed an out-of-bounds read of the row map in the block-row incomplete factorization kernels used by
rocsparse_spic0androcsparse_spilu0. Eight of the nine BSR kernels indexed the map with the raw block index and never compared it against the block row count, so a launch grid rounded up past that count read outside the map allocation. All nine now bounds check, which is what the CSR kernels in the same family already did.Fixed
rocsparse_spic0androcsparse_spilu0silently truncating the launch grid for matrices with more block rows than the launch grid can hold. The block row count is taken from the sparse matrix descriptor as a 64-bit value and was narrowed into the 32-bit grid extent without a clamp. The binding grid.x limit is(2^32 - 1) / blockDim.xblocks, becausegridDim.x * blockDim.xmust stay below2^32, rather than the 2147483647 thatmaxGridSize[0]reports. All BSR kernels now clamp the extent and recover the remaining block rows inside the kernel.Fixed an integer overflow in
rocsparse_Xgtsv_interleaved_batch, which produced incorrect results whenbatch_count * morbatch_stride * mexceededINT32_MAX. The interleaved layout makes every arraybatch_stride * melements long, but the solver kernels computed the row offsets into those arrays in 32-bit arithmetic. The offsets are now computed in 64-bit arithmetic.Fixed kernel launches whose
grid.x * blockDim.xexceeded the 32-bit work-item dispatch limit. CDNA rejects those launches (hipErrorInvalidConfiguration/rocsparse_status_internal_error).rocsparse_axpby(both thebetascaling ofyand theaxpyiupdate),rocsparse_axpyi,rocsparse_rotiandrocsparse_rot,rocsparse_scatter, ELL SpMV (non-transposed and transposed), and COO AoS SpMV (atomic and transposed) now clampgrid.xto(2^32 - 1) / blockDim.xand cover the remainder with a grid-stride loop.Fixed an integer overflow in the binary search that maps nonzeros to rows in the nnz-split algorithms of
rocsparse_spmv(CSR) androcsparse_spmm(CSR and CSC). With 32-bit indices and matrices with more than2^30rows, the search midpoint(left + right) / 2could overflow, causing out-of-bounds reads or incorrect row assignments. The midpoint is now computed asleft + (right - left) / 2.Fixed an invalid kernel launch in legacy
rocsparse_Xbsr2csrand the 64-bit BSR-to-CSR conversion, where the 1024-thread kernels overflowed the 32-bit work-item grid limit at about 4.2 million block rows because the grid was not clamped.Fixed an overflow issue in the COO atomic SpMV kernels used by
rocsparse_spmvandrocsparse_v2_spmvfor COO and COO AoS matrices. When using 64-bit indices andnnz>=2^32, a 32-bit element-ID calculation could overflow and prevent some nonzeros from being accumulated. The kernels now compute element IDs in 64-bit arithmetic.
rocWMMA (2.2.2)#
Added#
Added support for the
gfx1250-stricttarget.Added community samples for grouped MoE GEMM (
grouped_gemm_moe), Stream-K GEMM (simple_gemm_streamk), and FlashAttention v2 (simple_flash_attn_v2).Added host tests for test-kernel generation.
Changed#
Reduced test build time through shared support objects, split GEMM instantiations, and lower template overhead.
Consolidated documentation dependencies in
rocm-docs-core.
Resolved issues#
Fixed HIP and OpenMP dependency discovery in the installed CMake package.
Fixed host/device annotations on constexpr functions for compatibility with stricter Clang diagnostics.
Fixed resource leaks in DLRM and hipRTC regression tests.
RPP (3.2.0)#
Added#
Support for building RPP on non-x86 architectures using the SIMDE (SIMD Everywhere) library for portable SIMD intrinsics; x86 builds are unchanged.
rppGetStatusString()helper that maps anRppStatuscode to its enumerator name string.Support for the
gfx1250-strictGPU target in HIP builds.RPP_RETURN_IF_ERRORmacro inrppdefs.hfor propagating nestedRppStatusfailures from the enclosing function.
Changed#
On non-x86 architectures,
simde(found viapkg-config) is now a required RPP build dependency and is propagated to consumers of therppCMake package.
Resolved issues#
Corrected
rppt_logto returnRPP_ERROR_INVALID_SRC_DATATYPEinstead of the misleadingRPP_ERROR_INVALID_DST_DATATYPEfor unsupported source datatypes.Fixed error propagation through the HIP kernel dispatcher: kernel launch failures now return
RPP_ERROR_HIP_LAUNCHinstead ofRPP_SUCCESS, which previously masked launch errors as pixel mismatches on GPUs with no matching kernel binary.
ROCm known issues#
ROCm known issues are noted on GitHub. These issues will be fixed in a future ROCm release. For known issues related to individual components, review the ROCm component changelogs.
AI workloads might fail with an illegal memory access error on AMD Ryzen APUs under WSL2#
AI workloads running under WSL2 with ROCm 10.2.0 on AMD Ryzen APUs (gfx1150, gfx1151, gfx1152, gfx1103, and gfx1153) might fail with an illegal memory access error. The failure terminates the workload and has been observed with PyTorch and llama.cpp.
Observed error messages include:
ROCm error: an illegal memory access was encountered
torch.AcceleratorError: CUDA error:
an illegal memory access was encountered
HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION:
The agent attempted to access memory beyond the largest legal address.
GPU fence timeout after 30000ms — possible device fault
As a workaround, use ROCm 10.1.0 or the latest ROCm 10.3.0 nightly build. See GitHub issue #10097.
ComfyUI workloads might show lower performance on some AMD Radeon GPUs and Ryzen APUs#
ComfyUI-based AI workloads using the AOTriton backend might show lower performance on some AMD Radeon GPUs and Ryzen APUs, such as AMD Radeon RX 7900 series GPUs and Ryzen AI Max 300 Series (gfx1151) APUs. As a workaround, ComfyUI users can use an alternative attention backend, for example --use-flash-attention. For more details on choosing the best attention backends for ComfyUI image and video generation workloads, see Understanding Attention Algorithms and Their Backends for Image and Video Generation. See GitHub issue #10098.
Hugging Face model training throughput might regress on AMD Instinct MI350 Series GPUs#
Hugging Face model training workloads might see lower training throughput on AMD Instinct MI355X and MI350X (gfx950) GPUs, compared to builds that use AOTriton 0.11.2b. Impacted models include BERT, DistilBERT, RoBERTa-large, and GPT-2. As a workaround, rebuild PyTorch and pin AOTriton to version 0.11.2b.
Note
This issue is partially addressed in ROCm 10.2.0 for some Hugging Face models. See the ROCm 10.2.0 resolved issues entry for details.
See GitHub issue #10099.
rocHPL throughput might regress on AMD Instinct MI325X GPUs#
rocHPL workloads running on AMD Instinct MI325X GPUs (gfx942) in an 8-GPU configuration might see lower throughput compared to previous ROCm releases, dropping from roughly 570,000 GFLOPS to 463,000 GFLOPS. As a workaround, set ROCBLAS_USE_HIPBLASLT=0 to use the Tensile backend instead of hipBLASLt for rocHPL:
ROCBLAS_USE_HIPBLASLT=0 <rocHPL command>
See GitHub issue #10100.
GPU Mode2 reset might time out on AMD Instinct MI300A GPUs#
When a workload triggers a hardware hang or RAS error on AMD Instinct MI300A GPUs (gfx942), the amdgpu Mode2 reset used to recover the GPU might time out with failed to send mode2 reset, error code -62. After the timeout, the SMU stops responding, and subsequent GPU management calls fail with amdsmi_init() timed out after 60s. The GPU driver may be unresponsive. See GitHub issue #10101.
AMD Instinct MI300A systems might reboot spontaneously on RHEL 10.3#
AMD Instinct MI300A systems might reboot spontaneously on Linux kernels that enable the X86_FEATURE_INVLPGB broadcast TLB flush, including RHEL 10.3. This happens because the TLBSYNC instruction used by the broadcast TLB flush sometimes hangs on MI300A, triggering a CPU machine check watchdog that sync-floods the data fabric. As a workaround, add clearcpuid=419 to the kernel command line. See GitHub issue #10102.
CRIU restore fails on AMD Instinct MI350X GPUs#
CRIU restore fails for GPU workloads on AMD Instinct MI350X GPUs. When criu restore is called for any workload that creates a GPU queue, the AMDKFD_IOC_CRIU_OP restore ioctl returns Invalid argument and restore exits with:
amdgpu_plugin: Restore ioctl failed: Invalid argument
amdgpu_plugin: Failed to restore (ret:-1)
Restoring FAILED.
Checkpoint operations (criu dump) complete successfully; only restore is affected. No workaround is available. See GitHub issue #10103.
An in-place amdgpu-dkms upgrade might fail to load the new driver on Linux#
When upgrading amdgpu-dkms in place without rebooting, modprobe -r amdgpu does not unload the full module stack, and the old amdttm and amdkcl modules stay resident in memory. Loading the new driver then fails with modprobe: ERROR: could not insert 'amdgpu': Invalid argument because amddrm_ttm_helper rejects stale symbol versions exported by the leftover modules.
As a workaround, unload all remaining amdgpu-related modules before reinstalling the driver:
sudo modprobe -r amdgpu amdttm
lsmod | grep -E '^(amdgpu|amdxcp|amddrm|amdttm|amd_sched|amdkcl)'
If any modules remain listed, unload each with sudo modprobe -r <module_name>, remove the old driver packages, install the new packages, and load the driver with sudo modprobe amdgpu. See GitHub issue #10104.
AGFHC rocTwitch and Sprites subtests might fail on AMD Instinct GPUs#
The AGFHC Sprites subtest might fail with TEST_GFX_SPRITES_FAIL, reporting NO_FREE_RESOURCE (0x5) from the embers allocator and terminating firexs2 with return code 255. This failure occurs only when the rocTwitch and Sprites subtests run together; running either subtest independently completes successfully. See GitHub issue #10105.
RCCL communicator initialization fails when each rank is bound to a single GPU#
In RCCL jobs where each rank sees only one GPU — for example, with Slurm’s --gpu-bind=closest, or per-rank HIP_VISIBLE_DEVICES or ROCR_VISIBLE_DEVICES settings — communicator initialization fails. ncclCommInitRank returns ncclInternalError, which applications and rccl-tests report as “internal error - please report this issue to the NCCL developers”. Jobs where each rank sees two or more GPUs, or all GPUs on the node, are not affected.
As a workaround, let every rank see all GPUs on the node. The application is then responsible for selecting the GPU closest to its CPU binding; applications that do not will lose CPU–GPU locality. Alternatively, rebuild RCCL from a source revision that contains the fix and preload it ahead of the packaged library, which preserves the existing binding. See GitHub issue #10106.
VCN decode might hang on AMD Instinct MI325X GPUs#
Video decode workloads using rocDecode on AMD Instinct MI325X GPUs might hang intermittently when decode bursts are separated by idle gaps of approximately one second or longer. This happens because each idle gap triggers a VCN power-gate cycle; on power-up, the VCN firmware might read the ring-buffer size register before the hardware restores it, reading 0 and preventing submitted commands from completing. Only the affected VCN instance stalls; the remaining VCN instances on the GPU continue operating. In distributed training jobs that decode video in the input pipeline, the stalled rank stops progressing and the job fails with a collective-communication timeout. This issue will be resolved in a VCN firmware update for an upcoming Best Known Configuration (BKC) release. See GitHub issue #10107.
rocdecode VP9 video decoding might cause a VCN ring timeout and GPU reset on AMD Instinct MI350 Series GPUs#
Decoding VP9 video clips using rocdecode on AMD Instinct MI350 Series (gfx950) GPUs might cause the VCN video-decode engine to hang with a ring timeout. The ring reset fails, triggering a full GPU mode 1 reset. This occurs because of a regression in the VCN firmware. This issue will be resolved in a VCN firmware update for an upcoming Best Known Configuration (BKC) release. See GitHub issue #10108.
Packed FP32 instructions might produce unexpected data on AMD Instinct MI350X GPUs#
On AMD Instinct MI350X GPU (gfx950), kernels that use v_pk_add_f32, v_pk_mul_f32, or v_pk_fma_f32 instructions with certain op_sel encodings can produce unexpected results. As a workaround, the LLVM compiler corrects these encodings by commuting operands for packed ADD/MUL and splitting packed FMA into two scalar v_fma_f32 instructions. The assembler also rejects invalid op_sel values in hand-written assembly. Kernels compiled with an earlier ROCm release that contain the affected encodings must be recompiled to benefit from this workaround. See GitHub issue #10109.
ROCm resolved issues#
The following notable issues have been fixed in ROCm 10.2.0.
Valgrind workloads might stop responding on WSL2 with high system RAM#
Previously, running workloads under Valgrind on WSL2 might stop responding during ROCm runtime initialization on AMD Radeon GPUs, such as the AMD Radeon RX 9060, on systems with a large amount of system RAM. This happened because WSL reserved GPU address space upfront in a way that could collide with Valgrind’s reserved memory range as system RAM grows. Workloads that don’t use Valgrind are unaffected. See GitHub issue #8793.
Concurrent rocprofv3 profiling causes node reset on AMD Instinct MI300A GPUs#
Previously, running rocprofv3 --pmc or --kernel-trace concurrently across multiple GPUs of a single AMD Instinct MI300A node might hard-reset the node, terminating all co-resident jobs and requiring a manual power cycle to recover. See GitHub issue #8229.
Hugging Face model training throughput for specific models is restored on AMD Instinct MI350X#
ROCm 10.2.0 restores Hugging Face model training throughput to previous levels for BART, GPT-2, and Llama 2 70B Chat workloads on AMD Instinct MI350X (gfx950) GPUs, recovering the throughput loss reported in the ROCm 10.1.0 known issues. See GitHub issue #8878.
Note
A residual performance gap remains on some models. See the ROCm 10.2.0 known issues for current status and workarounds.
ROCm upcoming changes#
LLVM symbolic links#
The compiler enables -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=On, which moves LLVM runtime libraries one level deeper, from lib/llvm/lib to lib/llvm/lib/x86_64-unknown-linux-gnu, and adds an x86_64-unknown-linux-gnu directory to the Clang resource directory containing compiler_rt and sanitizer libraries.
To maintain backwards compatibility with older 10.x releases, the following symbolic links are installed:
Symbolic links in
lib/llvm/libfor libraries that moved intolib/llvm/lib/x86_64-unknown-linux-gnu.x86_64-pc-linux-gnu -> x86_64-unknown-linux-gnuin the Clang resource directory, so that projects hardcoding-target x86_64-pc-linux-gnucontinue to function.libclang_rt*inlib/llvm/lib/clang/24/lib/linux, pointing into thex86_64-unknown-linux-gnudirectory one level up.
These symbolic links will be removed in the next major version of ROCm.