Using Origami and Stream-K with hipBLASLt#
hipBLASLt supports the Origami with Stream-K library, which reduces library sizes for a wide range of General Matrix-Matrix Multiplication (GEMM) shapes and sizes. It also provides more consistent performance, which might be better in some cases. Stream-K partitions an equal share of the aggregate inner-loop iterations among physical processing elements, which provides a near-perfect utilization of computing resources. For more information about Stream-K, see Stream-K: Work-centric Parallel Decomposition for Dense Matrix-Matrix Multiplication on the GPU on the arXiv website.
Configuring the kernel selection strategy#
The TENSILE_SOLUTION_SELECTION_METHOD environment variable controls the hipBLASLt kernel selection strategy for GEMM operations.
These variables apply to all GEMMs in an application.
Set this variable to 2 to enable the Origami with Stream-K library or leave it set to 0 to continue to use the default settings.
Note
On the AMD Instinct™ MI350 series, Origami with Stream-K is the only kernel selection strategy available.
There is no alternative library and the TENSILE_SOLUTION_SELECTION_METHOD variable has no effect.
TENSILE_SOLUTION_SELECTION_METHOD=0(Standard tuned libraries)Kernels are selected from the standard tuned libraries.
The heuristic best kernel is selected from the standard tuning grid.
User-driven tuning (tunable ops) only accesses kernels from the standard grid and free-size libraries.
This option does NOT use any Stream-K kernels.
TENSILE_SOLUTION_SELECTION_METHOD=2(Stream-K)This enables the optional Origami with Stream-K solution selection to use a GEMM scheduling algorithm that results in consistently good peak GEMM performance with far fewer tuned kernels.
The heuristic best kernel is selected from the Origami with Stream-K library.
User-driven tuning (tunable ops) considers all kernels from the standard grid, the free-size library, and the Origami with Stream-K library.
Note
Stream-K supports a different range of data types on different AMD GPUs. For example, the MI300A APU supports
a wider variety of data types, including FP32, FP16, BF16, FP8, and BF8.
TENSILE_SOLUTION_SELECTION_METHOD=2 is used to enable Stream-K on all MI300 platforms.
Configuring the kernel launch behavior#
You can control persistent kernel launch behavior using the environment variables listed in the following table. These variables apply to persistent kernels, including Stream-K and persistent DataParallel, throughout an application. By default, Stream-K uses a model to predict the optimal grid size to use when launching a GEMM kernel at runtime. You can adjust the number of workgroups using the settings below. The listed legacy aliases remain supported. If both names are set, the preferred name takes precedence, even if its value is invalid; there is no fallback to the legacy value.
Environment variable |
Legacy alias |
Description |
|---|---|---|
|
|
Default: |
|
|
Request a fixed number of workgroups, subject to kernel launch limits. Default: |
|
|
Set the CU budget used for grid sizing. Default: |
|
|
Multiply the CU count by this factor when dynamic grid selection and the CU cap are disabled and no fixed grid is set. Kernel launch limits still apply. Default: |
|
|
Retained for compatibility; currently has no effect on workgroup mapping. Default: |
|
|
Debug override for a selected Hybrid kernel: |
Hybrid assignment policy#
For an already selected Hybrid kernel, hipblaslt-bench --hybrid_assignment_policy accepts the
case-sensitive values Default, DynamicWorkQueue, and Auto. Default preserves existing
behavior, including heuristic mode selection when --sm_count_target is positive.
DynamicWorkQueue forces dynamic assignment; Auto always asks the Origami heuristic to choose the mode.
These settings control how the selected kernel assigns work; they do not select the kernel family.
The legacy option --streamk_tile_scheduling accepts off|0, on|1, and auto|2
(case-insensitive), respectively. Matching old and new values are accepted; conflicting values are rejected.
Omitting both options leaves the existing HIPBLASLT_MATMUL_DESC_STREAMK_TILE_SCHEDULING_EXT
attribute unset, preserving the library default.
TensileLite benchmark YAML uses the same policy names in GlobalParameters.HybridAssignmentPolicy.
The legacy StreamKHybridMode values 0, 1, and 2 remain accepted; if both keys are present,
their lists must describe the same policies in the same order.
The Hybrid debug environment override in the table takes precedence over the normal policy.
Recommendations for using Stream-K#
Stream-K is especially advantageous in certain situations. Follow these guidelines when choosing a kernel selection strategy, based on your application and the desired performance.
Wide range of GEMM sizes: Stream-K is a better choice for applications that handle a variety of GEMM shapes and sizes.
Non-uniform dimensions: Stream-K is particularly beneficial for GEMMs where one dimension is significantly larger than the others.
Consistent performance: Stream-K provides more consistent peak performance than the default selection method by evenly distributing work across the available compute units.
Managing Stream-K resource use#
Follow these guidelines to optimize how Stream-K uses resources:
Promoting concurrency: Use
TENSILE_PERSISTENT_FIXED_GRIDto limit the number of workgroups and leave resources available for other kernels.The following example limits the GEMM kernels to 64 workgroups:
export TENSILE_PERSISTENT_FIXED_GRID=64
Setting a compute-unit budget: Use
TENSILE_PERSISTENT_MAX_CUSto limit the CU budget used for grid sizing.This example requests a grid sized for 32 compute units:
export TENSILE_PERSISTENT_MAX_CUS=32
Adapting to the available compute units#
When other work occupies part of the device, hipBLASLt can measure how many compute units (CUs)
are available to its GEMMs and use that count as an implicit sm_count_target.
Every few launches, a GEMM kernel records which CUs its workgroups ran on. hipBLASLt reads the
result asynchronously and updates a per-stream CU estimate, the adaptive hint.
The hint affects solution selection and the Stream-K Hybrid grid size and assignment mode.
When the hint differs from the one the heuristic query ranked under, the top heuristic result
is re-selected at launch.
Requirements#
Adaptive mode applies to a hipblasLtMatmul call when all of the following are true:
The call uses the C API (
hipblasLtMatmul). The hipBLASLtExt API, grouped GEMM, rocRoller, and launches during stream capture are not adapted.HIPBLASLT_MATMUL_DESC_STREAMK_TILE_SCHEDULING_EXTis set toHIPBLASLT_STREAMK_TILE_SCHEDULING_AUTOorHIPBLASLT_STREAMK_TILE_SCHEDULING_ON. UnderONthe dynamic grid ignores the hint, but solution selection still uses it.Neither
HIPBLASLT_MATMUL_DESC_SM_COUNT_TARGETnorhipblasLtSetSmCountTargetsets a positive value.HIPBLASLT_ADAPTIVE_SM_COUNT=1is set.
export HIPBLASLT_ADAPTIVE_SM_COUNT=1
These conditions do not depend on the device or the device library. Whenever they hold, the top heuristic result reports a larger workspace, launches are tracked for probing, and the timing clients refuse to run (see the behavior notes and timing sections below).
The hint itself only moves when the launched kernels carry the occupancy probe, which requires both:
A device library built with the occupancy probe, using
inv build --emit-occupancy-probeor the CMake option-DTENSILELITE_EMIT_OCCUPANCY_PROBE=ON.A gfx950 device. The probe is only emitted for gfx950 Stream-K Hybrid kernels.
Otherwise the hint stays 0 (no hint), unless HIPBLASLT_ADAPTIVE_SM_COUNT_FORCE sets it.
The environment variables that control adaptive mode are listed in the following table.
They are read once per process. Boolean variables accept 1, on, or true and 0, off, or false
(case-insensitive). An empty variable keeps its default. An invalid or out-of-range value keeps the default
and logs a warning.
Environment variable |
Description |
|---|---|
|
Enable adaptive mode. Default: |
|
Launches per stream between probes. Range: |
|
GEMMs with fewer than this many MFLOPs ( |
|
Number of latest probe readings that decide the hint. The hint changes only when all of them are below it or all are above it, and then takes the smallest reading. No hint counts as the device’s CU count. A single reading of the device’s CU count clears the hint at once, and a new hint then needs a full set of readings below that count. Range: |
|
|
|
Use this fixed CU count as the hint for every query and launch instead of measuring it. A positive value enables adaptive mode on its own. Range: |
|
With |
Behavior notes#
Workspace: In adaptive mode, the top heuristic result reports
workspaceSize = min(MAX_WORKSPACE_BYTES, max(required, 128 MiB))so a re-selection or a regridded launch has room to run. SetHIPBLASLT_MATMUL_PREF_MAX_WORKSPACE_BYTESto your workspace budget. A re-selected solution must fit the workspace passed tohipblasLtMatmul.Re-selection: Only the top heuristic result is re-selected. An algorithm obtained by solution index or from a lower-ranked heuristic result is never replaced, and neither is the algorithm of a launch on a capturing stream. With uniform summation order, a re-selected solution must also keep that order at the launch hint. If neither the heuristic’s pick nor the algorithm fits the workspace at the launch hint, the algorithm launches at the hint it was queried under.
Precedence: An explicit
sm_count_targeton the descriptor or handle always wins over the adaptive hint. AHIPBLASLT_TUNING_OVERRIDE_FILEentry or a tuning-cache result (HIPBLASLT_TUNING_MODE=cache|tune) also wins: its solution is never re-selected.Probing cost: GEMMs below
HIPBLASLT_ADAPTIVE_SM_COUNT_TINY_MFLOPSnever probe. Each probe records an event on the stream. While one thread launches a probe, other threads’ launches on the same stream run without updating the estimate.Missed probes: A probe comes due every
HIPBLASLT_ADAPTIVE_SM_COUNT_PROBE_EVERYlaunches. When the next one comes due and the previous one was still not taken, that counts as a missed probe, for example when only tiny GEMMs ran or the selected solutions do not carry the probe, such as a solution other than a Stream-K Hybrid kernel chosen at the hint. Two missed probes in a row clear the hint until new readings arrive.Graph capture: Launches on a capturing stream are not adapted: they are not probed and run without a hint, including
HIPBLASLT_ADAPTIVE_SM_COUNT_FORCE.Detection scope: The probe detects co-running work that takes whole CUs, for example a CU-masked stream. It does not detect work that shares CUs with the GEMM.
Timing with adaptive mode#
Measured performance depends on the hint, which can change during a run. For this reason
hipblaslt-bench, hipblaslt-sequence, and hipblaslt-api-overhead exit with an error when
HIPBLASLT_ADAPTIVE_SM_COUNT is enabled without HIPBLASLT_ADAPTIVE_SM_COUNT_FORCE.
To time the solution adaptive mode picks at a given hint, set HIPBLASLT_ADAPTIVE_SM_COUNT_FORCE
to that CU count. To include the probe cost, also set HIPBLASLT_ADAPTIVE_SM_COUNT_FORCE_PROBE=1.
HIPBLASLT_ADAPTIVE_SM_COUNT_FORCE=192 HIPBLASLT_ADAPTIVE_SM_COUNT_FORCE_PROBE=1 \
hipblaslt-bench -m 4096 -n 4096 -k 4096 --precision f16_r --hybrid_assignment_policy Auto