Using hipBLASLt offline tuning#
hipblaslt-bench can help find the best-performing GEMM kernel for a given set of GEMM problems
and provide the best solution index for a given problem size.
This index can be used directly in future GEMM calls through the User Offline Tuning mechanism.
However, these indices cannot be reused across library releases or across different device architectures.
Use the command line interface to access this functionality. See hipBLASLt clients for more details.
Using hipblaslt-bench to run the tuning with the best GEMM kernel#
To find and use the best GEMM kernel for a problem, follow these steps:
Generate the tuning command line by setting the environment variable
HIPBLASLT_LOG_MASK=32before calling any hipBLASLt APIs. For more details on how to usehipblaslt-bench, see Logging and heuristics. In the Bash shell, set the following environment variable:export HIPBLASLT_LOG_MASK=32
The following command uses sample_hipblaslt_gemm.cpp as an example:
./sample_hipblaslt_gemm
The tuning command displays the following log entry:
hipblaslt-bench --api_method c -m 1024 -n 512 -k 1024 --lda 1024 --ldb 1024 --ldc 1024 --ldd 1024 --stride_a 0 --stride_b 0 --stride_c 0 --stride_d 0 --alpha 1.000000 --beta 1.000000 --transA N --transB N --batch_count 1 --a_type f16_r --b_type f16_r --c_type f16_r --d_type f16_r --scale_type f32_r --bias_type f32_r --compute_type f32_r --algo_method index --solution_index 56073
Set the environment variable
HIPBLASLT_TUNING_FILE=<file_name>to tune and store the tuning result, which indicates the best solution indices for the GEMM problems. The<file_name>points to the tuning file.In the Bash shell, set the following environment variable:
export HIPBLASLT_TUNING_FILE=tuning.txt
Additionally, you can set the environment variable to specify that the solution found in the tuning stage is under the constraint of the max workspace size setting:
export HIPBLASLT_TUNING_USER_MAX_WORKSPACE=<value> (Default value is: 128 * 1024 * 1024)
The default settings for the following parameters in
hipblaslt-benchare changed in the tuning environment.--iters |-i <value> (Default value is: 1000) --cold_iters |-j <value> (Default value is: 1000) --requested_solution <value> (Default value is: -1) --rotating <value> (Default value is: 512)
After the tuning completes, the expected output is displayed as follows:
./hipblaslt-bench --api_method c -m 1024 -n 512 -k 1024 --lda 1024 --ldb 1024 --ldc 1024 --ldd 1024 --stride_a 0 --stride_b 0 --stride_c 0 --stride_d 0 --alpha 1.000000 --beta 1.000000 --transA N --transB N --batch_count 1 --a_type f16_r --b_type f16_r --c_type f16_r --d_type f16_r --scale_type f32_r --bias_type f32_r --compute_type f32_r --algo_method index --solution_index 56073 Winner: transA,transB,grouped_gemm,batch_count,m,n,k,alpha,lda,stride_a,beta,ldb,stride_b,ldc,stride_c,ldd,stride_d,a_type,b_type,c_type,d_type,compute_type,scaleA,scaleB,scaleC,scaleD,amaxD,activation_type,bias_vector,bias_type,rotating_buffer,hipblaslt-Gflops,hipblaslt-GB/s,us,solution_index,kernel_name N,N,0,1,1024,512,1024,1,1024,1048576,1,1024,524288,1024,524288,1024,524288,f16_r,f16_r,f16_r,f16_r,f32_r,0,0,0,0,0,none,0,f32_r,512,66613.8,363.509,16.1189,56537,<kernel_name>
Set the environment variable
HIPBLASLT_TUNING_OVERRIDE_FILE=<file_name>to load the tuning file and override the default kernel selection with the optimal kernel choices, where<file_name>points to the tuning file.In the Bash shell, set the following environment variable:
export HIPBLASLT_TUNING_OVERRIDE_FILE=tuning.txt
For example, you can use
hipblaslt-benchwithalgo_methodset toheuristicto obtain the solutions for a problem, which include the best tuning solution index../hipblaslt-bench --api_method c -m 1024 -n 512 -k 1024 --lda 1024 --ldb 1024 --ldc 1024 --ldd 1024 --stride_a 0 --stride_b 0 --stride_c 0 --stride_d 0 --alpha 1.000000 --beta 1.000000 --transA N --transB N --batch_count 1 --a_type f16_r --b_type f16_r --c_type f16_r --d_type f16_r --scale_type f32_r --bias_type f32_r --compute_type f32_r --algo_method heuristic --requested_solution 1 --print_kernel_info transA,transB,grouped_gemm,batch_count,m,n,k,alpha,lda,stride_a,beta,ldb,stride_b,ldc,stride_c,ldd,stride_d,a_type,b_type,c_type,d_type,compute_type,scaleA,scaleB,scaleC,scaleD,amaxD,activation_type,bias_vector,bias_type,rotating_buffer,hipblaslt-Gflops,hipblaslt-GB/s,us,solution_index,kernel_name [0]: N,N,0,1,1024,512,1024,1,1024,1048576,1,1024,524288,1024,524288,1024,524288,f16_r,f16_r,f16_r,f16_r,f32_r,0,0,0,0,0,none,0,f32_r,512,37575.2,205.047,28.5758,56537,<kernel_name>
How tuned entries are validated#
A tuning file records a kernel_name next to each solution_index. When the file is replayed,
the index is treated as a lookup hint: hipBLASLt resolves it in the current library and uses it only
if it still names the same kernel. If it does not, that entry is rejected and the problem falls back
to normal heuristic selection.
Validation is per entry, so upgrading hipBLASLt thins a tuning file rather than discarding it, and
only the shapes that failed need re-tuning. A row that carries a solution_name instead is
validated the same way. Files written by an older hipBLASLt carry no name at all, and those are used
only when the file’s Git Version line matches the running build. A build made outside a git
checkout has no version to match, so it uses none of them.
kernel_name names the compiled kernel, not the whole solution. The index still selects the
solution, including its GSU, WGM and StaggerU defaults, and several solutions can share one compiled
kernel, which is why the name is paired with the index rather than used on its own. What validation
guarantees is that the index resolves to the same compiled kernel as when the row was written. It
does not detect a rebuild that changes only that solution’s defaults, whose row is used with the new
defaults, and it cannot find a kernel that moved to a different index: that row is rejected and the
shape falls back to heuristic selection.
Runtime tuning#
Instead of tuning offline with hipblaslt-bench, hipBLASLt can benchmark candidate kernels itself
the first time an application runs a GEMM shape and remember the winner. Runtime tuning is off until
you ask for it, with two environment variables:
export HIPBLASLT_TUNING_MODE=<off|cache|tune>
export HIPBLASLT_TUNING_CACHE_PATH=<file_name>
tune benchmarks supported candidates for each shape it has not seen before, at that shape’s first
matmul, and appends the winner to the cache file. cache only replays what the file already
contains, validating each entry as described above. off is the default and changes nothing.
Either mode can also read a file written by hipblaslt-bench.
Which kernel a matmul launches depends on whether it passes an algorithm. A call to
hipblasLtMatmul with an explicit algo always launches that algorithm: the cache serves such a
caller through hipblasLtMatmulAlgoGetHeuristic, which returns the cached winner as its first
result. A call with algo=nullptr leaves the choice to the library and launches the cached entry
when there is a usable one. In tune mode the call that benchmarks a shape records the winner
either way, but only a call with algo=nullptr launches it straight away. A caller that queried the
heuristic before the shape was tuned, and reuses that algorithm, keeps launching it until it queries
the heuristic again.
Rows written by tune mode carry a schema version and the complete problem key, and every column of
such a row is required: a row with a missing or malformed value, including one a crash cut short, is
ignored. Strides a problem does not use, such as the batch strides of a single batch or the E strides
of an epilogue without an E tensor, are recorded as zero, and so is a scalar or vector scaling format
when neither A nor B is scaled, so a problem matches its own entry whether it arrives through
the C API or the C++ extension. Rows without a schema version, written by hipblaslt-bench or by
hand, keep the historical matching: they distinguish transpose, shape and the principal datatypes,
but not leading dimensions, batch strides, epilogue details or device identity. Do not put problems
that differ only in one of those omitted fields in such a file.
HIPBLASLT_TUNING_CACHE_PATH and HIPBLASLT_TUNING_OVERRIDE_FILE are mutually exclusive. With a
tuning mode set, only the cache is consulted and the override file is ignored.
Tuning is serialised within a process. A thread that meets an untuned shape while another thread is
tuning waits for it, so every shape tune mode meets is tuned, whatever the thread timing. Work
that other threads or processes run on the same GPU during a search still shares the device with it
and can shift the timings, so tune in a run whose GPU is otherwise quiet.
A process running in a secure execution context (set-user-ID, set-group-ID, or another
credential-changing exec such as file capabilities) ignores every HIPBLASLT_TUNING_* variable,
so tuning stays off in it.
Only one process at a time may write a given cache file. Nothing coordinates separate processes, so
the ranks of a multi-process job such as MPI would each benchmark the same shapes and append to the
same file unsynchronised. Tune once in a single-rank warm-up run, then start the job with
HIPBLASLT_TUNING_MODE=cache, which only reads:
export HIPBLASLT_TUNING_CACHE_PATH=tuning.txt
HIPBLASLT_TUNING_MODE=tune ./your_application # one rank, once
mpirun -n 8 env HIPBLASLT_TUNING_MODE=cache ./your_application
A typical run writes the cache once and reuses it afterwards:
export HIPBLASLT_TUNING_CACHE_PATH=tuning.txt
HIPBLASLT_TUNING_MODE=tune ./your_application # benchmark and record
HIPBLASLT_TUNING_MODE=cache ./your_application # replay only
The fastest measured candidate wins outright, with no minimum improvement over the default choice,
matching how hipblaslt-bench selects. The benchmarking effort can be adjusted:
export HIPBLASLT_TUNING_ALL_KERNELS=<0|1> (Default value is: 1)
export HIPBLASLT_TUNING_MAX_CANDIDATES=<value> (Default value is: 128)
export HIPBLASLT_TUNING_COLD_ITERS=<value> (Default value is: 1000)
export HIPBLASLT_TUNING_HOT_ITERS=<value> (Default value is: 1000)
export HIPBLASLT_TUNING_ROTATING_MB=<value> (Default value is: 512)
export HIPBLASLT_TUNING_FLUSH_ICACHE=<0|1> (Default value is: 1)
export HIPBLASLT_TUNING_BUDGET_MS_PER_SHAPE=<value> (Default value is: 300000, five minutes)
Each candidate is measured, and the winner picked, by the same code hipblaslt-bench uses:
HIPBLASLT_TUNING_COLD_ITERS untimed launches, then HIPBLASLT_TUNING_HOT_ITERS launches timed as
a single span and reported as the per-launch mean, the fastest candidate winning. Every candidate gets the full measurement, so there is no cheap ranking round
that could discard the real winner before it has been timed properly.
Candidates are measured against rotating copies of the tensors, so that successive launches do not
read and write the same cache lines. Without this every candidate is timed with its inputs already
resident, which is not how a production GEMM runs and which measurably changes which kernel appears
fastest. HIPBLASLT_TUNING_ROTATING_MB is the memory the rotation may use, and 0 disables it.
Rotation is declined for problems whose buffer sizes cannot be established from the problem
description alone, such as broadcast inputs with a zero batch stride and swizzled A or B.
Those shapes are still tuned, just with their inputs cache-resident. As in hipblaslt-bench, a
problem never rotates over more blocks than a candidate has launches to cycle through them, so very
small problems use fewer blocks than the memory budget would allow.
The instruction cache is invalidated between timed launches, the same way and with the same kernel
the bench client uses, so a candidate is not timed with its own code already resident. It costs
roughly 5% of tuning time and measurably improves agreement with hipblaslt-bench.
HIPBLASLT_TUNING_FLUSH_ICACHE=0 turns it off. The per-flush cost is measured once per device and
subtracted, so the recorded time stays the GEMM time.
Where a shape’s leading kernels are separated by more than measurement noise, this is enough to make
tuning land on the same kernel run after run. Where they are within a couple of percent of each
other, the choice stays unstable and tuning may pick a kernel marginally slower than the heuristic
default; that is a property of the measurement, not of the cache, and hipblaslt-bench behaves the
same way on those shapes.
By default every kernel that can run the problem is measured, not just the ones the selection
heuristic ranks highest, because the fastest kernel is not always inside that ranked prefix. Setting
HIPBLASLT_TUNING_ALL_KERNELS to 0 restricts the search to the first HIPBLASLT_TUNING_MAX_CANDIDATES
kernels hipblasLtMatmulAlgoGetHeuristic returns, in its order, which is faster but can miss the
best one.
Tuning one shape this thoroughly takes time. An exhaustive search over several hundred candidates
takes minutes, and it is paid once per shape in the first process that runs with tune.
HIPBLASLT_TUNING_BUDGET_MS_PER_SHAPE puts a wall-clock ceiling on one shape, covering enumeration
and setup as well as the timed measurements. It defaults to five minutes, so that a first matmul
cannot block a live application indefinitely, and 0 removes the ceiling entirely.
A search the ceiling cuts short still records its best candidate, marked incomplete. Candidates are not measured in order of expected performance, so the best of a truncated prefix is usually not the shape’s best kernel. It is never worse than not tuning at all, though: the kernel the call would otherwise launch is put at the front of the candidate list and measured first, and a truncated search is recorded only when that kernel was measured, so the recorded winner is at least as fast as what the call would have run untuned. Recording it means a shape too large to finish gets some of the benefit immediately rather than none.
An incomplete entry is replayed like any other. What the marking buys is that tune mode revisits the
shape when a run comes along that can finish the search, and the finished row then takes precedence,
so a partial answer never becomes permanent. When a shape has both, replay prefers a finished search
to an incomplete one and a newer row to an older one of the same kind, in the process that tuned it
and in every later one alike. Widening a finished search under a ceiling that stops it therefore
leaves the finished search in use until a run completes the wider one. To finish such a shape, raise or clear
HIPBLASLT_TUNING_BUDGET_MS_PER_SHAPE and run it in tune mode again; as a rough guide a
2048x1024x2048 FP16 shape takes about 146 seconds on MI300X, and the cost grows with the problem.
Each row records the ceiling it was written under and the search that produced it: whether every
kernel or a ranked prefix was searched and how long that prefix was, the workspace limit, the
iteration counts, and whether the flush and rotation were on. A finished row is final only for a run
that would search no more than it did, so widening the search, for example from a ranked prefix to
every kernel, to a larger workspace, or to more iterations, tunes the shape again, while narrowing it
does not. An incomplete row is benchmarked again when the current ceiling beats the one it was
written under, with 0 beating every finite one, or when the search differs from the one that produced
it. Re-running the same workload with the same settings therefore costs nothing: the search would
measure the same candidates and stop in the same place, so it is not repeated and no second row is
appended. Within a single process a shape is benchmarked at most once in any case, including when an
attempt fails after tuning-start and when a later call cannot use the winner, for example because
it allows less workspace. Such a call runs on another row that supports it, or on default selection,
and a later process that meets it with no row it can use tunes it again. Rows written before these
columns existed count as finished searches.
Use the ceiling to bound how long tuning may run, not as a way to tune faster; to tune faster, reduce the candidate list or the iteration counts.
The ceiling is checked between candidates, since a batch of launches already submitted cannot be recalled. One candidate may therefore overrun it by as much as its own measurement takes. A search that measures every candidate is kept however long it took, because a complete search gives the right answer regardless of its duration.
The time hipblaslt-bench prints for a kernel is its own benchmark, separate from the tuner’s
measurement, so the two can disagree between kernels that are within noise of each other. Both run
the same measurement code and size their rotation by the same rules, but on different buffers: the
tuner rotates copies of your tensors in memory it owns, and leaves block-scaled and w4a16 scales in
place, since only their caller knows their layout. The tuner always times with HIP events;
hipblaslt-bench times with HIP events only with --use_gpu_timer and otherwise uses a
synchronized CPU timer. Treat hipblaslt-bench’s benchmark as the reference when a shape matters
enough to check by hand.
Benchmarking never writes to your buffers. Candidates run against memory the library allocates for the purpose, and only the winner runs on the real output.
Reading the tuning output#
Because the first call on a new shape can block for minutes, cache and tune mode report what
they are doing without needing any logging variable. A first tune run looks like this:
tuning-cache: mode=tune path=tuning.txt load=not-found loaded=0
tuning-cache: tuning-start m=2048 n=1024 k=2048 batch=1 trans=TN types=f16_r/f16_r; this call will block until it finishes
tuning-cache: tuning-done winner=247670 elapsed=145.7s persisted=yes
tuning-cache: summary shapes=1 matched=1 fellback=0 tuned=1 invalidated=0
Replaying that cache prints only the first and last lines, since a cache hit says nothing per call:
tuning-cache: mode=cache path=tuning.txt load=ok loaded=1
tuning-cache: summary shapes=1 matched=1 fellback=0 tuned=0 invalidated=0
load is ok, not-found, read-error or no-path, the last meaning
HIPBLASLT_TUNING_CACHE_PATH is unset and tuning does nothing.
The summary is how you confirm a deployed cache is actually being used. shapes counts distinct
problems the process looked up, not calls, so it stays comparable with loaded. matched is how
many of them a cache entry served and fellback how many fell through to default selection; those
are the shapes still worth tuning. tuned counts distinct shapes this run benchmarked, whether or
not their winners reached the file; persisted on each tuning-done line is what tells you that.
invalidated counts entries rejected because a rebuild moved the kernel they named. The summary is
written as the process exits normally, so a run that is killed or aborts does not produce one, and it
is left out entirely when a process loaded no entries and looked nothing up.
The output is bounded. A shape that cannot be tuned reports its reason once rather than once per call, and shapes are announced at most once each. A shape declined for one call can still be tuned on a later one, and that tune is reported:
tuning-cache: tuning-skipped in-place C==D with nonzero beta cannot be measured without mutating its input; using default selection
tuning-partial in place of tuning-done means the budget stopped the search and its best
candidate was recorded as incomplete. tuning-skipped means the tuner understood the problem and
declined it, which is expected for some shapes forever. tuning-fallback means an attempt failed and is worth investigating; an
attempt that failed after tuning-start is not retried in the same process.
Setting HIPBLASLT_LOG_LEVEL=4 keeps all of the above, timestamps it like the rest of the library
log, and adds per-problem cache hit, miss and invalidation lines, the scratch and candidate setup,
a progress heartbeat every ten seconds during a long search, and the winner’s measured time against
the baseline. At that level nothing is collapsed, so every attempt is reported. See
hipBLASLt environment variables for where each of these is written.
Limitations#
Tuning happens on the C API execution path. Callers of the C++ extension API can replay a cache but do not tune.
Grouped GEMM is not served from the cache, and neither is a pointer-array batch called with
algo=nullptr. Grouped GEMM, pointer-array batch, rocRoller, fused all-to-all calls and HIP graph capture are excluded from tuning. A fused all-to-all call writes to its peer ranks, which its candidates would do too, and the ranks do not coordinate a search.Calls dispatched to rocRoller bypass direct cache replay when
algo=nullptr. This includes calls usingHIPBLASLT_USE_ROCROLLER=1and block-scaled calls selected for rocRoller by default. Cache rows currently accept only nonnegative Tensile solution indexes; rocRoller uses negative encoded indexes.An uncached in-place problem where
CandDalias andbetais nonzero runs with normal selection instead of being tuned. Repeated benchmark launches would otherwise overwrite and reuse C rather than measuring the caller’s input.One process should write a given cache file at a time. Multi-process jobs should tune in a single-rank warm-up and then run in
cachemode; see above.alpha,betaand whetherCandDalias are not part of the lookup key, so an entry tuned at one value ofbetacan serve a caller using another.An explicit
algopassed tohipblasLtMatmulalways launches as given, except that in adaptivesm_count_targetmode the top heuristic result can be swapped for the solution picked at the current hint (see Using Origami and Stream-K with hipBLASLt). A caller that reuses an algorithm it queried before its shape was tuned does not pick up the winner until it queries the heuristic again.