.. meta:: :description: Programmers guide for the rocBLAS documentation and API reference library :keywords: rocBLAS, ROCm, API, Linear Algebra, documentation, programming, .. _programmers-guide: ******************************************************************** rocBLAS programming guide ******************************************************************** This topic covers internal details that are required to program with rocBLAS. It includes a discussion of the source code organization, stream and device management, device memory allocation, and other technical considerations. ================================ Source code organization ================================ The rocBLAS code can be found in the `rocBLAS folder `_ of the `rocm-libraries GitHub `_. It is split into three major parts: * The ``library`` directory contains all the source code for the library. * The ``clients`` directory contains all the test code and code to build clients. * Infrastructure such as ``docs`` and ``cmake`` to support the library. The library directory ----------------------- Here is the structure of the ``library`` directory, which contains the following content for rocBLAS. library/include ^^^^^^^^^^^^^^^^^^^^^^^^ This directory contains C98 include files for the external API. These files also contain Doxygen comments that document the API. library/src/blas[1,2,3] ^^^^^^^^^^^^^^^^^^^^^^^^ This includes source code for Level 1, 2, and 3 BLAS functions in ``.cpp`` and ``.hpp`` files. * The ``*.cpp`` files contain the following: * External C functions that call or instantiate templated functions with an ``_impl`` extension. * The ``_impl`` functions, which have argument checking and logging. In turn, they call functions with a ``_template`` extension. * The ``*_imp.hpp`` files contain: * ``_template`` functions that can be exported to rocSOLVER. They usually call the ``_launcher`` functions. * API implementations that can be instantiated for the original APIs with integer arguments using ``rocblas_int`` and again for the ILP64 API with integer arguments as ``int64_t``. * The ``*_kernels.cpp`` files contain: * ``_launcher`` functions that invoke or launch kernels using ``ROCBLAS_LAUNCH_KERNEL`` or related macros. * ``_kernel`` functions that run on the device. library/src/blas_ex ^^^^^^^^^^^^^^^^^^^^^^^^ This contains the source code for mixed-precision BLAS. library/src/src64 ^^^^^^^^^^^^^^^^^^^^^^^^ This directory contains the ILP64 source code for Level 1, 2, and 3 BLAS and mixed-precision functions in ``blas_ex``. The files normally end with ``_64`` before the file type extension, for example, ``_64.cpp``. The API integers are ``int64_t`` instead of ``rocblas_int``. The function behavior is kept identical at the higher level by instantiable macros and C++ templates. Only at the kernel dispatch level does the code diverge by providing a ``_64`` version, in which the invocation is controlled using the ``ROCBLAS_API`` macro. The directory structure mirrors the level organization used for the parent directory ``library/src``. Device kernel code ^^^^^^^^^^^^^^^^^^^^^^^^ Most BLAS device functions (kernels) are C++ templated functions based on the data type. In C++ host code, any duplicate instantiations of templates can be handled by the linker and the duplicates are ignored. LLVM device code instantiations, however, are not handled this way, so you must avoid duplicate instantiations in multiple code units. Kernel templates should therefore only be provided as C++ template prototypes in the include files unless they must be instantiated. Try to instantiate all forms in a single unit, for example, in a ``.cpp`` file, and expose a launcher C++ interface to invoke the device calls where possible. This is especially important for ILP64 implementations where it's best to reuse the LP64 instantiations without any duplication to avoid bloating the library size. library/src/blas3/Tensile ^^^^^^^^^^^^^^^^^^^^^^^^^ This includes code for calling Tensile from rocBLAS and YAML files with Tensile tuning configurations. library/src/include ^^^^^^^^^^^^^^^^^^^^^^^^ This includes the internal include files for: * Handle-related code * Device memory allocation * Logging * Numerical checking * Utility code The clients directory ----------------------- The ``clients`` directory contains all test code and code to build clients. clients/gtest ^^^^^^^^^^^^^^^^^^^^^^^^ Code for the rocblas-test client. This client is used to test rocBLAS. clients/benchmarks ^^^^^^^^^^^^^^^^^^^^^^^^ Code for the rocblas-benchmark client. This client is used to benchmark rocBLAS functions. clients/include ^^^^^^^^^^^^^^^^^^^^^^^^ This contains code for testing and benchmarking individual rocBLAS functions and utility code for testing. The test harness functions are templated by data type and are defined in separate files for each function form: non-batched, batched, and strided_batched. When a function also supports the ILP64 API, then both forms can be tested by the same template and are controlled by the ``Arguments`` API member variable. This follows the pattern for Fortran API testing and includes ``FORTRAN_64`` for the ILP64 format. The code for benchmarking ``gemm_ex`` (in ``testing_gemm_ex.hpp``) using rocblas-bench tries to reuse device memory between consecutive calls. To get the best memory reuse, leading to better performance, the larger GEMMs should be listed first in the YAML input file. Device memory is only reused between consecutive calls with the same precision, so GEMM operations should be grouped by their precisions. clients/common ^^^^^^^^^^^^^^^^^^^^^^^^ This folder includes common code used by both rocblas-benchmark and rocblas-test. clients/samples ^^^^^^^^^^^^^^^^^^^^^^^^ This directory contains sample code for calling the rocBLAS functions. Infrastructure -------------- * CMake is used to build and package rocBLAS. There are ``CMakeLists.txt`` files throughout the code. * Doxygen, Breathe, Sphinx, and ReadTheDocs are used to produce the documentation. The content for the documentation is taken from these sources: * Doxygen comments in the include files in the directory ``library/include`` * Files in the ``docs`` folder. * Jenkins is used to automate continuous integration (CI) testing. * clang-format is used to format the C++ code. ===================================== Handle, stream, and device management ===================================== This section covers handles, streams, and devices, including multiple streams and devices. Handle ------- You must create a ``rocBLAS_handle``, as shown below, before calling any other rocBLAS functions: .. code-block:: cpp rocblas_handle handle; if(rocblas_create_handle(&handle) != rocblas_status_success) return EXIT_FAILURE; After you have finished calling the rocBLAS functions, destroy the handle: .. code-block:: cpp if(rocblas_destroy_handle(handle) != rocblas_status_success) return EXIT_FAILURE; The handle you create uses the default stream and device. To use a non-default stream and non-default device, follow this approach: .. code-block:: cpp int deviceId = non_default_device_id; if(hipSetDevice(deviceId) != hipSuccess) return EXIT_FAILURE; //optional call to rocblas_initialize rocblas_initialize(); // note the order, call hipSetDevice before hipStreamCreate hipStream_t stream; if(hipStreamCreate(&stream) != hipSuccess) return EXIT_FAILURE; rocblas_handle handle; if(rocblas_create_handle(&handle) != rocblas_status_success) return EXIT_FAILURE; if(rocblas_set_stream(handle, stream) != rocblas_status_success) return EXIT_FAILURE; To use the library with a non-default device within a host thread, the device must be set using ``hipSetDevice()`` before creating the handle. The device in the host thread must not be changed between ``hipStreamCreate`` and ``hipStreamDestroy``. If the device in the host thread is changed between creating and destroying the stream, then the behavior is undefined. If you create a non-default stream, it is your responsibility to synchronize the old non-default stream and update the rocBLAS handle with the default or new non-default stream before destroying the old non-default stream. .. code-block:: cpp // Synchronize the non-default stream before destroying it if(hipStreamSynchronize(stream) != hipSuccess) return EXIT_FAILURE; // Reset the stream reference in the handle to either default or new non-default if(rocblas_set_stream(handle, 0) != rocblas_status_success) return EXIT_FAILURE; if(hipStreamDestroy(stream) != hipSuccess) return EXIT_FAILURE; .. note:: It is essential to reset the rocBLAS handle stream reference to avoid a ``hipErrorContextIsDestroyed`` error, which is handled internally. If this step is skipped, you might encounter this error with ``AMD_LOG_LEVEL`` logging or when using ``hipPeekAtLastError( )``. When switching from one non-default stream to another, you must complete all rocBLAS operations previously submitted with this handle on the old stream using the ``hipStreamSynchronize(old_stream)`` API before setting the new stream. .. code-block:: cpp // Synchronize the old stream if(hipStreamSynchronize(old_stream) != hipSuccess) return EXIT_FAILURE; // Create a new stream (this step can be done before the steps above) if(hipStreamCreate(&new_stream) != hipSuccess) return EXIT_FAILURE; // Set the handle to use the new stream (must come after synchronization & before deletion of old stream) if(rocblas_set_stream(handle, new_stream) != rocblas_status_success) return EXIT_FAILURE; // Destroy the old stream (this step is optional but must come after synchronization) if(hipStreamDestroy(old_stream) != hipSuccess) return EXIT_FAILURE; The call to ``hipStreamSynchronize`` above is necessary because the ``rocBLAS_handle`` contains allocated device memory that must not be shared by multiple asynchronous streams at the same time. If either the old or new stream is the default or ``NULL`` stream, it is not necessary to synchronize the old stream before destroying it, or before setting the new stream, because the synchronization is implicit. .. note:: You can switch from one non-default stream to another without calling ``hipStreamSynchronize()`` by enabling stream-order memory allocation. For more information, see :ref:`stream order alloc`. Creating the handle incurs a startup cost. There is an additional startup cost for GEMM functions to load GEMM kernels for a specific device. You can shift the GEMM startup cost to occur later after setting the device by calling ``rocblas_initialize()`` after calling ``hipSetDevice()``. This needs to happen once for each device. If you have two rocBLAS handles which use the same device, then you only need to call ``rocblas_initialize()`` once. If ``rocblas_initialize()`` is not called, then the first GEMM call incurs the startup cost. The ``rocBLAS_handle`` stores the following information: * Stream * Logging mode * Pointer mode * Atomics mode Stream and device management ----------------------------- HIP kernels are launched in a queue, which is also known as a stream. A stream represents a queue of work on a particular device. A ``rocBLAS_handle`` always has one stream, while a stream is always associated with one device. The ``rocBLAS_handle`` is passed as an argument to all rocBLAS functions that launch kernels. These kernels are launched in the handle's stream to run on that stream's device. If you do not create a stream, the ``rocBLAS_handle`` uses the default or ``NULL`` stream, which is maintained by the system. You cannot create or destroy the default stream. However, you can create a new non-default stream and bind it to the ``rocBLAS_handle`` using the commands ``hipStreamCreate()`` and ``rocblas_set_stream()``. rocBLAS supports non-blocking streams for functions requiring synchronization to guarantee results on the host. For functions like ``rocblas_Xnrm2``, the scalar result is copied from device to host when ``rocblas_pointer_mode == rocblas_pointer_mode_host``. This is accomplished by using ``hipMemcpyAsync()``, followed by ``hipStreamSynchronize()``. The stream in the ``rocBLAS_handle`` is synchronized. .. note:: The rocBLAS functions :any:`rocblas_set_vector`, :any:`rocblas_get_vector`, :any:`rocblas_set_matrix`, and :any:`rocblas_get_matrix` block on the default stream and are exceptions to the pattern above. If you create a stream, you are responsible for destroying it using ``hipStreamDestroy()``. If the handle has to switch from one non-default stream to another, then the old stream needs to be synchronized. After that, you need to create and set the new non-default stream using ``hipStreamCreate()`` and ``rocblas_set_stream()``, respectively. Then you can optionally destroy the old stream. HIP has two important device management functions: * ``hipSetDevice()``: Sets the default device to be used for subsequent HIP API calls from the thread. * ``hipGetDevice()``: Returns the default device ID for the calling host thread. The device which was set using ``hipSetDevice()`` when ``hipStreamCreate()`` was called is the one that is associated with a stream. If the device was not set using ``hipSetDevice()``, then the default device is used. You cannot switch the device in a stream between ``hipStreamCreate()`` and ``hipStreamDestroy()``. To use another device, create another stream. rocBLAS never sets a device. It only queries the device using ``hipGetDevice()``. If rocBLAS does not see a valid device, it returns an error message. Multiple streams and multiple devices ------------------------------------- If a machine has ``num`` GPU devices, they have the ``deviceID`` numbers 0, 1, 2, and so forth, which are equivalent to ``num - 1``. The default device has ``deviceID == 0``. Each ``rocBLAS_handle`` can only be used with a single device, but you can run ```` handles on ```` devices concurrently. .. _Device Memory allocation in detail: ======================== Device memory allocation ======================== This section presents the requirements and design details for rocBLAS device memory allocation, along with a series of examples. Requirements ------------- The following list of requirements motivate the design implementation for device memory allocation. * Some rocBLAS functions require temporary device memory. * Allocating and deallocating device memory is expensive and needs synchronizing. * Temporary device memory should be recycled across multiple rocBLAS function calls using the same ``rocblas_handle``. * The following schemes need to be supported: * **Default**: Functions allocate required device memory automatically. This has the disadvantage that allocation is a synchronizing event. * **Preallocate**: Query all the functions called using a ``rocblas_handle`` to find out how much device memory is needed. Preallocate the required device memory when the ``rocblas_handle`` is created. There are no more synchronizing allocations or deallocations. * **Manual**: Query a function to find out how much device memory is required. Allocate and deallocate the device memory before and after the function calls. This allows the user to control where the synchronizing allocation and deallocation occur. In all of the above schemes, temporary device memory needs to be held by the ``rocblas_handle`` and recycled if a subsequent function using the handle needs it. Design ------ * rocBLAS uses per-handle device memory allocation with out-of-band management. * The state of the device memory is stored in the ``rocblas_handle``. * For the rocBLAS user: * Functions are provided to query how much device memory a function needs. * An environment variable is provided to preallocate when the ``rocblas_handle`` is created. * Functions are provided to manually allocate and deallocate after the ``rocblas_handle`` is created. * The following two values are added to the ``rocblas_status`` enum to indicate how a rocBLAS function is changing the state of the temporary device memory in the ``rocblas_handle``: * ``rocblas_status_size_unchanged`` * ``rocblas_status_size_increased`` * For the rocBLAS developer: * Functions are provided to answer device memory size queries. * Functions are provided to allocate temporary device memory. * Opaque RAII objects are used to hold the temporary device memory. Allocated memory is returned to the handle automatically when it is no longer needed. The functions for the rocBLAS user are described in the :ref:`api-reference-guide`. The functions for the rocBLAS developer are described below. Answering device memory size queries in functions that need memory ------------------------------------------------------------------ Functions should contain code like the sample below to answer a query on how much temporary device memory is required. In this case, ``m * n * sizeof(T)`` bytes of memory is required. Here is an example: .. code-block:: c++ rocblas_status rocblas_function(rocblas_handle handle, ...) { if(!handle) return rocblas_status_invalid_handle; if (handle->is_device_memory_size_query()) { size_t size = m * n * sizeof(T); return handle->set_optimal_device_memory_size(size); } // rest of function } is_device_memory_size_query function ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: c++ bool _rocblas_handle::is_device_memory_size_query() const Indicates if the current function call is collecting information about the optimal device memory allocation size. Return value: * **true**: Information is being collected * **false**: Information is not being collected set_optimal_device_memory_size function ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: c++ rocblas_status _rocblas_handle::set_optimal_device_memory_size(size...) Sets the optimal sizes of device memory buffers in bytes for this function. The sizes are rounded up to the next multiple of 64 (or some other chunk size), and the running maximum is updated. Return value: * **rocblas_status_size_unchanged**: The maximum optimal device memory size did not change. This is the case where the function does not use device memory. * **rocblas_satus_size_increased**: The maximum optimal device memory size increased. * **rocblas_status_internal_error**: This function is not supposed to be collecting size information. rocblas_sizeof_datatype function ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: c++ size_t rocblas_sizeof_datatype(rocblas_datatype type) Returns the size of a rocBLAS data type. Answering device memory size queries in functions that do not need memory -------------------------------------------------------------------------- Here is an example: .. code-block:: c++ rocblas_status rocblas_function(rocblas_handle handle, ...) { if(!handle) return rocblas_status_invalid_handle; RETURN_ZERO_DEVICE_MEMORY_SIZE_IF_QUERIED(handle); // rest of function } RETURN_ZERO_DEVICE_MEMORY_SIZE_IF_QUERIED macro ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: c++ RETURN_ZERO_DEVICE_MEMORY_SIZE_IF_QUERIED(handle) This is a convenience macro that returns ``rocblas_status_size_unchanged`` if the function call is a memory size query. rocBLAS kernel device memory allocation ----------------------------------------- Device memory can be allocated for ``n`` floats using ``device_malloc`` as in this example: .. code-block:: c++ auto workspace = handle->device_malloc(n * sizeof(float)); if (!workspace) return rocblas_status_memory_error; float* ptr = static_cast(workspace); Example ^^^^^^^ To allocate multiple buffers: .. code-block:: c++ size_t size1 = m * n; size_t size2 = m * k; auto workspace = handle->device_malloc(size1, size2); if (!workspace) return rocblas_status_memory_error; void * w_buf1, * w_buf2; w_buf1 = workspace[0]; w_buf2 = workspace[1]; device_malloc function ^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: c++ auto workspace = handle->device_malloc(size...) * Returns an opaque RAII object lending allocated device memory to a particular rocBLAS function. * The object returned is convertible to ``void *`` or other pointer types if only one size is specified. * The individual pointers can be accessed using the subscript ``operator[]``. * The lifetime of the returned object is the lifetime of the borrowed device memory (RAII). * To simplify and optimize the code, only one successful allocation object can be alive at a time. * If the handle's device memory is currently being managed by rocBLAS, as in the default scheme, it is expanded in size as necessary. * If the user allocated (or pre-allocated) an explicit size of device memory, then that size is used as the limit, and no resizing or synchronization ever occurs. Parameters: - **size**: The size in bytes of memory to be allocated. Return value: - **On success**: Returns an opaque RAII object that evaluates to ``true`` when converted to ``bool``. - **On failure**: Returns an opaque RAII object that evaluates to ``false`` when converted to ``bool``. Performance degradation ----------------------- The ``rocblas_status`` enum value ``rocblas_status_perf_degraded`` indicates that a slower algorithm was used because of insufficient device memory for the optimal algorithm. Example ^^^^^^^ .. code-block:: c++ rocblas_status ret = rocblas_status_success; size_t size_for_optimal_algorithm = m + n + k; size_t size_for_degraded_algorithm = m; auto workspace_optimal = handle->device_malloc(size_for_optimal_algorithm); if (workspace_optimal) { // Algorithm using larger optimal memory } else { auto workspace_degraded = handle->device_malloc(size_for_degraded_algorithm); if (workspace_degraded) { // Algorithm using smaller degraded memory ret = rocblas_status_perf_degraded; } else { // Not enough device memory for either optimal or degraded algorithm ret = rocblas_status_memory_error; } } return ret; =================== Thread-safe logging =================== rocBLAS has thread-safe logging. This prevents garbled output when multiple threads are writing to the same file. Thread-safe logging is achieved by using ``rocblas_internal_ostream``, a class that can be used similarly to ``std::ostream``. It provides standardized methods for formatted output to either strings or files. The default constructor for ``rocblas_internal_ostream`` writes to strings, which are thread safe because they are owned by the calling thread. There are also ``rocblas_internal_ostream`` constructors for writing to files. The ``rocblas_internal_ostream::yaml_on`` and ``rocblas_internal_ostream::yaml_off`` I/O modifiers turn YAML formatting mode on and off. ``rocblas_cout`` and ``rocblas_cerr`` are the thread-safe versions of ``std::cout`` and ``std::cerr``. Many output identifiers have been marked "poisoned" in rocblas-test and rocblas-bench, to detect the use of non-thread-safe I/O. These include ``std::cout``, ``std::cerr``, ``printf``, ``fprintf``, ``fputs``, ``puts``, and others. The poisoning is not turned on in the library itself or in the samples to avoid imposing restrictions on the use of these symbols on outside users. ``rocblas_handle`` contains three ``rocblas_internal_ostream`` pointers for logging output: * ``static rocblas_internal_ostream* log_trace_os`` * ``static rocblas_internal_ostream* log_bench_os`` * ``static rocblas_internal_ostream* log_profile_os`` The user can also create ``rocblas_internal_ostream`` pointers and objects outside the handle. The following usage notes apply to ``rocblas_internal_ostream``: * Each ``rocblas_internal_ostream`` associated with a file points to a single ``rocblas_internal_ostream::worker`` with a ``std::shared_ptr`` for writing to the file. The worker is mapped from the device ID and ``inode`` corresponding to the file. More than one ``rocblas_internal_ostream`` can point to the same worker. * This means that if more than one ``rocblas_internal_ostream`` is writing to a single output file, they will share the same ``rocblas_internal_ostream::worker``. * The ``<<`` operator for ``rocblas_internal_ostream`` is overloaded. Output is first accumulated in ``rocblas_internal_ostream::os``, a ``std::ostringstream`` buffer. Each ``rocblas_internal_ostream`` has its own ``os`` ``std::ostringstream`` buffer, so strings in ``os`` are not garbled. * When ``rocblas_internal_ostream.os`` is flushed with either a ``std::endl`` or an explicit flush of ``rocblas_internal_ostream``, then ``rocblas_internal_ostream::worker::send`` pushes the string contents of ``rocblas_internal_ostream.os`` and a promise, which together are called a task, onto ``rocblas_internal_ostream.worker.queue``. * The ``send`` function uses the promise to asynchronously transfer data from ``rocblas_internal_ostream.os`` to ``rocblas_internal_ostream.worker.queue`` and wait for the worker to finish writing the string to the file. It also locks a mutex to ensure pushing the task onto the queue is atomic. * The ``ostream.worker.queue`` contains a number of tasks. When ``rocblas_internal_ostream`` is destroyed, all the ``tasks.string`` in ``rocblas_internal_ostream.worker.queue`` are printed to the ``rocblas_internal_ostream`` file and the ``std::shared_ptr`` to the ``ostream.worker`` is destroyed. If the reference count to the worker becomes ``0``, the worker's thread is sent a zero-length string telling it to exit. =========================== rocBLAS numerical checking =========================== rocBLAS provides the environment variable ``ROCBLAS_CHECK_NUMERICS``, which allows users to debug numerical abnormalities. Setting a value for ``ROCBLAS_CHECK_NUMERICS`` enables checks on the input and the output vectors/matrices of the rocBLAS functions for NaN (not-a-number), zero, infinity, and denormal/subnormal values. Numerical checking is available for the input and the output vectors for all level-1 and level-2 functions. In level 2 functions, only the general (ge) type input and the output matrix can be checked for numerical abnormalities. In level 3, GEMM is the only function to have numerical checking. .. note:: Performance degrades when numerical checking is enabled. ``ROCBLAS_CHECK_NUMERICS`` is a bitwise OR of zero or more bit masks with the following possible values: * ``ROCBLAS_CHECK_NUMERICS = 0``: The variable is not set, so there is no numerical checking. * ``ROCBLAS_CHECK_NUMERICS = 1``: Prints a fully informative message to the console. Indicates whether the input and the output Matrices/Vectors have a NaN, zero, infinity, or denormal value. * ``ROCBLAS_CHECK_NUMERICS = 2``: Prints the result of numerical checking only if the input and the output Matrices/Vectors have a NaN, infinity, or denormal value. * ``ROCBLAS_CHECK_NUMERICS = 4``: Returns ``rocblas_status_check_numeric_fail`` status if there is a NaN, infinity, or denormal value. * ``ROCBLAS_CHECK_NUMERICS = 8``: Ignores denormal values if there are no NaN or infinity values present. Here is an example showing how to use ``ROCBLAS_CHECK_NUMERICS``: .. code-block:: bash ROCBLAS_CHECK_NUMERICS=4 ./rocblas-bench -f gemm -i 1 -j 0 This command returns ``rocblas_status_check_numeric_fail`` if the input and the output matrices of a BLAS level-3 GEMM function have a NaN, infinity, or denormal value. If there are no numerical abnormalities, then ``rocblas_status_success`` is returned. .. note:: In stream capture mode, all numerical checking is skipped and ``rocblas_status_success`` is returned. =============================================== rocBLAS order of argument checking and logging =============================================== Argument checking differs between legacy BLAS and rocBLAS. Legacy BLAS ------------- Legacy BLAS has two types of argument checking: * Error-return for an incorrect argument. Legacy BLAS implements this with a call to the function ``XERBLA``. * Quick-return-success when an argument allows for the subprogram to be a no-operation or a constant result. Level-2 and Level-3 BLAS subprograms have both error-return and quick-return-success. Level-1 BLAS subprograms have only quick-return-success. rocBLAS -------- For a full list of error return codes, see the ``rocblas_status`` enumeration in :ref:`api-datatypes`. * ``rocblas_status_invalid_handle``: If the handle is a NULL pointer. * ``rocblas_status_invalid_size``: For an invalid size, increment, or leading dimension argument. * ``rocblas_status_invalid_value``: For unsupported enum values. * ``rocblas_status_success``: For quick-return-success. * ``rocblas_status_invalid_pointer``: For NULL argument pointers. Differences between rocBLAS and legacy BLAS ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ rocBLAS has the following differences from legacy BLAS: * It has a C API, returning a ``rocblas_status`` type indicating the success of the call. * In rocBLAS, a pointer to a scalar return value is passed as the last argument. In legacy BLAS, the following functions return a scalar result: ``dot``, ``nrm2``, ``asum``, ``amax``, and ``amin``. * The first argument is a ``rocblas_handle`` argument. This is an opaque pointer to rocBLAS resources, corresponding to a single HIP stream. * Scalar arguments like alpha and beta are pointers on either the host or device, controlled by the pointer mode of the rocBLAS handle. In cases where the other arguments do not dictate an early return, if the alpha and beta pointers are ``NULL``, the function returns ``rocblas_status_invalid_pointer``. * Vector and matrix arguments are always pointers to device memory. * When ``rocblas_pointer_mode == rocblas_pointer_mode_host``, the alpha and beta values are inspected. Based on their values, a decision is made regarding which vector and matrix pointers must be dereferenced. If any of the dereferenced pointers is a NULL pointer, ``rocblas_status_invalid_pointer`` is returned. * If ``rocblas_pointer_mode == rocblas_pointer_mode_device``, rocBLAS does NOT check if the vector or matrix pointers will dereference a NULL pointer. This is to avoid slowing down execution to fetch and inspect alpha and beta values. * The ``ROCBLAS_LAYER`` environment variable controls the option to log argument values. * rocBLAS has added functionality, including the following: * batched * strided_batched * mixed precision in ``gemm_ex``, ``gemm_batched_ex``, and ``gemm_strided_batched_ex`` The following changes were made to accommodate the new features: * Changes to the logging functionality. See the Logging section below for more details. * For batched and strided_batched L2 and L3 functions, there is a quick-return-success for ``batch_count == 0`` and an invalid-size error for ``batch_count < 0``. * For batched and strided_batched L1 functions, there is a quick-return-success for ``batch_count <= 0``. * When ``rocblas_pointer_mode == rocblas_pointer_mode_device``, alpha and beta are not copied from the device to host for quick-return-success checks. In this case, the quick-return-success checks are omitted. This still provides a correct result, but the operation is slower. * For strided_batched functions, there is no argument checking for the stride. To access elements in a strided_batched_matrix, for example, the C matrix in GEMM, the zero-based index is calculated as ``i1 + i2 * ldc + i3 * stride_c``, where ``i1 = 0, 1, 2, ..., m-1``, ``i2 = 0, 1, 2, ..., n-1``, and ``i3 = 0, 1, 2, ..., batch_count -1``. An incorrect stride can result in a core dump due a segmentation fault. It can also produce an indeterminate result if there is a memory overlap in the output matrix between different values of ``i3``. Device memory size queries -------------------------- The following details apply when performing queries on the memory size: * When ``handle->is_device_memory_size_query()`` is ``true``, the call is a device memory size query, not a normal call. * No logging should be performed during device memory size queries. * If the rocBLAS kernel doesn't require temporary device memory, the macro ``RETURN_ZERO_DEVICE_MEMORY_SIZE_IF_QUERIED(handle)`` can be called after checking that ``handle != nullptr``. * If the rocBLAS kernel requires temporary device memory, then it should be set, and the kernel returned, by calling ``return handle->set_optimal_device_memory_size(size...)``, where ``size...`` is a list of one or more sizes for different sub-problems. The sizes are rounded up and added. Logging -------- There is logging before a quick-return-success or error-return, except under these circumstances: * When ``handle == nullptr``, it returns ``rocblas_status_invalid_handle``. * When ``handle->is_device_memory_size_query()``, it returns ``true``. Vectors and matrices are logged with their addresses and are always on device memory. Scalar values in device memory are logged as their addresses. Scalar values in host memory are logged as their values, with a ``nullptr`` logged as ``NaN`` (``std::numeric_limits::quiet_NaN()``). rocBLAS control flow -------------------- 1. If ``handle == nullptr``, then return ``rocblas_status_invalid_handle``. 2. If the function does not require temporary device memory, then call the macro ``RETURN_ZERO_DEVICE_MEMORY_SIZE_IF_QUERIED(handle);``. 3. If the function requires temporary device memory and ``handle->is_device_memory_size_query()`` is ``true``, then validate any pointers and arguments required to determine the optimal size of temporary device memory. Return ``rocblas_status_invalid_pointer`` or ``rocblas_status_invalid_size`` if the arguments are invalid. Otherwise return ``handle->set_optimal_device_memory_size(size...);``, where ``size...`` is a list of one or more sizes of temporary buffers, which are allocated later using ``handle->device_malloc(size...)``. 4. Perform logging if it is enabled, ensuring not to dereference ``nullptr`` arguments. 5. Check for unsupported enum values. Return ``rocblas_status_invalid_value`` if an enum value is invalid. 6. Check for invalid sizes. Return ``rocblas_status_invalid_size`` if the size arguments are invalid. 7. Return ``rocblas_status_invalid_pointer`` if any pointers used to determine quick return conditions are ``NULL``. 8. If the quick return conditions are met: * If there is no return value, return ``rocblas_status_success``. * If there is a return value: * If the return value pointer argument is a ``NULL`` pointer, return ``rocblas_status_invalid_pointer``. * Otherwise, return ``rocblas_status_success`` 9. If any pointers not checked in step 7 are ``NULL`` but must be dereferenced, return ``rocblas_status_invalid_pointer``. Only when ``rocblas_pointer_mode == rocblas_pointer_mode_host`` can it be efficiently determined whether some vector/matrix arguments must be dereferenced. 10. (Optional) Allocate device memory, returning ``rocblas_status_memory_error`` if the allocation fails. 11. If all the checks above pass, launch the kernel and return ``rocblas_status_success``. Legacy L1 BLAS "single vector" ------------------------------- Below are four code snippets from NETLIB for "single vector" legacy Level-1 BLAS. They have quick-return-success for ``(n <= 0) || (incx <= 0)``: .. code-block:: bash DOUBLE PRECISION FUNCTION DASUM(N,DX,INCX) IF (N.LE.0 .OR. INCX.LE.0) RETURN DOUBLE PRECISION FUNCTION DNRM2(N,X,INCX) IF (N.LT.1 .OR. INCX.LT.1) THEN return = ZERO SUBROUTINE DSCAL(N,DA,DX,INCX) IF (N.LE.0 .OR. INCX.LE.0) RETURN INTEGER FUNCTION IDAMAX(N,DX,INCX) IDAMAX = 0 IF (N.LT.1 .OR. INCX.LE.0) RETURN IDAMAX = 1 IF (N.EQ.1) RETURN Legacy L1 BLAS "two vector" --------------------------- Below are seven legacy Level-1 BLAS codes from NETLIB. They have quick-return-success for ``(n <= 0)``. In addition, for ``DAXPY``, there is quick-return-success for ``(alpha == 0)``: .. code-block:: SUBROUTINE DAXPY(N,alpha,DX,INCX,DY,INCY) IF (N.LE.0) RETURN IF (alpha.EQ.0.0d0) RETURN SUBROUTINE DCOPY(N,DX,INCX,DY,INCY) IF (N.LE.0) RETURN DOUBLE PRECISION FUNCTION DDOT(N,DX,INCX,DY,INCY) IF (N.LE.0) RETURN SUBROUTINE DROT(N,DX,INCX,DY,INCY,C,S) IF (N.LE.0) RETURN SUBROUTINE DSWAP(N,DX,INCX,DY,INCY) IF (N.LE.0) RETURN DOUBLE PRECISION FUNCTION DSDOT(N,SX,INCX,SY,INCY) IF (N.LE.0) RETURN SUBROUTINE DROTM(N,DX,INCX,DY,INCY,DPARAM) DFLAG = DPARAM(1) IF (N.LE.0 .OR. (DFLAG+TWO.EQ.ZERO)) RETURN Legacy L2 BLAS ----------------- Below are the code snippets from NETLIB for legacy Level-2 BLAS. They have both argument checking and quick-return-success: .. code-block:: SUBROUTINE DGER(M,N,ALPHA,X,INCX,Y,INCY,A,LDA) INFO = 0 IF (M.LT.0) THEN INFO = 1 ELSE IF (N.LT.0) THEN INFO = 2 ELSE IF (INCX.EQ.0) THEN INFO = 5 ELSE IF (INCY.EQ.0) THEN INFO = 7 ELSE IF (LDA.LT.MAX(1,M)) THEN INFO = 9 END IF IF (INFO.NE.0) THEN CALL XERBLA('DGER ',INFO) RETURN END IF IF ((M.EQ.0) .OR. (N.EQ.0) .OR. (ALPHA.EQ.ZERO)) RETURN .. code-block:: SUBROUTINE DSYR(UPLO,N,ALPHA,X,INCX,A,LDA) INFO = 0 IF (.NOT.LSAME(UPLO,'U') .AND. .NOT.LSAME(UPLO,'L')) THEN INFO = 1 ELSE IF (N.LT.0) THEN INFO = 2 ELSE IF (INCX.EQ.0) THEN INFO = 5 ELSE IF (LDA.LT.MAX(1,N)) THEN INFO = 7 END IF IF (INFO.NE.0) THEN CALL XERBLA('DSYR ',INFO) RETURN END IF IF ((N.EQ.0) .OR. (ALPHA.EQ.ZERO)) RETURN .. code-block:: SUBROUTINE DGEMV(TRANS,M,N,ALPHA,A,LDA,X,INCX,BETA,Y,INCY) INFO = 0 IF (.NOT.LSAME(TRANS,'N') .AND. .NOT.LSAME(TRANS,'T') .AND. .NOT.LSAME(TRANS,'C')) THEN INFO = 1 ELSE IF (M.LT.0) THEN INFO = 2 ELSE IF (N.LT.0) THEN INFO = 3 ELSE IF (LDA.LT.MAX(1,M)) THEN INFO = 6 ELSE IF (INCX.EQ.0) THEN INFO = 8 ELSE IF (INCY.EQ.0) THEN INFO = 11 END IF IF (INFO.NE.0) THEN CALL XERBLA('DGEMV ',INFO) RETURN END IF IF ((M.EQ.0) .OR. (N.EQ.0) .OR. ((ALPHA.EQ.ZERO).AND. (BETA.EQ.ONE))) RETURN .. code-block:: SUBROUTINE DTRSV(UPLO,TRANS,DIAG,N,A,LDA,X,INCX) INFO = 0 IF (.NOT.LSAME(UPLO,'U') .AND. .NOT.LSAME(UPLO,'L')) THEN INFO = 1 ELSE IF (.NOT.LSAME(TRANS,'N') .AND. .NOT.LSAME(TRANS,'T') .AND. .NOT.LSAME(TRANS,'C')) THEN INFO = 2 ELSE IF (.NOT.LSAME(DIAG,'U') .AND. .NOT.LSAME(DIAG,'N')) THEN INFO = 3 ELSE IF (N.LT.0) THEN INFO = 4 ELSE IF (LDA.LT.MAX(1,N)) THEN INFO = 6 ELSE IF (INCX.EQ.0) THEN INFO = 8 END IF IF (INFO.NE.0) THEN CALL XERBLA('DTRSV ',INFO) RETURN END IF IF (N.EQ.0) RETURN Legacy L3 BLAS ---------------- Below is a code snippet from NETLIB for legacy Level-3 BLAS dgemm. It has both argument checking and quick-return-success: .. code-block:: SUBROUTINE DGEMM(TRANSA,TRANSB,M,N,K,ALPHA,A,LDA,B,LDB,BETA,C,LDC) NOTA = LSAME(TRANSA,'N') NOTB = LSAME(TRANSB,'N') IF (NOTA) THEN NROWA = M NCOLA = K ELSE NROWA = K NCOLA = M END IF IF (NOTB) THEN NROWB = K ELSE NROWB = N END IF // Test the input parameters. INFO = 0 IF ((.NOT.NOTA) .AND. (.NOT.LSAME(TRANSA,'C')) .AND. + (.NOT.LSAME(TRANSA,'T'))) THEN INFO = 1 ELSE IF ((.NOT.NOTB) .AND. (.NOT.LSAME(TRANSB,'C')) .AND. + (.NOT.LSAME(TRANSB,'T'))) THEN INFO = 2 ELSE IF (M.LT.0) THEN INFO = 3 ELSE IF (N.LT.0) THEN INFO = 4 ELSE IF (K.LT.0) THEN INFO = 5 ELSE IF (LDA.LT.MAX(1,NROWA)) THEN INFO = 8 ELSE IF (LDB.LT.MAX(1,NROWB)) THEN INFO = 10 ELSE IF (LDC.LT.MAX(1,M)) THEN INFO = 13 END IF IF (INFO.NE.0) THEN CALL XERBLA('DGEMM ',INFO) RETURN END IF // Quick return if possible. IF ((M.EQ.0) .OR. (N.EQ.0) .OR. (((ALPHA.EQ.ZERO).OR. (K.EQ.0)).AND. (BETA.EQ.ONE))) RETURN .. raw:: latex \newpage ================================= rocBLAS benchmarking and testing ================================= There are three client executables that can be used with rocBLAS. They are: * rocblas-bench * rocblas-gemm-tune * rocblas-test To build these clients, follow the instructions in the :doc:`../install/Linux_Install_Guide` or :doc:`../install/Windows_Install_Guide` guides. After the build, the rocBLAS clients can be found in the ``rocBLAS/build/release/clients/staging`` directory. .. note:: The ``rocblas-bench`` and ``rocblas-test`` executables use AMD's AOCL-BLAS library with ILP64 support (64-bit integers) as the host reference BLAS to verify correctness. On Linux, ``install.sh -dc`` automatically builds AOCL 5.2 from source. On Windows, AOCL 4.2+ can be manually installed from `AMD Developer Central `_. There is a known issue with thread oversubscription in AOCL-BLAS that can cause these executables to hang. If the number of threads matches the total number of CPU threads, thread oversubscription can occur, which causes the process to hang. To prevent this issue, the number of threads used by the AOCL-BLAS library should be smaller than the number of available CPU cores. You can configure this setting using the ``OMP_NUM_THREADS`` environment variable. For example, on a server with 32 cores, limit the number of threads to 28 by setting ``export OMP_NUM_THREADS=28``. The next three sections provide a brief explanation of each rocBLAS client and how to use it. rocblas-bench -------------- rocblas-bench is used to measure performance and verify the correctness of rocBLAS functions. It includes a command line interface. For more information, run this command: .. code-block:: bash rocBLAS/build/release/clients/staging/rocblas-bench --help The following table lists all the data types in rocBLAS: .. list-table:: Data types in rocBLAS :widths: 25 25 :header-rows: 1 * - Data type - acronym * - Real 16-bit brain floating point - bf16_r * - Real half - f16_r (h) * - Real float - f32_r (s) * - Real double - f64_r (d) * - Complex float - f32_c (c) * - Complex double - f64_c (z) * - Integer 32 - i32_r * - Integer 8 - i8_r All options for problem types in rocBLAS for GEMM are shown here: * N: Not transposed * T: Transposed * C: Complex conjugate (for a real data type, C is the same as T) .. list-table:: Various matrix operations :widths: 25 25 25 :header-rows: 1 * - Problem types - problem_type - Data type * - NN - Cijk_Ailk_Bljk - real/complex * - NT - Cijk_Ailk_Bjlk - real/complex * - TN - Cijk_Alik_Bljk - real/complex * - TT - Cijk_Alik_Bjlk - real/complex * - NC - Cijk_Ailk_BjlkC - complex * - CN - Cijk_AlikC_Bljk - complex * - CC - Cijk_AlikC_BjlkC - complex * - TC - Cijk_Alik_BjlkC - complex * - CT - Cijk_AlikC_Bjlk - complex For example, NT means A * B\ :sup:`T`\. GEMM functions can be divided into two main categories. * HPA functions (HighPrecisionAccumulate) where the compute data type is different from the input data type (A/B): All HPA functions must be called using the ``gemm_ex`` API in rocblas-bench (and not ``gemm``). The ``gemm_ex`` function name consists of three letters: A/B data type, C/D data type, and compute data type. * Non-HPA functions where the input (A/B), output (C/D), and compute data types are all the same: Non-HPA cases can be called using ``gemm`` or ``gemm_ex`` but ``gemm`` is recommended. The following table shows all possible GEMM functions in rocBLAS. .. list-table:: All GEMM functions in rocBLAS :widths: 20 30 10 10 10 :header-rows: 1 * - Function - Kernel name - A/B data type - C/D data type - Compute data type * - hgemm - __HB - f16_r - f16_r - f16_r * - hgemm_batched - __HB_GB - f16_r - f16_r - f16_r * - hgemm_strided_batched - __HB - f16_r - f16_r - f16_r * - sgemm - __SB - f32_r - f32_r - f32_r * - sgemm_batched - __SB_GB - f32_r - f32_r - f32_r * - sgemm_strided_batched - __SB - f32_r - f32_r - f32_r * - dgemm - __DB - f64_r - f64_r - f64_r * - dgemm_batched - __DB_GB - f64_r - f64_r - f64_r * - dgemm_strided_batched - __DB - f64_r - f64_r - f64_r * - cgemm - __CB - f32_c - f32_c - f32_c * - cgemm_batched - __CB_GB - f32_c - f32_c - f32_c * - cgemm_strided_batched - __CB - f32_c - f32_c - f32_c * - zgemm - __ZB - f64_c - f64_c - f64_c * - zgemm_batched - __ZB_GB - f64_c - f64_c - f64_c * - zgemm_strided_batched - __ZB - f64_c - f64_c - f64_c * - HHS - __HHS_BH - f16_r - f16_r - f32_r * - HHS_batched - __HHS_BH_GB - f16_r - f16_r - f32_r * - HHS_strided_batched - __HHS_BH - f16_r - f16_r - f32_r * - HSS - __HSS_BH - f16_r - f32_r - f32_r * - HSS_batched - __HSS_BH_GB - f16_r - f32_r - f32_r * - HSS_strided_batched - __HSS_BH - f16_r - f32_r - f32_r * - BBS - __BBS_BH - bf16_r - bf16_r - f32_r * - BBS_batched - __BBS_BH_GB - bf16_r - bf16_r - f32_r * - BBS_strided_batched - __BBS_BH - bf16_r - bf16_r - f32_r * - BSS - __BSS_BH - bf16_r - f32_r - f32_r * - BSS_batched - __BSS_BH_GB - bf16_r - f32_r - f32_r * - BSS_strided_batched - __BSS_BH - bf16_r - f32_r - f32_r * - I8II - __I8II_BH - I8 - I - I * - I8II_batched - __I8II_BH_GB - I8 - I - I * - I8II_strided_batched - __I8II_BH - I8 - I - I Benchmarking the performance of a GEMM function using rocblas-bench ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This method is only recommended to test a few sizes. Otherwise, refer to the next section. The following listing shows how to configure rocblas-bench to call each of the GEMM functions: Non-HPA cases (``gemm``) .. code-block:: bash #dgemm $ ./rocblas-bench -f gemm --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r d --lda 1024 --ldb 2048 --ldc 1024 --ldd 1024 --alpha 1.1 --beta 1.0 # dgemm batched $ ./rocblas-bench -f gemm_batched --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r d --lda 1024 --ldb 2048 --ldc 1024 --ldd 1024 --alpha 1.1 --beta 1 --batch_count 5 # dgemm strided batched $ ./rocblas-bench -f gemm_strided_batched --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r d --lda 1024 --stride_a 4096 --ldb 2048 --stride_b 4096 --ldc 1024 --stride_c 2097152 --ldd 1024 --stride_d 2097152 --alpha 1.1 --beta 1 --batch_count 5 # sgemm $ ./rocblas-bench -f gemm --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r s --lda 1024 --ldb 2048 --ldc 1024 --ldd 1024 --alpha 1.1 --beta 1 # sgemm batched $ ./rocblas-bench -f gemm_batched --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r s --lda 1024 --ldb 2048 --ldc 1024 --ldd 1024 --alpha 1.1 --beta 1 --batch_count 5 # sgemm strided batched $ ./rocblas-bench -f gemm_strided_batched --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r s --lda 1024 --stride_a 4096 --ldb 2048 --stride_b 4096 --ldc 1024 --stride_c 2097152 --ldd 1024 --stride_d 2097152 --alpha 1.1 --beta 1 --batch_count 5 # hgemm (this function is not really very fast. Use HHS instead, which is faster and more accurate) $ ./rocblas-bench -f gemm --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r h --lda 1024 --ldb 2048 --ldc 1024 --ldd 1024 --alpha 1.1 --beta 1 # hgemm batched $ ./rocblas-bench -f gemm_batched --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r h --lda 1024 --ldb 2048 --ldc 1024 --ldd 1024 --alpha 1.1 --beta 1 --batch_count 5 # hgemm strided batched $ ./rocblas-bench -f gemm_strided_batched --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r h --lda 1024 --stride_a 4096 --ldb 2048 --stride_b 4096 --ldc 1024 --stride_c 2097152 --ldd 1024 --stride_d 2097152 --alpha 1.1 --beta 1 --batch_count 5 # cgemm $ ./rocblas-bench -f gemm --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r c --lda 1024 --ldb 2048 --ldc 1024 --ldd 1024 --alpha 1.1 --beta 1 # cgemm batched $ ./rocblas-bench -f gemm_batched --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r c --lda 1024 --ldb 2048 --ldc 1024 --ldd 1024 --alpha 1.1 --beta 1 --batch_count 5 # cgemm strided batched $ ./rocblas-bench -f gemm_strided_batched --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r c --lda 1024 --stride_a 4096 --ldb 2048 --stride_b 4096 --ldc 1024 --stride_c 2097152 --ldd 1024 --stride_d 2097152 --alpha 1.1 --beta 1 --batch_count 5 # zgemm $ ./rocblas-bench -f gemm --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r z --lda 1024 --ldb 2048 --ldc 1024 --ldd 1024 --alpha 1.1 --beta 1 # zgemm batched $ ./rocblas-bench -f gemm_batched --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r z --lda 1024 --ldb 2048 --ldc 1024 --ldd 1024 --alpha 1.1 --beta 1 --batch_count 5 # zgemm strided batched $ ./rocblas-bench -f gemm_strided_batched --transposeA N --transposeB T -m 1024 -n 2048 -k 512 -r z --lda 1024 --stride_a 4096 --ldb 2048 --stride_b 4096 --ldc 1024 --stride_c 2097152 --ldd 1024 --stride_d 2097152 --alpha 1.1 --beta 1 --batch_count 5 # cgemm (NC) $ ./rocblas-bench -f gemm --transposeA N --transposeB C -m 1024 -n 2048 -k 512 -r c --lda 1024 --ldb 2048 --ldc 1024 --ldd 1024 --alpha 1.1 --beta 1 # cgemm batched (NC) $ ./rocblas-bench -f gemm_batched --transposeA N --transposeB C -m 1024 -n 2048 -k 512 -r c --lda 1024 --ldb 2048 --ldc 1024 --ldd 1024 --alpha 1.1 --beta 1 --batch_count 5 # cgemm strided batched (NC) $ ./rocblas-bench -f gemm_strided_batched --transposeA N --transposeB C -m 1024 -n 2048 -k 512 -r c --lda 1024 --stride_a 4096 --ldb 2048 --stride_b 4096 --ldc 1024 --stride_c 2097152 --ldd 1024 --stride_d 2097152 --alpha 1.1 --beta 1 --batch_count 5 HPA cases (``gemm_ex``) .. code-block:: bash # HHS $ ./rocblas-bench -f gemm_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type h --lda 1024 --b_type h --ldb 2048 --c_type h --ldc 1024 --d_type h --ldd 1024 --compute_type s --alpha 1.1 --beta 1 # HHS batched $ ./rocblas-bench -f gemm_batched_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type h --lda 1024 --b_type h --ldb 2048 --c_type h --ldc 1024 --d_type h --ldd 1024 --compute_type s --alpha 1.1 --beta 1 --batch_count 5 # HHS strided batched $ ./rocblas-bench -f gemm_strided_batched_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type h --lda 1024 --stride_a 4096 --b_type h --ldb 2048 --stride_b 4096 --c_type h --ldc 1024 --stride_c 2097152 --d_type h --ldd 1024 --stride_d 2097152 --compute_type s --alpha 1.1 --beta 1 --batch_count 5 # HSS $ ./rocblas-bench -f gemm_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type h --lda 1024 --b_type h --ldb 2048 --c_type s --ldc 1024 --d_type s --ldd 1024 --compute_type s --alpha 1.1 --beta 1 # HSS batched $ ./rocblas-bench -f gemm_batched_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type h --lda 1024 --b_type h --ldb 2048 --c_type s --ldc 1024 --d_type s --ldd 1024 --compute_type s --alpha 1.1 --beta 1 --batch_count 5 # HSS strided batched $ ./rocblas-bench -f gemm_strided_batched_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type h --lda 1024 --stride_a 4096 --b_type h --ldb 2048 --stride_b 4096 --c_type s --ldc 1024 --stride_c 2097152 --d_type s --ldd 1024 --stride_d 2097152 --compute_type s --alpha 1.1 --beta 1 --batch_count 5 # BBS $ ./rocblas-bench -f gemm_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type bf16_r --lda 1024 --b_type bf16_r --ldb 2048 --c_type bf16_r --ldc 1024 --d_type bf16_r --ldd 1024 --compute_type s --alpha 1.1 --beta 1 # BBS batched $ ./rocblas-bench -f gemm_batched_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type bf16_r --lda 1024 --b_type bf16_r --ldb 2048 --c_type bf16_r --ldc 1024 --d_type bf16_r --ldd 1024 --compute_type s --alpha 1.1 --beta 1 --batch_count 5 # BBS strided batched $ ./rocblas-bench -f gemm_strided_batched_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type bf16_r --lda 1024 --stride_a 4096 --b_type bf16_r --ldb 2048 --stride_b 4096 --c_type bf16_r --ldc 1024 --stride_c 2097152 --d_type bf16_r --ldd 1024 --stride_d 2097152 --compute_type s --alpha 1.1 --beta 1 --batch_count 5 # BSS $ ./rocblas-bench -f gemm_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type bf16_r --lda 1024 --b_type bf16_r --ldb 2048 --c_type s --ldc 1024 --d_type s --ldd 1024 --compute_type s --alpha 1.1 --beta 1 # BSS batched $ ./rocblas-bench -f gemm_batched_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type bf16_r --lda 1024 --b_type bf16_r --ldb 2048 --c_type s --ldc 1024 --d_type s --ldd 1024 --compute_type s --alpha 1.1 --beta 1 --batch_count 5 # BSS strided batched $ ./rocblas-bench -f gemm_strided_batched_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type bf16_r --lda 1024 --stride_a 4096 --b_type bf16_r --ldb 2048 --stride_b 4096 --c_type s --ldc 1024 --stride_c 2097152 --d_type s --ldd 1024 --stride_d 2097152 --compute_type s --alpha 1.1 --beta 1 --batch_count 5 # I8II $ ./rocblas-bench -f gemm_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type i8_r --lda 1024 --b_type i8_r --ldb 2048 --c_type i32_r --ldc 1024 --d_type i32_r --ldd 1024 --compute_type i32_r --alpha 1.1 --beta 1 # I8II batched $ ./rocblas-bench -f gemm_batched_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type i8_r --lda 1024 --b_type i8_r --ldb 2048 --c_type i32_r --ldc 1024 --d_type i32_r --ldd 1024 --compute_type i32_r --alpha 1.1 --beta 1 --batch_count 5 # I8II strided batched $ ./rocblas-bench -f gemm_strided_batched_ex --transposeA N --transposeB T -m 1024 -n 2048 -k 512 --a_type i8_r --lda 1024 --stride_a 4096 --b_type i8_r --ldb 2048 --stride_b 4096 --c_type i32_r --ldc 1024 --stride_c 2097152 --d_type i32_r --ldd 1024 --stride_d 2097152 --compute_type i32_r --alpha 1.1 --beta 1 --batch_count 5 Setting rocblas-bench parameters in a YAML file ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To benchmark many sizes, it is recommended to use rocblas-bench with the batch call to eliminate the latency involved with loading the GEMM library that rocBLAS links to. The batch call takes a YAML file with a list of all problem sizes. You can have multiple sizes of different types in one YAML file. The benchmark setting is different from the direct call to rocblas-bench. A sample setting for each function is listed below. After you have created the YAML file, benchmark the sizes as follows: .. code-block:: bash rocBLAS/build/release/clients/staging/rocblas-bench --yaml problem-sizes.yaml Here are the configurations for each function: Non-HPA cases (``gemm``) .. code-block:: bash # dgemm - { rocblas_function: "rocblas_dgemm", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10 } # dgemm batched - { rocblas_function: "rocblas_dgemm_batched", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5 } # dgemm strided batched - { rocblas_function: "rocblas_dgemm_strided_batched", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5, stride_a: 4096, stride_b: 4096, stride_c: 2097152, stride_d: 2097152 } # sgemm - { rocblas_function: "rocblas_sgemm", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10 } # sgemm batched - { rocblas_function: "rocblas_sgemm_batched", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5 } # sgemm strided batched - { rocblas_function: "rocblas_sgemm_strided_batched", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5, stride_a: 4096, stride_b: 4096, stride_c: 2097152, stride_d: 2097152 } # hgemm - { rocblas_function: "rocblas_hgemm", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10 } # hgemm batched - { rocblas_function: "rocblas_hgemm_batched", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5 } # hgemm strided batched - { rocblas_function: "rocblas_hgemm_strided_batched", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5, stride_a: 4096, stride_b: 4096, stride_c: 2097152, stride_d: 2097152 } # cgemm - { rocblas_function: "rocblas_cgemm", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10 } # cgemm batched - { rocblas_function: "rocblas_cgemm_batched", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5 } # cgemm strided batched - { rocblas_function: "rocblas_cgemm_strided_batched", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5, stride_a: 4096, stride_b: 4096, stride_c: 2097152, stride_d: 2097152 } # zgemm - { rocblas_function: "rocblas_zgemm", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10 } # zgemm batched - { rocblas_function: "rocblas_zgemm_batched", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5 } # zgemm strided batched - { rocblas_function: "rocblas_zgemm_strided_batched", transA: "N", transB: "T", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5, stride_a: 4096, stride_b: 4096, stride_c: 2097152, stride_d: 2097152 } # cgemm - { rocblas_function: "rocblas_cgemm", transA: "N", transB: "C", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10 } # cgemm batched - { rocblas_function: "rocblas_cgemm_batched", transA: "N", transB: "C", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5 } # cgemm strided batched - { rocblas_function: "rocblas_cgemm_strided_batched", transA: "N", transB: "C", M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5, stride_a: 4096, stride_b: 4096, stride_c: 2097152, stride_d: 2097152 } HPA cases (``gemm_ex``) .. code-block:: bash # HHS - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: f16_r, b_type: f16_r, c_type: f16_r, d_type: f16_r, compute_type: f32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10 } # HHS batched - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: f16_r, b_type: f16_r, c_type: f16_r, d_type: f16_r, compute_type: f32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5 } # HHS strided batched - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: f16_r, b_type: f16_r, c_type: f16_r, d_type: f16_r, compute_type: f32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5, stride_a: 4096, stride_b: 4096, stride_c: 2097152, stride_d: 2097152 } # HSS - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: f16_r, b_type: f16_r, c_type: f16_r, d_type: f16_r, compute_type: f32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10 } # HSS batched - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: f16_r, b_type: f16_r, c_type: f32_r, d_type: f32_r, compute_type: f32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5 } # HSS strided batched - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: f16_r, b_type: f16_r, c_type: f32_r, d_type: f32_r, compute_type: f32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5, stride_a: 4096, stride_b: 4096, stride_c: 2097152, stride_d: 2097152 } # BBS - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: bf16_r, b_type: bf16_r, c_type: bf16_r, d_type: bf16_r, compute_type: f32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10 } # BBS batched - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: bf16_r, b_type: bf16_r, c_type: bf16_r, d_type: bf16_r, compute_type: f32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5 } # BBS strided batched - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: bf16_r, b_type: bf16_r, c_type: bf16_r, d_type: bf16_r, compute_type: f32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5, stride_a: 4096, stride_b: 4096, stride_c: 2097152, stride_d: 2097152 } # BSS - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: bf16_r, b_type: bf16_r, c_type: f32_r, d_type: f32_r, compute_type: f32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10 } # BSS batched - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: bf16_r, b_type: bf16_r, c_type: f32_r, d_type: f32_r, compute_type: f32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5 } # BSS strided batched - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: bf16_r, b_type: bf16_r, c_type: f32_r, d_type: f32_r, compute_type: f32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5, stride_a: 4096, stride_b: 4096, stride_c: 2097152, stride_d: 2097152 } # I8II - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: i8_r, b_type: i8_r, c_type: i32_r, d_type: i32_r, compute_type: i32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10 } # I8II batched - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: i8_r, b_type: i8_r, c_type: i32_r, d_type: i32_r, compute_type: i32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5 } # I8II strided batched - { rocblas_function: "rocblas_gemm_ex", transA: "N", transB: "T", a_type: i8_r, b_type: i8_r, c_type: i32_r, d_type: i32_r, compute_type: i32_r, M: 1024, N: 2048, K: 512, lda: 1024, ldb: 2048, ldc: 1024, ldd: 1024, cold_iters: 2, iters: 10, batch_count: 5, stride_a: 4096, stride_b: 4096, stride_c: 2097152, stride_d: 2097152 } For example, the performance of SGEMM using rocblas-bench on an AMD vega20 machine returns the following: .. code-block:: bash ./rocblas-bench -f gemm -r f32_r --transposeA N --transposeB N -m 4096 -n 4096 -k 4096 --alpha 1 --lda 4096 --ldb 4096 --beta 0 --ldc 4096 transA,transB,M,N,K,alpha,lda,ldb,beta,ldc,rocblas-Gflops,us N,N,4096,4096,4096,1,4096,4096,0,4096,11941.5,11509.4 Logging affects performance, so only use it to log the command under evaluation, then run the command without logging to measure performance. Use of logging to advise on rocblas-bench arguments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A useful way of finding the parameters that can be used with ``./rocblas-bench -f gemm`` is to turn on logging by setting the environment variable ``ROCBLAS_LAYER=2``. For example, if the user runs: .. code-block:: bash ROCBLAS_LAYER=2 ./rocblas-bench -f gemm -i 1 -j 0 The above command logs the following: .. code-block:: bash ./rocblas-bench -f gemm -r f32_r --transposeA N --transposeB N -m 128 -n 128 -k 128 --alpha 1 --lda 128 --ldb 128 --beta 0 --ldc 128 The user can copy and change the above command. For example, to change the datatype to IEEE-64 bit and the size to 2048, use the following: .. code-block:: bash ./rocblas-bench -f gemm -r f64_r --transposeA N --transposeB N -m 2048 -n 2048 -k 2048 --alpha 1 --lda 2048 --ldb 2048 --beta 0 --ldc 2048 Benchmarking ILP64 APIs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To measure performance on the ILP64 API functions, when they exist, add the argument ``--api 1`` rather than changing the function name set in ``-f``. .. note:: rocblas-bench has the flag ``-v 1`` for norm based correctness checks against CPU reference. rocblas-bench has the flag ``-t 1`` for equality and tolerance based correctness checks against CPU reference. Benchmarking special case gemv_batched and gemv_strided_batched functions using rocblas-bench ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This covers how to benchmark the performance of special case ``gemv_batched`` and ``gemv_strided_batched`` functions for mixed precision (HSH, HSS, TST, TSS). The following command launches rocblas-bench for ``rocblas_hshgemv_batched`` with half-precision input, single-precision compute, and half-precision output (HSH): .. code-block:: bash ./rocblas-bench -f gemv_batched --a_type f16_r --c_type f16_r --compute_type f32_r --transposeA N -m 128 -n 128 --alpha 1 --lda 128 --incx 1 --beta 1 --incy 1 --batch_count 2 For the above command, instead of using the ``-r`` parameter to specify the precision, pass three additional arguments (``a_type``, ``c_type``, and ``compute_type``) to resolve the ambiguity of using mixed-precision compute. This mixed-precision support is only available for ``gemv_batched``, ``gemv_strided_batched``, and rocBLAS extension functions (for example, ``axpy_ex``, ``scal_ex``, and ``gemm_ex``). For more information, see the :ref:`api-reference-guide`. .. _rocblas_bench_stream_sync: rocblas-bench timing ^^^^^^^^^^^^^^^^^^^^^ rocblas-bench uses ``hipEvent_t`` recording to time API calls and ignore the overhead of any ``hipStreamSynchronize`` call. To switch back to the earlier timing approach that uses ``hipStreamSynchronize`` to ensure work completion, set the environment variable ``ROCBLAS_BENCH_STREAM_SYNC=1``. rocblas-gemm-tune ----------------- rocblas-gemm-tune is used to find the best performing GEMM kernel for each of a given set of GEMM problems. It has a command line interface, which mimics the ``--yaml`` input used by rocblas-bench (see the section above for details). To generate the expected ``--yaml`` input, you can use profile logging by setting the environment variable ``ROCBLAS_LAYER=4``. For more information on rocBLAS logging, see :ref:`logging`. Here is an example input file: .. code-block:: bash - {'rocblas_function': 'gemm_ex', 'transA': 'N', 'transB': 'N', 'M': 320, 'N': 588, 'K': 4096, 'alpha': 1, 'a_type': 'f32_r', 'lda': 320, 'b_type': 'f32_r', 'ldb': 6144, 'beta': 0, 'c_type': 'f32_r', 'ldc': 320, 'd_type': 'f32_r', 'ldd': 320, 'compute_type': 'f32_r', 'device': 0} - {'rocblas_function': 'gemm_ex', 'transA': 'N', 'transB': 'N', 'M': 512, 'N': 3096, 'K': 512, 'alpha': 1, 'a_type': 'f16_r', 'lda': 512, 'b_type': 'f16_r', 'ldb': 512, 'beta': 0, 'c_type': 'f16_r', 'ldc': 512, 'd_type': 'f16_r', 'ldd': 512, 'compute_type': 'f32_r', 'device': 0} The expected output looks like this (the selected solution index will differ): .. code-block:: bash transA,transB,M,N,batch_count,K,alpha,beta,lda,ldb,ldc,input_type,output_type,compute_type,solution_index N,N,320,588,1,4096,1,0,320,6144,320,f32_r,f32_r,f32_r,-3798 N,N,512,3096,1,512,1,0,512,512,512,f16_r,f16_r,f32_r,-4556 Where the far right values (``solution_index``) are the indices of the best performing kernels for those GEMMs in the rocBLAS kernel library. These indices can be used directly in future GEMM calls but cannot be reused across library releases, different device architectures, or across different GEMM backends. The values are the same indices that ``rocblas_gemm_ex_get_solutions`` reports, so they can be passed straight to ``rocblas_gemm_ex`` with ``rocblas_gemm_algo_solution_index`` for both Tensile and hipBLASLt backends. Solutions provided by the legacy Tensile backend are reported as negative indices (with bias), while positive indices belong to the hipBLASLt backend. A problem is left out of the output when its fastest kernel comes from neither backend, as is the case when the internal GEMV kernel wins a problem with ``M`` or ``N`` equal to one. Such a kernel has no index a later call can name, and rocBLAS already selects it by its own heuristics. See `example_user_driven_tuning.cpp `_ for sample code showing how to use kernels directly via their indices. If the output is stored in a file, you can use the results to override the default kernel selection by setting the environment variable ``ROCBLAS_TENSILE_GEMM_OVERRIDE_PATH=``, where ```` points to the file. Because the overrides are applied to the Tensile backend only, an entry whose ``solution_index`` names no Tensile solution is skipped with a warning while the remaining entries are still applied. A load failure is reported only when a file leaves no override in place. Positive indices in an override file are still interpreted as raw one-based Tensile indices for compatibility with files generated by earlier releases or manually get remapped back to Tensile indices. To see which entries were applied, set the environment variable ``TENSILE_DB=0x40000``. Setting ``ROCBLAS_USE_HIPBLASLT=1`` forces the hipBLASLt backend which does not use this override file. The override file is similarly bypassed on architectures where hipBLASLt is the default backend. rocblas-test ------------- rocblas-test is used to perform rocBLAS unit tests. It uses the GoogleTest framework. The tests are in five categories: * quick * pre_checkin * nightly * stress * known_bug To run the quick tests, use the following command: .. code-block:: bash ./rocblas-test --gtest_filter=*quick* To run the other tests using the ``rocblas-test`` command, replace ``*quick*`` with ``*pre_checkin*``, ``*nightly*``, or ``*known_bug*``. The pattern for ``--gtest_filter`` is: .. code-block:: bash --gtest_filter=POSTIVE_PATTERNS[-NEGATIVE_PATTERNS] ``gtest_filter`` can also be used to run tests for a particular function and a particular set of input parameters. For example, to run all quick tests for the function ``rocblas_saxpy``, use this example: .. code-block:: bash ./rocblas-test --gtest_filter=*quick*axpy*f32_r* The default verbosity shows test category totals and specific test failure details, matching an implicit environment variable setting of ``GTEST_LISTENER=NO_PASS_LINE_IN_LOG``. To generate output listing each individual test that runs, use the following command: .. code-block:: bash GTEST_LISTENER=PASS_LINE_IN_LOG ./rocblas-test --gtest_filter=*quick* ``rocblas-test`` can be driven by tests specified in a YAML file using the ``--yaml`` argument. As the pre_checkin and nightly test categories can require hours to run, a short smoke test set is provided using a YAML file. This ``rocblas_smoke.yaml`` test set only takes a few minutes to test a few small problem sizes for every function: .. code-block:: bash ./rocblas-test --yaml rocblas_smoke.yaml The following test situations require additional consideration: * YAML extension for lock-step multiple variable scanning Both rocblas-test and rocblas-bench can use an extension to scan over multiple variables in lock step implemented by the ``Arguments`` class. For this purpose, set the ``Arguments`` member variable ``scan`` to the range to scan over and use ``*c_scan_value`` to retrieve the values. This can be used to avoid all combinations of YAML variable values that are normally generated, for example, ``- { scan: [32..256..32], M: *c_scan_value, N: *c_scan_value, lda: *c_scan_value }``. * Large memory tests (stress category) Some tests in the stress category might attempt to allocate more RAM than available. While these tests should automatically get skipped, in some cases, such as when running in a Docker container, they could result in a process termination. To limit the peak RAM allocation in GB, use this environment variable: .. code-block:: bash ROCBLAS_CLIENT_RAM_GB_LIMIT=32 ./rocblas-test --gtest_filter=*stress* If the ``rocblas-test`` process is being killed due to an out of memory (OOM) condition, then reduce the client limit by half and try again. It is not advisable to run any stress tests if you have less than 8 GB VRAM available to your graphics device. Use the test subtraction syntax if combining with other filters, for example, ``-*stress*``. * Long-running tests The rocblas-test process will be terminated if a single test takes longer than the configured timeout. To change the timeout, use the environment variable ``ROCBLAS_TEST_TIMEOUT``, which takes a value in seconds (with a default of 600 seconds): .. code-block:: bash ROCBLAS_TEST_TIMEOUT=900 ./rocblas-test --gtest_filter=*stress* * Debugging rocblas-test The rocblas-test process internally catches signals which might interfere with debugger use. To disable this feature, set the environment variable ``ROCBLAS_TEST_NO_SIGACTION``: .. code-block:: bash ROCBLAS_TEST_NO_SIGACTION=1 rocgdb ./rocblas-test --gtest_filter=*stress* On Linux the rocblas-test process now aborts when catching signals to log a stack dump. To disable this feature, set the environment variable ``ROCBLAS_TEST_CONTINUE_ON_FATAL_SIGNAL``: .. code-block:: bash ROCBLAS_TEST_CONTINUE_ON_FATAL_SIGNAL=1 ./rocblas-test --gtest_filter=*stress* Adding a new rocBLAS unit test ------------------------------- To add new data-driven tests to the rocBLAS GoogleTest Framework, follow these steps: #. Create a C++ header file with the name ``testing_.hpp`` in the ``include`` subdirectory, with templated functions for a specific rocBLAS routine. Some examples include: * ``testing_gemm.hpp`` * ``testing_gemm_ex.hpp`` In the new ``testing_*.hpp`` file, create a templated function which returns ``void`` and accepts a ``const Arguments&`` parameter, for example: .. code-block:: cpp template void testing_gemm_ex(const Arguments& arg) { // ... } This function is used for a YAML file-driven argument testing. It will be invoked by the dispatch code for each permutation of the YAML-driven parameters. Additionally, a template function for tests that handle bad arguments should be created, as follows: .. code-block:: cpp template void testing_gemv_bad_arg(const Arguments& arg) { // ... } These ``bad_arg`` test function templates can be used to set arguments programmatically when it is simpler than the YAML approach, for example, to pass NULL pointers. It is expected that the member variable values in the ``Arguments`` parameter will not be utilized with the common exception of the ``api`` member variable of ``Arguments`` which can drive selection of C, FORTRAN, C_64, or FORTRAN_64 API bad argument tests. All functions should be generalized with template parameters as much as possible, to avoid copy-and-paste code. In this function, use the following macros and functions to check results: .. code-block:: cpp HIP_CHECK_ERROR Verifies that a HIP call returns success ROCBLAS_CHECK_ERROR Verifies that a rocBLAS call returns success EXPECT_ROCBLAS_STATUS Verifies that a rocBLAS call returns a certain status unit_check_general Check that two answers agree (see unit.hpp) near_check_general Check that two answers are close (see near.hpp) .. code-block:: cpp DAPI_CHECK Verifies either LP64 or ILP64 function form returns success (based on Arguments member variable API) DAPI_EXPECT Verifies either LP64 or ILP64 function form returns a certain status DAPI_DISPATCH Invoke either LP64 or ILP64 function form In addition, you can use GoogleTest macros such as the ones below, provided they are guarded by ``#ifdef GOOGLE_TEST``: .. code-block:: cpp EXPECT_EQ ASSERT_EQ EXPECT_TRUE ASSERT_TRUE ... .. note:: The ``device_vector`` template allocates memory on the device. You must check whether converting the ``device_vector`` to ``bool`` returns ``false``\ . If so, report a HIP memory error and then exit the current function, following this example: .. code-block:: cpp // allocate memory on device device_vector dx(size_x); device_vector dy(size_y); if(!dx || !dy) { CHECK_HIP_ERROR(hipErrorOutOfMemory); return; } The general outline of the function should be: #. Convert any scalar arguments (for example, ``alpha`` and ``beta``\ ) to ``double``. #. If the problem-size arguments are invalid, use a ``safe_size`` to allocate arrays, call the rocBLAS routine with the original arguments, and verify that it returns ``rocblas_status_invalid_size``, then return. #. Set up the host and device arrays (see ``rocblas_vector.hpp`` and ``rocblas_init.hpp``\ ). #. Call a CBLAS or other reference implementation on the host arrays. #. Call rocBLAS using both device pointer mode and host pointer mode, verifying that every rocBLAS call is successful by wrapping it in ``ROCBLAS_CHECK_ERROR()``. #. If ``arg.unit_check`` is enabled, use ``unit_check_general`` or ``near_check_general`` to validate the results. #. If ``arg.norm_check`` is enabled, calculate and print out norms. (This is now deprecated.) #. If ``arg.timing`` is enabled, perform benchmarking (currently under refactoring). #. Create a C++ file with the name ``_gtest.cpp`` in the ``gtest`` subdirectory, where ```` is a non-type-specific shorthand for the function(s) being tested. Here are some examples: * ``gemm_gtest.cpp`` * ``trsm_gtest.cpp`` * ``blas1_gtest.cpp`` In the C++ file, follow these steps: A. Include the header files related to the tests, as well as ``type_dispatch.hpp``, for example: .. code-block:: c++ #include "testing_syr.hpp" #include "type_dispatch.hpp" B. Wrap the body with an anonymous namespace, to minimize namespace collisions: .. code-block:: c++ namespace { C. Create a templated class which accepts any number of type parameters followed by one anonymous trailing type parameter defaulted to ``void`` (to be used with ``enable_if``\ ). Choose the number of type parameters based on how likely it is in the future that the function will support a mixture of that many different types, for example, input type (\ ``Ti``\ ), output type (\ ``To``\ ), and compute type (\ ``Tc``\ ). If the function will never support more than one or two type parameters, then that many can be used. If the function might be expanded later to support mixed types, then those should be planned for ahead of time and placed in the template parameters. Unless the number of type parameters is greater than one and is always fixed, later type parameters should default to earlier ones. This is so that a subset of type arguments can used and that code which works for functions that take one type parameter can be used for functions that take one or more type parameters. Here is an example: .. code-block:: c++ template< typename Ti, typename To = Ti, typename Tc = To, typename = void> Have the primary definition of this class template derive from the ``rocblas_test_invalid`` class, for example: .. code-block:: c++ template struct syr_testing : rocblas_test_invalid { }; D. Create one or more partial specializations of the class template conditionally enabled by the type parameters matching legal combinations of types. If the first type argument is ``void``\ , then these partial specializations must not apply. This is so the default based on ``rocblas_test_invalid`` can behave correctly when ``void`` is passed to indicate failure. In the partial specializations, derive from the ``rocblas_test_valid`` class. In the partial specializations, create a functional ``operator()`` which takes a ``const Arguments&`` parameter and calls the templated test functions (usually in ``include/testing_*.hpp``\ ) with the specialization's template arguments when the ``arg.function`` string matches the function name. If ``arg.function`` does not match any function related to this test, mark it as a test failure. Follow this example: .. code-block:: c++ template struct syr_testing || std::is_same_v> > : rocblas_test_valid { void operator()(const Arguments& arg) { if(!strcmp(arg.function, "syr")) testing_syr(arg); else FAIL() << "Internal error: Test called with unknown function: " << arg.function; } }; E. If necessary, create a type dispatch function for this function (or group of functions it belongs to) in ``include/type_dispatch.hpp``. If possible, use one of the existing dispatch functions, even if it covers a superset of the allowable types. The purpose of ``type_dispatch.hpp`` is to perform runtime type dispatch in a single place, rather than copying it across several test files. The type dispatch function takes a ``template`` template parameter of ``template class`` and a function parameter of type ``const Arguments&``. It looks at the runtime type values in ``Arguments``\ and instantiates the template with one or more static type arguments, corresponding to the dynamic runtime type arguments. It treats the passed template as a functor, passing the ``Arguments`` argument to a particular instantiation of it. The combinations of types handled by this "runtime type to template type instantiation mapping" function can be general because the type combinations which do not apply to a particular test case will have the template argument set to derive from ``rocblas_test_invalid``\ , which will not create any unresolved instantiations. If unresolved instantiation compile or link errors occur, then the ``enable_if<>`` condition in step D needs to be reworked to indicate ``false`` for type combinations which do not apply. The return type of this function needs to be ``auto``\ , picking up the return type of the functor. If the runtime type combinations do not apply, then this function should return ``TEST{}(arg)``\ , where ``TEST`` is the template parameter. However, this is less important than step D above in excluding invalid type combinations with ``enable_if``\ , since this only excludes them at run-time and they need to be excluded at compile-time by step D to avoid unresolved references or invalid instantiations, for example: .. code-block:: c++ template