rocSPARSE user guide

rocSPARSE user guide#

This topic discusses how to use rocSPARSE, including a discussion of device and stream management, storage formats, pointer mode, and how hipSPARSE interacts with rocSPARSE.

HIP device management#

Before starting a HIP kernel, you can call hipSetDevice() to set the device to run the kernel on, for example, device 2. Unless you explicitly specify a different device, HIP kernels always run on device 0. This HIP (and CUDA) device management approach is not specific to the rocSPARSE library. rocSPARSE honors this approach and assumes you have already set the preferred device before a rocSPARSE routine call.

After you set the device, you can create a handle with rocsparse_create_handle(). Subsequent rocSPARSE routines take this handle as an input parameter. rocSPARSE only queries the specified device (using hipGetDevice()) and does not set the device for users. It’s your responsibility to provide a valid device to rocSPARSE and ensure device safety. If it’s not a valid device, rocSPARSE returns an error message.

The handle should be destroyed at the end using rocsparse_destroy_handle() to release the resources consumed by the rocSPARSE library. You cannot switch devices between rocsparse_create_handle() and rocsparse_destroy_handle(). To change the device, you must destroy the current handle and create another rocSPARSE handle on a new device.

Note

hipSetDevice() and hipGetDevice() are not part of the rocSPARSE API. They are part of the HIP Device Management API.

HIP stream management#

HIP kernels are always launched in a queue, which is also known as a stream. If you don’t explicitly specify a stream, the system provides and maintains a default stream, which you cannot create or destroy. However, you can freely create a new stream using hipStreamCreate() and bind it to a rocSPARSE handle using rocsparse_set_stream(). The rocSPARSE routines invoke HIP kernels. A rocSPARSE handle is always associated with a stream, which rocSPARSE passes to the kernels inside the routine. One rocSPARSE routine only takes one stream in a single invocation. If you create a stream, you are responsible for destroying it. See the HIP Stream Management API for more information.

Asynchronous execution#

All rocSPARSE library functions are non-blocking and execute asynchronously with respect to the host, except for functions which allocate memory themselves, preventing asynchronicity. These functions might return immediately or before the actual computation has finished. To force synchronization, use either hipDeviceSynchronize() or hipStreamSynchronize(). This ensures all previously executed rocSPARSE functions on the device or the stream have completed.

Multiple streams and multiple devices#

If a system has multiple HIP devices, you can run multiple rocSPARSE handles concurrently. However, you cannot run a single rocSPARSE handle concurrently on multiple discrete devices. Each handle can only be associated with a single device, and a new handle should be created for each additional device.

Graph support for rocSPARSE#

Many of the rocSPARSE functions can be captured into a graph node using the HIP Graph Management APIs. See Functions supported with graph capture to determine whether a rocSPARSE routine is supported or not. For a list of graph-related HIP APIs, see the HIP Graph Management API.

The following code creates a graph with rocsparse_function() as the graph node.

CHECK_HIP_ERROR((hipStreamBeginCapture(stream, hipStreamCaptureModeGlobal));
rocsparse_<function>(<arguments>);
CHECK_HIP_ERROR(hipStreamEndCapture(stream, &graph));

The captured graph can be launched as shown below:

CHECK_HIP_ERROR(hipGraphInstantiate(&instance, graph, NULL, NULL, 0));
CHECK_HIP_ERROR(hipGraphLaunch(instance, stream));

Graph support requires asynchronous HIP APIs.

Functions supported with graph capture#

The following functions support graph capture:

Sparse level 1 functions#

Function name	yes	no
`rocsparse_Xaxpyi()`	x
`rocsparse_Xdoti()`	x
`rocsparse_Xdotci()`	x
`rocsparse_Xgthr()`	x
`rocsparse_Xgthrz()`	x
`rocsparse_Xroti()`	x
`rocsparse_Xsctr()`	x

Sparse level 2 functions#

Function name	yes	no
`rocsparse_Xbsrmv_analysis()`		x
`rocsparse_bsrmv_clear()`		x
`rocsparse_Xbsrmv()`	x
`rocsparse_Xbsrxmv()`	x
`rocsparse_Xbsrsv_buffer_size()`	x
`rocsparse_Xbsrsv_analysis()`		x
`rocsparse_bsrsv_zero_pivot()`		x
`rocsparse_bsrsv_clear()`
`rocsparse_Xbsrsv_solve()`	x
`rocsparse_Xcoomv()`	x
`rocsparse_Xcsrmv_analysis()`		x
`rocsparse_Xcsrmv()`	x
`rocsparse_csrmv_clear()`		x
`rocsparse_Xcsrsv_buffer_size()`	x
`rocsparse_Xcsrsv_analysis()`		x
`rocsparse_csrsv_zero_pivot()`		x
`rocsparse_csrsv_clear()`		x
`rocsparse_Xcsrsv_solve()`	x
`rocsparse_Xcsritsv_buffer_size()`		x
`rocsparse_Xcsritsv_analysis()`		x
`rocsparse_csritsv_zero_pivot()`		x
`rocsparse_csritsv_clear()`		x
`rocsparse_Xcsritsv_solve()`		x
`rocsparse_Xcsritsv_solve()`		x
`rocsparse_Xellmv()`	x
`rocsparse_Xgebsrmv()`	x
`rocsparse_Xgemvi_buffer_size()`	x
`rocsparse_Xgemvi()`	x
`rocsparse_Xhybmv()`	x

Sparse level 3 functions#

Function name	yes	no
`rocsparse_Xcsrmm()`	x
`rocsparse_Xcsrsm_buffer_size()`	x
`rocsparse_Xcsrsm_analysis()`		x
`rocsparse_csrsm_zero_pivot()`		x
`rocsparse_csrsm_clear()`		x
`rocsparse_Xcsrsm_solve()`	x
`rocsparse_Xbsrmm()`	x
`rocsparse_Xbsrsm_buffer_size()`	x
`rocsparse_Xbsrsm_analysis()`		x
`rocsparse_bsrsm_zero_pivot()`		x
`rocsparse_bsrsm_clear()`		x
`rocsparse_Xbsrsm_solve()`	x
`rocsparse_Xgebsrmm()`	x
`rocsparse_Xgemmi()`	x

Sparse extra functions#

Function name	yes	no
`rocsparse_bsrgeam_nnzb()`		x
`rocsparse_Xbsrgeam()`		x
`rocsparse_Xbsrgemm_buffer_size()`		x
`rocsparse_bsrgemm_nnzb()`		x
`rocsparse_Xbsrgemm()`		x
`rocsparse_csrgeam_nnz()`		x
`rocsparse_Xcsrgeam()`		x
`rocsparse_Xcsrgemm_buffer_size()`		x
`rocsparse_csrgemm_nnz()`		x
`rocsparse_csrgemm_symbolic()`		x
`rocsparse_Xcsrgemm()`		x
`rocsparse_Xcsrgemm_numeric()`		x

Preconditioner functions#

Function name	yes	no
`rocsparse_Xbsric0_buffer_size()`	x
`rocsparse_Xbsric0_analysis()`		x
`rocsparse_bsric0_zero_pivot()`		x
`rocsparse_bsric0_clear()`		x
`rocsparse_Xbsric0()`	x
`rocsparse_Xbsrilu0_buffer_size()`	x
`rocsparse_Xbsrilu0_analysis()`		x
`rocsparse_bsrilu0_zero_pivot()`		x
`rocsparse_Xbsrilu0_numeric_boost()`	x
`rocsparse_bsrilu0_clear()`		x
`rocsparse_Xbsrilu0()`	x
`rocsparse_Xcsric0_buffer_size()`	x
`rocsparse_Xcsric0_analysis()`		x
`rocsparse_csric0_zero_pivot()`		x
`rocsparse_csric0_clear()`		x
`rocsparse_Xcsric0()`	x
`rocsparse_Xcsrilu0_buffer_size()`	x
`rocsparse_Xcsrilu0_numeric_boost()`	x
`rocsparse_Xcsrilu0_analysis()`		x
`rocsparse_csrilu0_zero_pivot()`		x
`rocsparse_csrilu0_clear()`		x
`rocsparse_Xcsrilu0()`	x
`rocsparse_csritilu0_buffer_size()`		x
`rocsparse_csritilu0_preprocess()`		x
`rocsparse_Xcsritilu0_compute()`		x
`rocsparse_Xcsritilu0_compute_ex()`		x
`rocsparse_Xcsritilu0_history()`		x
`rocsparse_Xgtsv_buffer_size()`	x
`rocsparse_Xgtsv()`	x
`rocsparse_Xgtsv_no_pivot_buffer_size()`	x
`rocsparse_Xgtsv_no_pivot()`	x
`rocsparse_Xgtsv_no_pivot_strided_batch_buffer_size()`	x
`rocsparse_Xgtsv_no_pivot_strided_batch()`	x
`rocsparse_Xgtsv_interleaved_batch_buffer_size()`	x
`rocsparse_Xgtsv_interleaved_batch()`	x
`rocsparse_Xgpsv_interleaved_batch_buffer_size()`	x
`rocsparse_Xgpsv_interleaved_batch()`	x

Conversion functions#

Function name	yes	no
`rocsparse_csr2coo()`	x
`rocsparse_csr2csc_buffer_size()`	x
`rocsparse_Xcsr2csc()`	x
`rocsparse_Xgebsr2gebsc_buffer_size()`	x
`rocsparse_Xgebsr2gebsc()`	x
`rocsparse_csr2ell_width()`	x
`rocsparse_Xcsr2ell()`	x
`rocsparse_Xcsr2hyb()`	x
`rocsparse_csr2bsr_nnz()`		x
`rocsparse_Xcsr2bsr()`		x
`rocsparse_csr2gebsr_nnz()`		x
`rocsparse_Xcsr2gebsr_buffer_size()`	x
`rocsparse_Xcsr2gebsr()`		x
`rocsparse_coo2csr()`	x
`rocsparse_ell2csr_nnz()`	x
`rocsparse_Xell2csr()`	x
`rocsparse_hyb2csr_buffer_size()`	x
`rocsparse_Xhyb2csr()`	x
`rocsparse_Xbsr2csr()`	x
`rocsparse_Xgebsr2csr()`	x
`rocsparse_Xgebsr2gebsr_buffer_size()`	x
`rocsparse_gebsr2gebsr_nnz()`		x
`rocsparse_Xgebsr2gebsr()`		x
`rocsparse_Xcsr2csr_compress()`		x
`rocsparse_create_identity_permutation()`	x
`rocsparse_inverse_permutation()`	x
`rocsparse_cscsort_buffer_size()`	x
`rocsparse_cscsort()`		x
`rocsparse_csrsort_buffer_size()`	x
`rocsparse_csrsort()`		x
`rocsparse_coosort_buffer_size()`	x
`rocsparse_coosort_by_row()`		x
`rocsparse_coosort_by_column()`		x
`rocsparse_Xdense2csr()`		x
`rocsparse_Xdense2csc()`		x
`rocsparse_Xdense2coo()`		x
`rocsparse_Xcsr2dense()`	x
`rocsparse_Xcsc2dense()`	x
`rocsparse_Xcoo2dense()`	x
`rocsparse_Xnnz_compress()`		x
`rocsparse_Xnnz()`		x
`rocsparse_Xprune_dense2csr_buffer_size()`	x
`rocsparse_Xprune_dense2csr_nnz()`		x
`rocsparse_Xprune_dense2csr()`		x
`rocsparse_Xprune_csr2csr_buffer_size()`	x
`rocsparse_Xprune_csr2csr_nnz()`		x
`rocsparse_Xprune_csr2csr()`		x
`rocsparse_Xprune_dense2csr_by_percentage_buffer_size()`	x
`rocsparse_Xprune_dense2csr_nnz_by_percentage()`		x
`rocsparse_Xprune_dense2csr_by_percentage()`		x
`rocsparse_Xprune_csr2csr_by_percentage_buffer_size()`	x
`rocsparse_Xprune_csr2csr_nnz_by_percentage()`		x
`rocsparse_Xprune_csr2csr_by_percentage()`		x
`rocsparse_Xbsrpad_value()`	x

Reordering functions#

Function name	yes	no
`rocsparse_Xcsrcolor()`		x

Utility functions#

Function name	yes	no
`rocsparse_Xcheck_matrix_csr_buffer_size()`		x
`rocsparse_Xcheck_matrix_csr()`		x
`rocsparse_Xcheck_matrix_csc_buffer_size()`		x
`rocsparse_Xcheck_matrix_csc()`		x
`rocsparse_Xcheck_matrix_coo_buffer_size()`		x
`rocsparse_Xcheck_matrix_coo()`		x
`rocsparse_Xcheck_matrix_gebsr_buffer_size()`		x
`rocsparse_Xcheck_matrix_gebsr()`		x
`rocsparse_Xcheck_matrix_gebsc_buffer_size()`		x
`rocsparse_Xcheck_matrix_gebsc()`		x
`rocsparse_Xcheck_matrix_ell_buffer_size()`		x
`rocsparse_Xcheck_matrix_ell()`		x
`rocsparse_check_matrix_hyb_buffer_size()`		x
`rocsparse_check_matrix_hyb()`		x

Sparse generic functions#

Function name	yes	no
`rocsparse_axpby()`	x
`rocsparse_gather()`	x
`rocsparse_scatter()`	x
`rocsparse_rot()`	x
`rocsparse_spvv()`		x
`rocsparse_sparse_to_dense()`		x
`rocsparse_dense_to_sparse()`		x
`rocsparse_spgemm()`		x
`rocsparse_v2_spmv_buffer_size()`		x
`rocsparse_spgeam_buffer_size()`		x
`rocsparse_spgeam()`		x
`rocsparse_sptrsv_buffer_size()`		x
`rocsparse_sptrsm_buffer_size()`		x
`rocsparse_sddmm_buffer_size()`		x
`rocsparse_sddmm_preprocess()`		x
`rocsparse_sparse_to_sparse_buffer_size()`		x
`rocsparse_sparse_to_sparse()`		x
`rocsparse_extract_buffer_size()`	x
`rocsparse_extract_nnz()`	x
`rocsparse_extract()`	x

For rocsparse_spmv(), rocsparse_spmm(), rocsparse_spsv(), and rocsparse_spsm(), hipGraph is supported when passing the buffer size or compute stages but is not supported when passing the preprocess stage.

For rocsparse_v2_spmv(), rocsparse_v2_sptrsv(), and rocsparse_v2_sptrsm(), hipGraph is supported when passing the compute stage but is not supported when passing the analysis stage.

For rocsparse_sddmm(), hipGraph is supported only when using the default algorithm.

Storage formats#

For information on the rocSPARSE sparse array storage formats, see storage formats.

Pointer mode#

The auxiliary functions rocsparse_set_pointer_mode() and rocsparse_get_pointer_mode() are used to set and get the value of the state variable rocsparse_pointer_mode. If rocsparse_pointer_mode is equal to rocsparse_pointer_mode_host, then scalar parameters must be allocated on the host. If rocsparse_pointer_mode is equal to rocsparse_pointer_mode_device, then scalar parameters must be allocated on the device.

There are two types of scalar parameter:

Scaling parameters, such as alpha and beta, used, for example, in rocsparse_scsrmv() and rocsparse_scoomv().
Scalar results from functions such as rocsparse_sdoti() or rocsparse_cdotci().

For scalar parameters such as alpha and beta, memory can be allocated on the host heap or stack when rocsparse_pointer_mode is equal to rocsparse_pointer_mode_host. The kernel launch is asynchronous, and if the scalar parameter is on the heap, it can be freed after the kernel launch returns. When rocsparse_pointer_mode is equal to rocsparse_pointer_mode_device, the scalar parameter must not be changed until the kernel completes.

For scalar results, when rocsparse_pointer_mode is equal to rocsparse_pointer_mode_host, the function blocks the CPU until the GPU has copied the result back to the host. When rocsparse_pointer_mode is equal to rocsparse_pointer_mode_device, the function returns after the asynchronous launch. Similar to the vector and matrix results, the scalar result is only available when the kernel has completed execution.

Activity logging [Deprecated]#

Four different environment variables can be set to enable logging in rocSPARSE: ROCSPARSE_LAYER, ROCSPARSE_LOG_TRACE_PATH, ROCSPARSE_LOG_BENCH_PATH, and ROCSPARSE_LOG_DEBUG_PATH.

ROCSPARSE_LAYER is a bit mask that enables logging, where several logging modes for rocsparse_layer_mode can be specified as follows:

`ROCSPARSE_LAYER` not set	Logging is disabled.
`ROCSPARSE_LAYER` set to `1`	Trace logging is enabled.
`ROCSPARSE_LAYER` set to `2`	Bench logging is enabled.
`ROCSPARSE_LAYER` set to `3`	Trace logging and bench logging are enabled.
`ROCSPARSE_LAYER` set to `4`	Debug logging is enabled.
`ROCSPARSE_LAYER` set to `5`	Trace logging and debug logging are enabled.
`ROCSPARSE_LAYER` set to `6`	Bench logging and debug logging are enabled.
`ROCSPARSE_LAYER` set to `7`	Trace logging and bench logging and debug logging are enabled.

When logging is enabled, each rocSPARSE function call writes the function name and function arguments to the logging stream. The default logging output is streamed to stderr.

Note

Performance will degrade when logging is enabled. By default, the environment variable ROCSPARSE_LAYER is not set and logging is disabled.

To capture activity logging in a file, set the following environment variables as required:

ROCSPARSE_LOG_TRACE_PATH specifies a path and file name to capture trace logging streamed to that file.
ROCSPARSE_LOG_BENCH_PATH specifies a path and file name to capture bench logging.
ROCSPARSE_LOG_DEBUG_PATH specifies a path and file name to capture debug logging.

Note

If the file cannot be opened, the logging output is streamed to stderr.

Warning

Trace, debug, and bench logging is deprecated and will be removed in a future release

ROC-TX support in rocSPARSE#

The ROC-TX library contains application code instrumentation APIs to support the high-level correlation of runtime API or activity events. When integrated with rocSPARSE, ROC-TX enables users to view the call stack of rocSPARSE and HIP API functions in profiling tools such as rocProfiler, offering better insights into runtime behavior and performance bottlenecks.

To enable ROC-TX profiling, set the environment variable ROCSPARSE_ROCTX=1 when running the program with rocProf:

ROCSPARSE_ROCTX=1 /opt/rocm/bin/rocprofv3 --kernel-trace --marker-trace --hip-trace --output-format pftrace -- ./example_program

This will generate a .pftrace file which can then be viewed using the Perfetto UI.

Note

ROC-TX support in rocSPARSE is unavailable on Windows and is not supported in the static library version on Linux.

rocSPARSE Fortran bindings#

Fortran functionality for rocSPARSE is offered by hipFORT. For more information, consult the hipFORT API file list.

hipSPARSE#

hipSPARSE is a SPARSE marshalling library with multiple supported backends. It sits between the application and a “worker” SPARSE library, marshalling inputs into the backend library and marshalling results back to the application. hipSPARSE exports an interface that does not require the client to change, regardless of the chosen backend. hipSPARSE supports rocSPARSE and NVIDIA CUDA cuSPARSE as backends.

hipSPARSE focuses on convenience and portability. If performance outweighs these factors, then it’s best to use rocSPARSE itself. hipSPARSE can be found on GitHub.

rocSPARSE user guide

Contents

rocSPARSE user guide#

HIP device management#

HIP stream management#

Asynchronous execution#

Multiple streams and multiple devices#

Graph support for rocSPARSE#

Functions supported with graph capture#

Sparse level 1 functions#

Sparse level 2 functions#

Sparse level 3 functions#

Sparse extra functions#

Preconditioner functions#

Conversion functions#

Reordering functions#

Utility functions#

Sparse generic functions#

Storage formats#

Pointer mode#

Activity logging [Deprecated]#

ROC-TX support in rocSPARSE#

rocSPARSE Fortran bindings#

hipSPARSE#