hipBLASLtExt operation API reference#

hipBLASLt has the following extension operation APIs that are independent to gemm operations. These extensions support:

  1. hipblasltExtSoftmax

    Softmax for 2D-tensor. Currently, it performs softmax on the second dimension of input tensor and assumes the input to be contigious on the second dimension. For sample code, refer to client_extop_softmax.cpp.

  2. hipblasltExtLayerNorm

    Converts a 2D tensor using LayerNorm to generate a new 2D normalized tensor. it is an independent function used to just call and get result. For sample code, refer to sample_hipblaslt_ext_op_layernorm.cpp.

  3. hipblasltExtAMax

    Abs maximum value of a 2D tensor. it is an independent function used to just call and get result. For sample code, refer to sample_hipblaslt_ext_op_amax.cpp.

  4. hipblasltExtAMaxWithScale

    Abs maximum value and scaled output of a 2D tensor. it is an independent function used to just call and get result. For sample code, refer to sample_hipblaslt_ext_op_amax_with_scale.cpp.

These APIs are explained in detail below.

hipblasltExtSoftmax()#

hipblasStatus_t hipblasltExtSoftmax(hipDataType datatype, uint32_t m, uint32_t n, uint32_t dim, void *output, void *input, hipStream_t stream)#

Perform softmax on given tensor.

This function computes softmax on given 2D-tensor along specified dimension.

Parameters:

  • datatype – [in] Datatype of input/output tensor, currently support HIP_R_32F only.

  • m – [in] The first dimension of input/output tensor.

  • n – [in] The second dimension of input/output tensor. Currently only values less than or equal to 256 are supported.

  • dim – [in] Specified dimension to perform softmax on. Currently 1 is the only valid value.

  • input – [in] input tensor buffer.

  • stream – [in] The HIP stream where all the GPU work will be submitted.

  • output – [out] output tensor buffer.

Return values:

  • HIPBLAS_STATUS_SUCCESS – If it runs successfully.

  • HIPBLAS_STATUS_INVALID_VALUE – If n is greater than 256.

  • HIPBLAS_STATUS_NOT_SUPPORTED – If dim is not 1 or datatype is not HIP_R_32F.

hipblasltExtLayerNorm()#

hipblasStatus_t hipblasltExtLayerNorm(hipDataType datatype, void *output, void *mean, void *invvar, void *input, uint32_t m, uint32_t n, float eps, void *gamma, void *beta, hipStream_t stream)#

Perform 2-D layernorm on with source input tensor and result output tensor.

This function computes layernorm on given 2D-tensor.

Parameters:

  • datatype – [in] Datatype of input/output tensor, currently support HIP_R_32F only.

  • output – [out] output tensor buffer. can’t be nullptr.

  • mean – [out] tensor buffer. can’t be nullptr.

  • invvar – [out] tensor buffer. 1 / sqrt(std). can’t be nullptr.

  • input – [in] tensor buffer. can’t be nullptr.

  • m – [in] The first dimension of input/output tensor.

  • n – [in] The second dimension of input/output tensor.

  • eps – [in] for sqrt to avoid inf value.

  • gamma – [in] tensor buffer. nullptr means calculation doesn’t involve gamma.

  • beta – [in] tensor buffer. nullptr means calculation doesn’t involve beta.

  • stream – [in] The HIP stream where all the GPU work will be submitted.

Return values:

  • HIPBLAS_STATUS_SUCCESS – If it runs successfully.

  • HIPBLAS_STATUS_INVALID_VALUE – If m is greater than 4096.

  • HIPBLAS_STATUS_NOT_SUPPORTED – if datatype is not HIP_R_32F.

hipblasltExtAMax()#

hipblasStatus_t hipblasltExtAMax(const hipDataType datatype, const hipDataType outDatatype, void *output, void *input, uint32_t m, uint32_t n, hipStream_t stream)#

Perform absmax on given 2-D tensor and output one value absmax(tensor) value.

This function computes amax on given 2D-tensor.

Parameters:

  • datatype – [in] Datatype of input tensor, currently support HIP_R_32F and HIP_R_16F only.

  • outDatatype – [in] Datatype of output tensor, currently support HIP_R_32F and HIP_R_16F only.

  • output – [out] Amax tensor buffer. can’t be nullptr.

  • input – [in] 2-D tensor buffer. can’t be nullptr.

  • m – [in] The first dimension of input/output tensor.

  • n – [in] The second dimension of input/output tensor.

  • stream – [in] The HIP stream where all the GPU work will be submitted.

Return values:

  • HIPBLAS_STATUS_SUCCESS – If it runs successfully.

  • HIPBLAS_STATUS_INVALID_VALUE – If m or n is 0, or input or output is nullptr.

  • HIPBLAS_STATUS_NOT_SUPPORTED – If datatype is not (HIP_R_32F or HIP_R_16F).

hipblasltExtAMaxWithScale()#

hipblasStatus_t hipblasltExtAMaxWithScale(const hipDataType datatype, const hipDataType outDatatype, const hipDataType scaleDatatype, void *output, void *outputD, void *input, void *inputScale, uint32_t m, uint32_t n, hipStream_t stream)#

Perform absmax and scaling on given 2-D tensor. Generate one absmax value and scaled 2-D tensor output.

This function computes amax and scaling on given 2D-tensor.

Parameters:

  • datatype – [in] Datatype of input tensor, currently support HIP_R_32F only.

  • outDatatype – [in] Datatype of output tensor, currently support HIP_R_32F and HIP_R_16F only.

  • scaleDatatype – [in] Datatype of outputD tensor, currently support HIP_R_8F_E4M3_FNUZ and HIP_R_8F_E5M2_FNUZ only.

  • output – [out] Amax tensor buffer. can’t be nullptr.

  • outputD – [out] scaled 2-D tensor buffer. can’t be nullptr.

  • input – [in] 2-D tensor buffer. can’t be nullptr.

  • inputScale – [in] 1-D tensor buffer. can’t be nullptr. only support float.

  • m – [in] The first dimension of input/output tensor.

  • n – [in] The second dimension of input/output tensor.

  • stream – [in] The HIP stream where all the GPU work will be submitted.

Return values:

  • HIPBLAS_STATUS_SUCCESS – If it runs successfully.

  • HIPBLAS_STATUS_INVALID_VALUE – If m or n is 0, or input, inputScale, output, or outputD is nullptr.

  • HIPBLAS_STATUS_NOT_SUPPORTED – If datatype is not HIP_R_32F, or scaleDatatype is not HIP_R_8F_E4M3_FNUZ or HIP_R_8F_E5M2_FNUZ.