hipBLASLtExt Operation Reference#

hipBLASLtExt Operation API Reference#

hipblasltExtSoftmax()#

hipblasStatus_t hipblasltExtSoftmax(hipDataType datatype, uint32_t m, uint32_t n, uint32_t dim, void *output, void *input, hipStream_t stream)#

Perform softmax on given tensor.

This function computes softmax on given 2D-tensor along specified dimension.

Parameters:

  • datatype – [in] Datatype of input/output tensor, currently support HIP_R_32F only.

  • m – [in] The first dimension of input/output tensor.

  • n – [in] The second dimension of input/output tensor. Currently only values less than or equal to 256 are supported.

  • dim – [in] Specified dimension to perform softmax on. Currently 1 is the only valid value.

  • input – [in] input tensor buffer.

  • stream – [in] The HIP stream where all the GPU work will be submitted.

  • output – [out] output tensor buffer.

Return values:

  • HIPBLAS_STATUS_SUCCESS – If it runs successfully.

  • HIPBLAS_STATUS_INVALID_VALUE – If n is greater than 256.

  • HIPBLAS_STATUS_NOT_SUPPORTED – If dim is not 1 or datatype is not HIP_R_32F.

hipblasltExtLayerNorm()#

hipblasStatus_t hipblasltExtLayerNorm(hipDataType datatype, void *output, void *mean, void *invvar, void *input, uint32_t m, uint32_t n, float eps, void *gamma, void *beta, hipStream_t stream)#

Perform 2-D layernorm on with source input tensor and result output tensor.

This function computes layernorm on given 2D-tensor.

Parameters:

  • datatype – [in] Datatype of input/output tensor, currently support HIP_R_32F only.

  • output – [out] output tensor buffer. can’t be nullptr.

  • mean – [out] tensor buffer. can’t be nullptr.

  • invvar – [out] tensor buffer. 1 / sqrt(std). can’t be nullptr.

  • input – [in] tensor buffer. can’t be nullptr.

  • m – [in] The first dimension of input/output tensor.

  • n – [in] The second dimension of input/output tensor.

  • eps – [in] for sqrt to avoid inf value.

  • gamma – [in] tensor buffer. nullptr means calculation doesn’t involve gamma.

  • beta – [in] tensor buffer. nullptr means calculation doesn’t involve beta.

  • stream – [in] The HIP stream where all the GPU work will be submitted.

Return values:

  • HIPBLAS_STATUS_SUCCESS – If it runs successfully.

  • HIPBLAS_STATUS_INVALID_VALUE – If m is greater than 4096.

  • HIPBLAS_STATUS_NOT_SUPPORTED – if datatype is not HIP_R_32F.

hipblasltExtAMax()#

hipblasStatus_t hipblasltExtAMax(const hipDataType datatype, const hipDataType outDatatype, void *output, void *input, uint32_t m, uint32_t n, hipStream_t stream)#

Perform absmax on given 2-D tensor and output one value absmax(tensor) value.

This function computes amax on given 1D-tensor.

Parameters:

  • datatype – [in] Datatype of input/output tensor, currently support HIP_R_32F and HIP_R_16F only.

  • outDatatype – [in] Datatype of input/output tensor, currently support HIP_R_32F and HIP_R_16F only.

  • output – [out] output tensor buffer. can’t be nullptr.

  • input – [in] tensor buffer. can’t be nullptr.

  • m – [in] The first dimension of input/output tensor.

  • n – [in] The first dimension of input/output tensor.

  • stream – [in] The HIP stream where all the GPU work will be submitted.

Return values:

  • HIPBLAS_STATUS_SUCCESS – If it runs successfully.

  • HIPBLAS_STATUS_INVALID_VALUE – If m is greater than 4096.

  • HIPBLAS_STATUS_NOT_SUPPORTED – if datatype is not (HIPBLASLT_R_32F or HIPBLASLT_R_16F).

hipblasLtExt Operation Usage#

Introduction#

hipBLASLt has extension operation APIs which is independent to gemm operation with. These extensions support:

  1. hipblasltExtSoftmax
    Softmax for 2D-tensor. Currently it performs softmax on second dimension of input tensor and it assumes input is contigious on second dimension.
    For sample usage, please refer to clients/benchmarks/client_ext_op_softmax.cpp
  2. hipblasltExtLayerNorm
    Convert a 2D tensor by LayerNorm to generate a new 2D normalized tensor.
    it is a independent function which can just call and get result.
    sample code is in clients/samples/ext_op/sample_hipblaslt_ext_op_layernorm.cpp
  3. hipblasltExtAMax
    Abs Maximum value of a 2D tensor.
    it is a independent function which can just call and get result.
    sample code is in clients/samples/ext_op/sample_hipblaslt_ext_op_amax.cpp