hipBLASLtExt operation API reference#
hipBLASLt has the following extension operation APIs that are independent to gemm operations. These extensions support:
hipblasltExtSoftmax
Softmax for 2D-tensor. Currently, it performs softmax on the second dimension of input tensor and assumes the input to be contigious on the second dimension. For sample code, refer to client_extop_softmax.cpp.
hipblasltExtLayerNorm
Converts a 2D tensor using LayerNorm to generate a new 2D normalized tensor. it is an independent function used to just call and get result. For sample code, refer to sample_hipblaslt_ext_op_layernorm.cpp.
hipblasltExtAMax
Abs maximum value of a 2D tensor. it is an independent function used to just call and get result. For sample code, refer to sample_hipblaslt_ext_op_amax.cpp.
hipblasltExtAMaxWithScale
Abs maximum value and scaled output of a 2D tensor. it is an independent function used to just call and get result. For sample code, refer to sample_hipblaslt_ext_op_amax_with_scale.cpp.
These APIs are explained in detail below.
hipblasltExtSoftmax()#
-
hipblasStatus_t hipblasltExtSoftmax(hipDataType datatype, uint32_t m, uint32_t n, uint32_t dim, void *output, void *input, hipStream_t stream)#
Perform softmax on given tensor.
This function computes softmax on given 2D-tensor along specified dimension.
- Parameters:
datatype – [in] Datatype of input/output tensor, currently support HIP_R_32F only.
m – [in] The first dimension of input/output tensor.
n – [in] The second dimension of input/output tensor. Currently only values less than or equal to 256 are supported.
dim – [in] Specified dimension to perform softmax on. Currently 1 is the only valid value.
input – [in] input tensor buffer.
stream – [in] The HIP stream where all the GPU work will be submitted.
output – [out] output tensor buffer.
- Return values:
HIPBLAS_STATUS_SUCCESS – If it runs successfully.
HIPBLAS_STATUS_INVALID_VALUE – If
n
is greater than 256.HIPBLAS_STATUS_NOT_SUPPORTED – If
dim
is not 1 ordatatype
is not HIP_R_32F.
hipblasltExtLayerNorm()#
-
hipblasStatus_t hipblasltExtLayerNorm(hipDataType datatype, void *output, void *mean, void *invvar, void *input, uint32_t m, uint32_t n, float eps, void *gamma, void *beta, hipStream_t stream)#
Perform 2-D layernorm on with source input tensor and result output tensor.
This function computes layernorm on given 2D-tensor.
- Parameters:
datatype – [in] Datatype of input/output tensor, currently support HIP_R_32F only.
output – [out] output tensor buffer. can’t be nullptr.
mean – [out] tensor buffer. can’t be nullptr.
invvar – [out] tensor buffer. 1 / sqrt(std). can’t be nullptr.
input – [in] tensor buffer. can’t be nullptr.
m – [in] The first dimension of input/output tensor.
n – [in] The second dimension of input/output tensor.
eps – [in] for sqrt to avoid inf value.
gamma – [in] tensor buffer. nullptr means calculation doesn’t involve gamma.
beta – [in] tensor buffer. nullptr means calculation doesn’t involve beta.
stream – [in] The HIP stream where all the GPU work will be submitted.
- Return values:
HIPBLAS_STATUS_SUCCESS – If it runs successfully.
HIPBLAS_STATUS_INVALID_VALUE – If
m
is greater than 4096.HIPBLAS_STATUS_NOT_SUPPORTED – if
datatype
is not HIP_R_32F.
hipblasltExtAMax()#
-
hipblasStatus_t hipblasltExtAMax(const hipDataType datatype, const hipDataType outDatatype, void *output, void *input, uint32_t m, uint32_t n, hipStream_t stream)#
Perform absmax on given 2-D tensor and output one value absmax(tensor) value.
This function computes amax on given 2D-tensor.
- Parameters:
datatype – [in] Datatype of input tensor, currently support HIP_R_32F and HIP_R_16F only.
outDatatype – [in] Datatype of output tensor, currently support HIP_R_32F and HIP_R_16F only.
output – [out] Amax tensor buffer. can’t be nullptr.
input – [in] 2-D tensor buffer. can’t be nullptr.
m – [in] The first dimension of input/output tensor.
n – [in] The second dimension of input/output tensor.
stream – [in] The HIP stream where all the GPU work will be submitted.
- Return values:
HIPBLAS_STATUS_SUCCESS – If it runs successfully.
HIPBLAS_STATUS_INVALID_VALUE – If
m
or n is 0, or input or output is nullptr.HIPBLAS_STATUS_NOT_SUPPORTED – If
datatype
is not (HIP_R_32F or HIP_R_16F).
hipblasltExtAMaxWithScale()#
-
hipblasStatus_t hipblasltExtAMaxWithScale(const hipDataType datatype, const hipDataType outDatatype, const hipDataType scaleDatatype, void *output, void *outputD, void *input, void *inputScale, uint32_t m, uint32_t n, hipStream_t stream)#
Perform absmax and scaling on given 2-D tensor. Generate one absmax value and scaled 2-D tensor output.
This function computes amax and scaling on given 2D-tensor.
- Parameters:
datatype – [in] Datatype of input tensor, currently support HIP_R_32F only.
outDatatype – [in] Datatype of output tensor, currently support HIP_R_32F and HIP_R_16F only.
scaleDatatype – [in] Datatype of outputD tensor, currently support HIP_R_8F_E4M3_FNUZ and HIP_R_8F_E5M2_FNUZ only.
output – [out] Amax tensor buffer. can’t be nullptr.
outputD – [out] scaled 2-D tensor buffer. can’t be nullptr.
input – [in] 2-D tensor buffer. can’t be nullptr.
inputScale – [in] 1-D tensor buffer. can’t be nullptr. only support float.
m – [in] The first dimension of input/output tensor.
n – [in] The second dimension of input/output tensor.
stream – [in] The HIP stream where all the GPU work will be submitted.
- Return values:
HIPBLAS_STATUS_SUCCESS – If it runs successfully.
HIPBLAS_STATUS_INVALID_VALUE – If
m
or n is 0, or input, inputScale, output, or outputD is nullptr.HIPBLAS_STATUS_NOT_SUPPORTED – If
datatype
is not HIP_R_32F, or scaleDatatype is not HIP_R_8F_E4M3_FNUZ or HIP_R_8F_E5M2_FNUZ.