Layernorm

Layernorm#

MIOpen: Layernorm
Layernorm

Enumerations

enum  miopenNormMode_t {
  MIOPEN_ELEMENTWISE_AFFINE = 0 ,
  MIOPEN_WEIGHT_BIAS ,
  MIOPEN_ELEMENTWISE_AFFINE_FUSED_ADD ,
  MIOPEN_WEIGHT_BIAS_FUSED_ADD = 3 ,
  MIOPEN_ELEMENTWISE_AFFINE_T5 ,
  MIOPEN_WEIGHT_BIAS_T5 = 5
}
 

Functions

miopenStatus_t miopenLayerNormForward (miopenHandle_t handle, miopenNormMode_t mode, const miopenTensorDescriptor_t xDesc, const void *x, const miopenTensorDescriptor_t weightDesc, const void *weight, const miopenTensorDescriptor_t biasDesc, const void *bias, const float epsilon, const int32_t normalized_dim, const miopenTensorDescriptor_t yDesc, void *y, const miopenTensorDescriptor_t meanDesc, void *mean, const miopenTensorDescriptor_t rstdDesc, void *rstd)
 Execute a layernorm forward layer. More...
 
miopenStatus_t miopenAddLayerNormForward (miopenHandle_t handle, miopenNormMode_t mode, const miopenTensorDescriptor_t xDesc, const void *x, const miopenTensorDescriptor_t x2Desc, const void *x2, const miopenTensorDescriptor_t weightDesc, const void *weight, const miopenTensorDescriptor_t biasDesc, const void *bias, const float epsilon, const int32_t normalized_dim, const miopenTensorDescriptor_t yDesc, void *y, const miopenTensorDescriptor_t meanDesc, void *mean, const miopenTensorDescriptor_t rstdDesc, void *rstd)
 Execute a add and layernorm forward layer. More...
 
miopenStatus_t miopenT5LayerNormForward (miopenHandle_t handle, miopenNormMode_t mode, const miopenTensorDescriptor_t xDesc, const void *x, const miopenTensorDescriptor_t weightDesc, const void *weight, const float epsilon, const miopenTensorDescriptor_t yDesc, void *y, const miopenTensorDescriptor_t rstdDesc, void *rstd)
 Execute a T5layernorm forward layer. More...
 
miopenStatus_t miopenGetT5LayerNormBackwardWorkspaceSize (miopenHandle_t handle, miopenNormMode_t mode, const miopenTensorDescriptor_t dyDesc, const miopenTensorDescriptor_t xDesc, const miopenTensorDescriptor_t weightDesc, const miopenTensorDescriptor_t rstdDesc, const miopenTensorDescriptor_t dxDesc, const miopenTensorDescriptor_t dwDesc, size_t *sizeInBytes)
 Helper function to query the minimum workspace size required by the T5layernorm backward call. More...
 
miopenStatus_t miopenT5LayerNormBackward (miopenHandle_t handle, miopenNormMode_t mode, void *workspace, size_t workspaceSizeInBytes, const miopenTensorDescriptor_t dyDesc, const void *dy, const miopenTensorDescriptor_t xDesc, const void *x, const miopenTensorDescriptor_t weightDesc, const void *weight, const miopenTensorDescriptor_t rstdDesc, const void *rstd, const miopenTensorDescriptor_t dxDesc, void *dx, const miopenTensorDescriptor_t dwDesc, void *dw)
 Execute a T5layernorm backward layer. More...
 

Detailed Description

Enumeration Type Documentation

◆ miopenNormMode_t

LayerNorm mode

Enumerator
MIOPEN_ELEMENTWISE_AFFINE 

initialized to ones for weights and zeros for biases

MIOPEN_WEIGHT_BIAS 

learnable weights and biases of the module of shape normalized_shape

MIOPEN_ELEMENTWISE_AFFINE_FUSED_ADD 

initialized to ones for weights and zeros for biases in addlayernorm

MIOPEN_WEIGHT_BIAS_FUSED_ADD 

learnable weights and biases of the module of shape normalized_shape in addlayernorm

MIOPEN_ELEMENTWISE_AFFINE_T5 

initialized to ones for weights and zeros for biases in t5layernorm

MIOPEN_WEIGHT_BIAS_T5 

learnable weights and biases of the module of shape normalized_shape in t5layernorm

Examples
/home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-miopen/checkouts/docs-6.2.2/include/miopen/miopen.h.

Function Documentation

◆ miopenAddLayerNormForward()

miopenStatus_t miopenAddLayerNormForward ( miopenHandle_t  handle,
miopenNormMode_t  mode,
const miopenTensorDescriptor_t  xDesc,
const void *  x,
const miopenTensorDescriptor_t  x2Desc,
const void *  x2,
const miopenTensorDescriptor_t  weightDesc,
const void *  weight,
const miopenTensorDescriptor_t  biasDesc,
const void *  bias,
const float  epsilon,
const int32_t  normalized_dim,
const miopenTensorDescriptor_t  yDesc,
void *  y,
const miopenTensorDescriptor_t  meanDesc,
void *  mean,
const miopenTensorDescriptor_t  rstdDesc,
void *  rstd 
)

Execute a add and layernorm forward layer.

Parameters
handleMIOpen handle (input)
modeLayerNorm mode (input)
xDescTensor descriptor for data input tensor x (input)
xData tensor x (input)
x2DescTensor descriptor for data input tensor x2 (input)
x2Data tensor x2 (input)
weightDescTensor descriptor for data input tensor weight (input)
weightData tensor weight (input)
biasDescTensor descriptor for data input tensor bias (input)
biasData tensor bias (input)
epsilonValue to stablize inverse variance calculation (input)
normalized_dimNomalized dimensions in the input array (input)
yDescTensor descriptor for output data tensor y (input)
yData tensor y (output)
meanDescTensor descriptor for output data tensor mean (input)
meanData tensor mean (output)
rstdDescTensor descriptor for output data tensor rstd (input)
rstdData tensor rstd (output)
Returns
miopenStatus_t
Examples
/home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-miopen/checkouts/docs-6.2.2/include/miopen/miopen.h.

◆ miopenGetT5LayerNormBackwardWorkspaceSize()

miopenStatus_t miopenGetT5LayerNormBackwardWorkspaceSize ( miopenHandle_t  handle,
miopenNormMode_t  mode,
const miopenTensorDescriptor_t  dyDesc,
const miopenTensorDescriptor_t  xDesc,
const miopenTensorDescriptor_t  weightDesc,
const miopenTensorDescriptor_t  rstdDesc,
const miopenTensorDescriptor_t  dxDesc,
const miopenTensorDescriptor_t  dwDesc,
size_t *  sizeInBytes 
)

Helper function to query the minimum workspace size required by the T5layernorm backward call.

Parameters
handleMIOpen Handle (input)
modeLayerNorm mode (input)
dyDescTensor descriptor for data input tensor dy (input)
xDescTensor descriptor for data input tensor x (input)
weightDescTensor descriptor for data input tensor weight (input)
rstdDescTensor descriptor for data input tensor rstd (input)
dxDescTensor descriptor for output data tensor dx (input)
dwDescTensor descriptor for output data tensor dw (input)
sizeInBytesPointer to data to return the minimum workspace size
Returns
miopenStatus_t
Examples
/home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-miopen/checkouts/docs-6.2.2/include/miopen/miopen.h.

◆ miopenLayerNormForward()

miopenStatus_t miopenLayerNormForward ( miopenHandle_t  handle,
miopenNormMode_t  mode,
const miopenTensorDescriptor_t  xDesc,
const void *  x,
const miopenTensorDescriptor_t  weightDesc,
const void *  weight,
const miopenTensorDescriptor_t  biasDesc,
const void *  bias,
const float  epsilon,
const int32_t  normalized_dim,
const miopenTensorDescriptor_t  yDesc,
void *  y,
const miopenTensorDescriptor_t  meanDesc,
void *  mean,
const miopenTensorDescriptor_t  rstdDesc,
void *  rstd 
)

Execute a layernorm forward layer.

Parameters
handleMIOpen handle (input)
modeLayerNorm mode (input)
xDescTensor descriptor for data input tensor x (input)
xData tensor x (input)
weightDescTensor descriptor for data input tensor weight (input)
weightData tensor weight (input)
biasDescTensor descriptor for data input tensor bias (input)
biasData tensor bias (input)
epsilonValue to stablize inverse variance calculation (input)
normalized_dimNomalized dimensions in the input array (input)
yDescTensor descriptor for output data tensor y (input)
yData tensor y (output)
meanDescTensor descriptor for output data tensor mean (input)
meanData tensor mean (output)
rstdDescTensor descriptor for output data tensor rstd (input)
rstdData tensor rstd (output)
Returns
miopenStatus_t
Examples
/home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-miopen/checkouts/docs-6.2.2/include/miopen/miopen.h.

◆ miopenT5LayerNormBackward()

miopenStatus_t miopenT5LayerNormBackward ( miopenHandle_t  handle,
miopenNormMode_t  mode,
void *  workspace,
size_t  workspaceSizeInBytes,
const miopenTensorDescriptor_t  dyDesc,
const void *  dy,
const miopenTensorDescriptor_t  xDesc,
const void *  x,
const miopenTensorDescriptor_t  weightDesc,
const void *  weight,
const miopenTensorDescriptor_t  rstdDesc,
const void *  rstd,
const miopenTensorDescriptor_t  dxDesc,
void *  dx,
const miopenTensorDescriptor_t  dwDesc,
void *  dw 
)

Execute a T5layernorm backward layer.

Parameters
handleMIOpen handle (input)
modeLayerNorm mode (input)
workspaceAddress of the allocated workspace data (input)
workspaceSizeInBytesSize in bytes of the allocated workspace data (input)
dyDescTensor descriptor for data input tensor dy (input)
dyData tensor dy (input)
xDescTensor descriptor for output data tensor x (input)
xData tensor x (input)
weightDescTensor descriptor for data input tensor weight (input)
weightData tensor weight (input)
rstdDescTensor descriptor for output data tensor rstd (input)
rstdData tensor rstd (output)
dxDescTensor descriptor for output data tensor dx (input)
dxData tensor dx (output)
dwDescTensor descriptor for output data tensor dw (input)
dwData tensor dw (output)
Returns
miopenStatus_t
Examples
/home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-miopen/checkouts/docs-6.2.2/include/miopen/miopen.h.

◆ miopenT5LayerNormForward()

miopenStatus_t miopenT5LayerNormForward ( miopenHandle_t  handle,
miopenNormMode_t  mode,
const miopenTensorDescriptor_t  xDesc,
const void *  x,
const miopenTensorDescriptor_t  weightDesc,
const void *  weight,
const float  epsilon,
const miopenTensorDescriptor_t  yDesc,
void *  y,
const miopenTensorDescriptor_t  rstdDesc,
void *  rstd 
)

Execute a T5layernorm forward layer.

Parameters
handleMIOpen handle (input)
modeLayerNorm mode (input)
xDescTensor descriptor for data input tensor x (input)
xData tensor x (input)
weightDescTensor descriptor for data input tensor weight (input)
weightData tensor weight (input)
epsilonValue to stablize inverse variance calculation (input)
yDescTensor descriptor for output data tensor y (input)
yData tensor y (output)
rstdDescTensor descriptor for output data tensor rstd (input)
rstdData tensor rstd (output)
Returns
miopenStatus_t
Examples
/home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-miopen/checkouts/docs-6.2.2/include/miopen/miopen.h.