Batchnorm

Batchnorm#

MIOpen: Batchnorm
Enumerations | Functions
Batchnorm

Enumerations

enum  miopenBatchNormMode_t {
  miopenBNPerActivation = 0 ,
  miopenBNSpatial = 1
}
 

Functions

miopenStatus_t miopenDeriveBNTensorDescriptor (miopenTensorDescriptor_t derivedBnDesc, const miopenTensorDescriptor_t xDesc, miopenBatchNormMode_t bn_mode)
 Derive tensor for gamma and beta from input tensor descriptor. More...
 
miopenStatus_t miopenBatchNormalizationForwardTraining (miopenHandle_t handle, miopenBatchNormMode_t bn_mode, void *alpha, void *beta, const miopenTensorDescriptor_t xDesc, const void *x, const miopenTensorDescriptor_t yDesc, void *y, const miopenTensorDescriptor_t bnScaleBiasMeanVarDesc, void *bnScale, void *bnBias, double expAvgFactor, void *resultRunningMean, void *resultRunningVariance, double epsilon, void *resultSaveMean, void *resultSaveInvVariance)
 Execute forward training layer for batch normalization. More...
 
miopenStatus_t miopenBatchNormalizationForwardInference (miopenHandle_t handle, miopenBatchNormMode_t bn_mode, void *alpha, void *beta, const miopenTensorDescriptor_t xDesc, const void *x, const miopenTensorDescriptor_t yDesc, void *y, const miopenTensorDescriptor_t bnScaleBiasMeanVarDesc, void *bnScale, void *bnBias, void *estimatedMean, void *estimatedVariance, double epsilon)
 Execute forward inference layer for batch normalization. More...
 
miopenStatus_t miopenBatchNormalizationBackward (miopenHandle_t handle, miopenBatchNormMode_t bn_mode, const void *alphaDataDiff, const void *betaDataDiff, const void *alphaParamDiff, const void *betaParamDiff, const miopenTensorDescriptor_t xDesc, const void *x, const miopenTensorDescriptor_t dyDesc, const void *dy, const miopenTensorDescriptor_t dxDesc, void *dx, const miopenTensorDescriptor_t bnScaleBiasDiffDesc, const void *bnScale, void *resultBnScaleDiff, void *resultBnBiasDiff, double epsilon, const void *savedMean, const void *savedInvVariance)
 Execute backwards propagation layer for batch normalization. More...
 

Detailed Description

Enumeration Type Documentation

◆ miopenBatchNormMode_t

enum miopenBatchNormMode_t

Batch Normalization layer mode

Enumerator
miopenBNPerActivation 

Element-wise normalization for fully connected layer

miopenBNSpatial 

Mini-batch spatial normalization for convolutional layers

Examples
/home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-miopen/checkouts/docs-6.1.2/include/miopen/miopen.h.

Function Documentation

◆ miopenBatchNormalizationBackward()

miopenStatus_t miopenBatchNormalizationBackward ( miopenHandle_t  handle,
miopenBatchNormMode_t  bn_mode,
const void *  alphaDataDiff,
const void *  betaDataDiff,
const void *  alphaParamDiff,
const void *  betaParamDiff,
const miopenTensorDescriptor_t  xDesc,
const void *  x,
const miopenTensorDescriptor_t  dyDesc,
const void *  dy,
const miopenTensorDescriptor_t  dxDesc,
void *  dx,
const miopenTensorDescriptor_t  bnScaleBiasDiffDesc,
const void *  bnScale,
void *  resultBnScaleDiff,
void *  resultBnBiasDiff,
double  epsilon,
const void *  savedMean,
const void *  savedInvVariance 
)

Execute backwards propagation layer for batch normalization.

Batch normalization pass for backwards propagation training pass. The method for backwards propagation batch normalization.

Takes in batch normalization mode bn_mode and input tensor data x, input activation tensor dy, output tensor dx, the learned tensors resultBNBiasDiff and resultBNScaleDiff with their descriptor.

If BOTH savedMean, and savedVariance are not null pointers then the method will use the saved mean and variance calculated by the forward training phase.

Parameters
handleMIOpen handle (input)
bn_modeBatch normalization mode (input)
alphaDataDiffFloating point scaling factor, allocated on the host (input)
betaDataDiffFloating point shift factor, allocated on the host (input)
alphaParamDiffFloating point scaling factor, allocated on the host (input)
betaParamDiffFloating point shift factor, allocated on the host (input)
xDescTensor descriptor for data input tensor x (input)
xData tensor x (input)
dyDescTensor descriptor for output data tensor y (input)
dyData tensor y (input)
dxDescTensor descriptor for output data tensor dx (input)
dxData delta tensor dx (output)
bnScaleBiasDiffDescTensor descriptor for BN scaling, shifting, saved variance and mean (input)
bnScaleBatch norm scaling, gamma, tensor (input)
resultBnScaleDiffTensor for dscale (output)
resultBnBiasDiffTensor for dbias (output)
epsilonValue to stabilize inverse variance calculation (input)
savedMeanSaved mini-batch mean for backwards pass (input)
savedInvVarianceSaved mini-bathc inverse variance for backwards pass (input)
Returns
miopenStatus_t
Examples
/home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-miopen/checkouts/docs-6.1.2/include/miopen/miopen.h.

◆ miopenBatchNormalizationForwardInference()

miopenStatus_t miopenBatchNormalizationForwardInference ( miopenHandle_t  handle,
miopenBatchNormMode_t  bn_mode,
void *  alpha,
void *  beta,
const miopenTensorDescriptor_t  xDesc,
const void *  x,
const miopenTensorDescriptor_t  yDesc,
void *  y,
const miopenTensorDescriptor_t  bnScaleBiasMeanVarDesc,
void *  bnScale,
void *  bnBias,
void *  estimatedMean,
void *  estimatedVariance,
double  epsilon 
)

Execute forward inference layer for batch normalization.

Batch normalization pass for forward inference pass. Takes in batch normalization mode bn_mode and input tensor x, output tensor y, bnBias and bnScale with their descriptor.

If either estimatedMean, or estimatedVariance are null pointers then the values for the mean and variance will be calculated from input data and this calculated mean and variance will be used to update input values. If variance is zero and epsilon is also zero, this function outputs NAN values. Input espilon value should always be non zero positive value.

Parameters
handleMIOpen handle (input)
bn_modeBatch normalization mode (input)
alphaFloating point scaling factor, allocated on the host (input)
betaFloating point shift factor, allocated on the host (input)
xDescTensor descriptor for data input tensor x (input)
xData tensor x (input)
yDescTensor descriptor for output data tensor y (input)
yData tensor y (output)
bnScaleBiasMeanVarDescTensor descriptor for BN scaling, shifting, saved variance and mean (input)
bnScaleBatch norm scaling, gamma, tensor (input)
bnBiasBatch norm bias, beta, tensor (input)
estimatedMeanRunning average saved during forward training (input)
estimatedVarianceRunning variance saved during forward training (input)
epsilonValue to stabilize inverse variance calculation (input)
Returns
miopenStatus_t
Examples
/home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-miopen/checkouts/docs-6.1.2/include/miopen/miopen.h.

◆ miopenBatchNormalizationForwardTraining()

miopenStatus_t miopenBatchNormalizationForwardTraining ( miopenHandle_t  handle,
miopenBatchNormMode_t  bn_mode,
void *  alpha,
void *  beta,
const miopenTensorDescriptor_t  xDesc,
const void *  x,
const miopenTensorDescriptor_t  yDesc,
void *  y,
const miopenTensorDescriptor_t  bnScaleBiasMeanVarDesc,
void *  bnScale,
void *  bnBias,
double  expAvgFactor,
void *  resultRunningMean,
void *  resultRunningVariance,
double  epsilon,
void *  resultSaveMean,
void *  resultSaveInvVariance 
)

Execute forward training layer for batch normalization.

Batch normalization pass for forward training pass. Takes in batch normalization mode bn_mode and input tensor x, output tensor y, bnBias and bnScale with their descriptor.

If either resultSaveMean, or resultSaveInvVariance are null pointers then the values for the mean and inverse variance will not be used.

Likewise, if either resultRunningMean, or resultRunningVariance are null pointers then the values for the running mean and variance will not be saved. Running averages and variances are scaled using an exponential averaging factor:

\[ \mu_{old} = \mu_{new}*factor + \mu_{old}*(1-factor) \]

where

\[ factor=1/(1+iteration) \]

Parameters
handleMIOpen handle (input)
bn_modeBatch normalization mode (input)
alphaFloating point scaling factor, allocated on the host (input)
betaFloating point shift factor, allocated on the host (input)
xDescTensor descriptor for data input tensor x (input)
xData tensor x (input)
yDescTensor descriptor for output data tensor y (input)
yData tensor y (output)
bnScaleBiasMeanVarDescTensor descriptor for BN scaling, shifting, saved variance and mean (input)
bnScaleBatch norm scaling, gamma, tensor (input)
bnBiasBatch norm bias, beta, tensor (input)
expAvgFactorExponential averaging factor (input)
resultRunningMeanRunning average saved for inference (output)
resultRunningVarianceRunning variance saved for inference (output)
epsilonValue to stablize inverse variance calculation (input)
resultSaveMeanSaved mini-batch mean for backwards pass (output)
resultSaveInvVarianceSaved mini-batch inverse variance for backwards pass (output)
Returns
miopenStatus_t
Examples
/home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-miopen/checkouts/docs-6.1.2/include/miopen/miopen.h.

◆ miopenDeriveBNTensorDescriptor()

miopenStatus_t miopenDeriveBNTensorDescriptor ( miopenTensorDescriptor_t  derivedBnDesc,
const miopenTensorDescriptor_t  xDesc,
miopenBatchNormMode_t  bn_mode 
)

Derive tensor for gamma and beta from input tensor descriptor.

This function takes the input tensor descriptor and outputs a derived tensor for the normalization scale (gamma) and shift (beta) tensors.

For an input tensor NCHW and spatial mode, the output derived tensor is 1C11, while for per-activation the derived tensor is 1CHW.

For an input tensor NCDHW and spatial mode, the output derived tensor is 1C111, while for per-activation the derived tensor is 1CDHW.

Parameters
derivedBnDescOutput derived tensor descriptor (output)
xDescInput tensor descriptor (input)
bn_modeBatch Normalization mode (input)
Returns
miopenStatus_t
Examples
/home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-miopen/checkouts/docs-6.1.2/include/miopen/miopen.h.