The paper puts forward the dynamic ReLU, It can dynamically adjust the corresponding segment activation function according to the input , And ReLU And its variety contrast , Only a small amount of extra computation can lead to a significant performance improvement , It can be embedded into the current mainstream model seamlessly

source ： Xiaofei's algorithm Engineering Notes official account

** The paper : Dynamic ReLU**

**Address of thesis ：https://arxiv.org/abs/2003.10027****Paper code ：https://github.com/Islanna/DynamicReLU**

## Introduction

ReLU It's a very important milestone in deep learning , Simple but powerful , It can greatly improve the performance of neural networks . There are a lot of ReLU Improved version , such as Leaky ReLU and PReLU, And the final parameters of the improved version and the original version are fixed . So the paper naturally thought of , If you can adjust according to the input characteristics ReLU Parameters of might be better .

Based on the above ideas , The paper puts forward the dynamic ReLU(DY-ReLU). Pictured 2 Shown ,DY-ReLU It's a piecewise function $f_{\\theta{(x)}}(x)$, Parameters are defined by the hyperfunction $\\theta{(x)}$ According to input $x$ obtain . Hyperfunctions $\\theta(x)$ The context of each dimension of the integrated input comes from the adaptation activation function $f_{\\theta{(x)}}(x)$, Can bring a small amount of extra computation , Significantly improve the expression of the network . in addition , This paper provides three forms of DY-ReLU, There are different sharing mechanisms in spatial location and dimension . Different forms of DY-ReLU For different tasks , The paper is also verified by experiments ,DY-ReLU In the key point recognition and image classification have a good improvement .

## Definition and Implementation of Dynamic ReLU

#### Definition

Define the original ReLU by $y=max\\{x, 0\\}$,$x$ Is the input vector , For the input $c$ Whitman's sign $x_c$, The activation value is calculated as $y_c=max\\{x_c, 0\\}$.ReLU It can be expressed as piecewise linear function $y_c=max_k\\{a^k_c x_c+b^k_c\\}$, Based on this piecewise function, this paper extends dynamic ReLU, Based on all the inputs $x=\\{x_c\\}$ The adaptive $a^k_c$,$b^k_c$：

factor $(a^k_c, b^k_c)$ It's a hyperfunction $\\theta(x)$ Output ：

$K$ Is the number of functions ,$C$ Is the number of dimensions , Activation parameters $(a^k_c, b^k_c)$ Not only with $x_c$ relevant , Also with the $x_{j\\ne c}$ relevant .

#### Implementation of hyper function $\\theta(x)$

This paper adopts the method similar to SE Module lightweight network for the implementation of hyperfunctions , For size $C\\times H\\times W$ The input of $x$, First, use global average pooling for compression , Then use two full connection layers ( The middle contains ReLU) To deal with , Finally, a normalization layer is used to constrain the results to -1 and 1 Between , The normalization layer uses $2\\sigma(x) - 1$,$\\sigma$ by Sigmoid function . The subnet has a common output $2KC$ Elements , They correspond to each other $a^{1:K}_{1:C}$ and $b^{1:K}_{1:C}$ Residual of , The final output is the sum of the initial value and the residual error ：

$\\alpha^k$ and $\\beta^k$ by $a^k_c$ and $b^k_c$ The initial value of the ,$\\lambda_a$ and $\\lambda_b$ Is the scalar used to control the size of the residual . about $K=2$ The situation of , The default parameter is $\\alpha^1=1$,$\\alpha^2=\\beta^1=\\beta^2=0$, It's the original ReLU, Scalar defaults to $\\lambda_a=1.0$,$\\lambda_b=0.5$.

#### Relation to Prior Work

DY-ReLU There's a lot of possibility , surface 1 It shows DY-ReLU With the original ReLU And the relationship between its variants . After learning the specific parameters ,DY-ReLU It is equivalent to ReLU、LeakyReLU as well as PReLU. And when $K=1$, bias $b^1_c=0$ when , Is equivalent to SE modular . in addition DY-ReLU It can also be a dynamic and efficient Maxout operator , Is equivalent to Maxout Of $K$ Convolutions are converted to $K$ A dynamic linear change , And then output the maximum again .

## Variations of Dynamic ReLU

This paper provides three forms of DY-ReLU, There are different sharing mechanisms in spatial location and dimension ：

#### DY-ReLU-A

Spatial location and dimensions are shared (spatial and channel-shared), The calculation is shown in the figure 2a Shown , Just output $2K$ Parameters , The calculation is the simplest , Expression is also the weakest .

#### DY-ReLU-B

Only spatial location sharing (spatial-shared and channel-wise), The calculation is shown in the figure 2b Shown , Output $2KC$ Parameters .

#### DY-ReLU-C

Spatial location and dimensions are not shared (spatial and channel-wise), Each element of each dimension has a corresponding activation function $max_k\\{a^k_{c,h,w} x_{c, h, w} + b^k_{c,h,w} \\}$. Although it's very expressive , But the parameters that need to be output ($2KCHW$) That's too much , Like the previous one, using full connection layer output directly will bring too much extra computation . Therefore, the paper has been improved , The calculation is shown in the figure 2c Shown , Decompose the spatial location to another attention Branch , Finally, the dimension parameter $[a^{1:K}_{1:C}, b^{1:K}_{1:C}]$ Multiply by the space position attention$[\\pi_{1:HW}]$.attention The calculation of is simple to use $1\\times 1$ Convolution and normalization methods , Normalization uses constrained softmax function ：

$\\gamma$ Is used to attention Average , The paper is set as $\\frac{HW}{3}$,$\\tau$ Is the temperature , Set a larger value at the beginning of training (10) Used to prevent attention Too sparse .

#### Experimental Results

Image classification and contrast experiment .

Key point recognition contrast experiment .

And ReLU stay ImageNet Compare in many ways .

Compared with other activation functions .

visualization DY-ReLU In different block I / O and slope change , We can see its dynamic nature .

## Conclustion

The paper puts forward the dynamic ReLU, It can dynamically adjust the corresponding segment activation function according to the input , And ReLU And its variety contrast , Only a small amount of extra computation can bring about huge performance improvement , It can be embedded into the current mainstream model seamlessly . There is a reference to APReLU, It's also dynamic ReLU, The subnet structure is very similar , but DY-ReLU because $max_{1\\le k \\le K}$ The existence of , The possibility and the effect are better than APReLU Bigger .

If this article helps you , Please give me a compliment or watch it ～

More on this WeChat official account 【 Xiaofei's algorithm Engineering Notes 】