CNN的反向传播DNN中的反向传播卷积神经网络中的反向传播

月见樽

发布于 2018-04-27 12:13:30

1.1K0

发布于 2018-04-27 12:13:30

文章被收录于专栏：文武兼修ing——机器学习与IC设计文武兼修ing——机器学习与IC设计

DNN中的反向传播

反向传播算法是神经网络的训练的基本算法组成之一，在训练神经网络时，训练分为两个步骤：计算梯度和更新权值。其中反向传播负责的是梯度的计算，而训练算法的区分主要在更新权值的方式上。对于DNN，基本的反向传播思路为：

$$\cfrac{dz}{dw_{i}} = \cfrac{dz}{da_{i+1}} \times \cfrac{da_{i+1}}{dw_{i}}$$

其中，$\cfrac{dz}{dw_{i}}$为输出（多为代价函数输出）对第i层的权值的梯度，$\cfrac{da_{i+1}}{dw_{i}}$为本层输出对权值的梯度。于是梯度的计算被分为反向传播链条上的几个部分，将复杂的求导分割为层内运算的求导，上一层的梯度可以由本层的梯度递归的求出。

卷积神经网络中的反向传播

卷积神经网络相比于多层感知机，增加了两种新的层次——卷积层与池化层。由于反向传播链的存在，要求出这两种层结构的梯度，仅需要解决输出对权值的梯度即可。

池化层的梯度

池化层用于削减数据量，在这一层上前向传播的数据会有损失，则在反向传播时，传播来的梯度也会有所损失。一般来说，池化层没有参数，于是仅需要计算梯度反向传播的结果。

理论分析

池化层的反向传播的方法是upsample，先将矩阵还原成原大小，之后：

对于最大值池化，将梯度放置于每个池化区域取得最大值的位置，其他位置为0
对于平均值池化，则把的所有子矩阵的各个池化局域的值取平均后放在还原后的子矩阵位置

例如对于矩阵：

$$ \left( \begin{array}{} 1& 2 \ 3& 4 \end{array} \right)$$

假设经过2*2的池化，还原为原来大小：

$$\left( \begin{array}{ccc} 0&0&0&0 \ 0&1& 2&0 \ 0&3&4&0 \ 0&0&0&0 \end{array} \right)$$

若是最大值池化，假设每个窗口的最大值位置都是左上，则传播结果为：

$$\left( \begin{array}{ccc} 1&0&2&0 \ 0&0& 0&0 \ 3&0&4&0 \ 0&0&0&0 \end{array} \right)$$

若是经过平均值池化，则传播结果为：

$$\left( \begin{array}{ccc} 0.25&0.25&0.5&0.5 \ 0.25&0.25& 0.5&0.5 \ 0.75&0.75&1&1 \ 0.75&0.75&1&1 \end{array} \right)$$

代码验证

最大值池化

test_model = pt.nn.MaxPool2d(2)
test = pt.autograd.Variable(pt.arange(0, 16).view(1, 1, 4, 4), requires_grad=True)
print(test)
result = test_model(test)
print(result)
result.backward(result.data)
print(test.grad)

Variable containing:
(0 ,0 ,.,.) =
   0   1   2   3
   4   5   6   7
   8   9  10  11
  12  13  14  15
[torch.FloatTensor of size 1x1x4x4]

Variable containing:
(0 ,0 ,.,.) =
   5   7
  13  15
[torch.FloatTensor of size 1x1x2x2]

Variable containing:
(0 ,0 ,.,.) =
   0   0   0   0
   0   5   0   7
   0   0   0   0
   0  13   0  15
[torch.FloatTensor of size 1x1x4x4]

平均值池化

test_model = pt.nn.AvgPool2d(2)
test = pt.autograd.Variable(pt.arange(0, 16).view(1, 1, 4, 4), requires_grad=True)
print(test)
result = test_model(test)
print(result)
result.backward(result.data)
print(test.grad)

Variable containing:
(0 ,0 ,.,.) =
   0   1   2   3
   4   5   6   7
   8   9  10  11
  12  13  14  15
[torch.FloatTensor of size 1x1x4x4]

Variable containing:
(0 ,0 ,.,.) =
   2.5000   4.5000
  10.5000  12.5000
[torch.FloatTensor of size 1x1x2x2]

Variable containing:
(0 ,0 ,.,.) =
  0.6250  0.6250  1.1250  1.1250
  0.6250  0.6250  1.1250  1.1250
  2.6250  2.6250  3.1250  3.1250
  2.6250  2.6250  3.1250  3.1250
[torch.FloatTensor of size 1x1x4x4]

卷积层梯度

卷积层具有权值，因此梯度计算包括反向传播的梯度和权值梯度

反向传播梯度

理论分析

对于卷积网络，前向传播公式为：

$$a^l= \sigma(z^l) = \sigma(a{l-1}*Wl +b^l) $$

其中$*$为卷积运算（不为乘法运算），DNN的反向传播公式为：

$$\delta^{l} = \cfrac{\partial J(W,b)}{\partial z^l} = \cfrac{\partial J(W,b)}{\partial z^{l+1}}\cfrac{\partial z^{l+1}}{\partial z^{l}} = \delta^{l+1}\cfrac{\partial z^{l+1}}{\partial z^{l}}$$

其中$\delta^{l}$为第l层的梯度，$\cfrac{\partial z^{l+1}}{\partial z^{l}}$为卷积层的输出对输入的梯度，则反向传播的梯度为：

$$\delta^{l-1} = \delta^{l}\cfrac{\partial z^{l}}{\partial z^{l-1}} = \delta{l}*rot180(W{l}) \odot \sigma{'}(z{l-1})$$

其中$rot180(W^{l})$为卷积核旋转180度的函数，*为卷积。

举一个例子来说，对于以下卷积等式：

$$\left( \begin{array}{ccc} a_{11}&a_{12}&a_{13} \ a_{21}&a_{22}&a_{23}\ a_{31}&a_{32}&a_{33} \end{array} \right) * \left( \begin{array}{ccc} w_{11}&w_{12}\ w_{21}&w_{22} \end{array} \right) = \left( \begin{array}{ccc} z_{11}&z_{12}\ z_{21}&z_{22} \end{array} \right)$$

对于$a_{11}$，有$z_{11} = a_{11}w_{11} + a_{12}w_{12} + a_{21}w_{21} + a_{22}w_{22}$，仅$z_{11}$与其有关，则有$\nabla a_{11} = \delta_{11}w_{11}$。

对于$a_{22}$，所有z项都和该数有关，有$\nabla a_{22} = \delta_{11}w_{22} + \delta_{12}w_{21} + \delta_{21}w_{12} + \delta_{22}w_{11}$

依次类推，可得：

$\left( \begin{array}{ccc} 0&0&0&0 \ 0&\delta_{11}& \delta_{12}&0 \ 0&\delta_{21}&\delta_{22}&0 \ 0&0&0&0 \end{array} \right) * \left( \begin{array}{ccc} w_{22}&w_{21}\ w_{12}&w_{11} \end{array} \right) = \left( \begin{array}{ccc} \nabla a_{11}&\nabla a_{12}&\nabla a_{13} \ \nabla a_{21}&\nabla a_{22}&\nabla a_{23}\ \nabla a_{31}&\nabla a_{32}&\nabla a_{33} \end{array} \right)$

代码验证

data = pt.autograd.Variable(pt.arange(0, 9).view(1, 1, 3, 3), requires_grad=True)
weight = pt.autograd.Variable(pt.arange(0,4).view(1,1,2,2),requires_grad=True)
bias = pt.autograd.Variable(pt.zeros(1),requires_grad=True)
result = pt.nn.functional.conv2d(data,weight,bias)
print(result)
result.backward(result.data)
print(data.grad)

Variable containing:
(0 ,0 ,.,.) =
  19  25
  37  43
[torch.FloatTensor of size 1x1x2x2]

Variable containing:
(0 ,0 ,.,.) =
    0   19   25
   38  144  118
   74  197  129
[torch.FloatTensor of size 1x1x3x3]

该代码中，前向传播为：

$$\left( \begin{array}{ccc} 0&1&2 \ 3&4&5\ 6&7&8 \end{array} \right) * \left( \begin{array}{ccc} 0&1\ 2&3 \end{array} \right) = \left( \begin{array}{ccc} 19&25\ 37&43 \end{array} \right)$$

反向传播为：

$$\left( \begin{array}{ccc} 0&0&0&0 \ 0&19&25&0 \ 0&37&43&0 \ 0&0&0&0 \end{array} \right) * \left( \begin{array}{ccc} 3&2\ 1&0 \end{array} \right) = \left( \begin{array}{ccc} 0&19&25 \ 38&144&118\ 74&197&129 \end{array} \right)$$

权值梯度

理论推导

类似于上一节的公式，对权值的梯度为：

$$\cfrac{\partial J(W,b)}{\partial W^{l}} = \cfrac{\partial J(W,b)}{\partial z^{l}}\cfrac{\partial z^{l}}{\partial W^{l}} =rot180(\deltal*a{l-1})$$

同样对于前向传播：

对于$z$，有以下：

$z_{11} = a_{11}w_{11} + a_{12}w_{12} + a_{21}w_{21} + a_{22}w_{22}$
$z_{12} = a_{12}w_{11} + a_{13}w_{12} + a_{22}w_{21} + a_{23}w_{22}$
$z_{21} = a_{21}w_{11} + a_{22}w_{12} + a_{31}w_{21} + a_{32}w_{22}$
$z_{22} = a_{22}w_{11} + a_{23}w_{12} + a_{32}w_{21} + a_{33}w_{22}$

有梯度：

$\nabla w_{11} = \delta_{11}a_{11} + \delta_{12}a_{12} + \delta_{21}a_{21} + \delta_{22}a_{22}$
$\nabla w_{12} = \delta_{11}a_{12} + \delta_{12}a_{13} + \delta_{21}a_{22} + \delta_{22}a_{12}$
$\nabla w_{21} = \delta_{11}a_{21} + \delta_{12}a_{22} + \delta_{21}a_{31} + \delta_{22}a_{32}$
$\nabla w_{22} = \delta_{11}a_{22} + \delta_{12}a_{23} + \delta_{32}a_{21} + \delta_{33}a_{22}$

反向传播为：

$\left( \begin{array}{ccc} 0&0&0&0 \ 0&\delta_{11}& \delta_{12}&0 \ 0&\delta_{21}&\delta_{22}&0 \ 0&0&0&0 \end{array} \right) * \left( \begin{array}{ccc} a_{11}&a_{12}&a_{13}\ a_{21}&a_{22}&a_{23}\ a_{31}&a_{32}&a_{33} \end{array} \right) = \left( \begin{array}{ccc} \nabla w_{22}&\nabla w_{21}\ \nabla w_{12}&\nabla w_{11} \end{array} \right)$

代码验证

data = pt.autograd.Variable(pt.arange(0, 9).view(1, 1, 3, 3), requires_grad=True)
weight = pt.autograd.Variable(pt.arange(0,4).view(1,1,2,2),requires_grad=True)
bias = pt.autograd.Variable(pt.zeros(1),requires_grad=True)
result = pt.nn.functional.conv2d(data,weight,bias)
print(result)
result.backward(result.data)
print(weight.grad)

Variable containing:
(0 ,0 ,.,.) =
  19  25
  37  43
[torch.FloatTensor of size 1x1x2x2]

Variable containing:
(0 ,0 ,.,.) =
  308  432
  680  804
[torch.FloatTensor of size 1x1x2x2]

反向传播为：

$$\left( \begin{array}{ccc} 0&0&0&0 \ 0&19&25&0 \ 0&37&43&0 \ 0&0&0&0 \end{array} \right) * \left( \begin{array}{ccc} 0&1&2 \ 3&4&5\ 6&7&8 \end{array} \right) = \left( \begin{array}{ccc} 804&680\ 432&308 \end{array} \right)$$

偏置梯度

偏置梯度较为简单，为上一层梯度和：

$$\cfrac{\partial J(W,b)}{\partial b^{l}} = \cfrac{\partial J(W,b)}{\partial z^{l}} \cdot \cfrac{dz{l}}{dbl}= \sum\limits_{u,v}(\delta^l)_{u,v}$$

data = pt.autograd.Variable(pt.arange(0, 9).view(1, 1, 3, 3), requires_grad=True)
weight = pt.autograd.Variable(pt.arange(0,4).view(1,1,2,2),requires_grad=True)
bias = pt.autograd.Variable(pt.zeros(1),requires_grad=True)
result = pt.nn.functional.conv2d(data,weight,bias)
print(result)
result.backward(result.data)
print(bias.grad)

Variable containing:
(0 ,0 ,.,.) =
  19  25
  37  43
[torch.FloatTensor of size 1x1x2x2]

Variable containing:
 124
[torch.FloatTensor of size 1]

有$124 = 19 + 25 + 37 + 43$

参考

刘建平Pinard

本文参与腾讯云自媒体分享计划，分享自作者个人站点/博客。

原始发表：2018.02.21 ，如有侵权请联系 cloudcommunity@tencent.com 删除

卷积神经网络

神经网络

编程算法

本文分享自作者个人站点/博客前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与腾讯云自媒体分享计划，欢迎热爱写作的你一起参与！

卷积神经网络

神经网络

编程算法

登录后参与评论

0 条评论

热度