Densely Connected Convolutional Networks CVPR2017 best paper Code: https://github.com/liuzhuang13/DenseNet
本文受到 ResNet and Highway Networks 的启发: bypass signal from one layer to the next via identity connections,这里主要是多加了几个 identity connections,发现这么干效果很好。
首先看看 一个 5层 的Dense Block 是怎么Densely Connected
上面5层的模块有多少连接了? 5*(5+1)/2=15
整个网络结构如下图所示:
Dense connectivity
where [x_0 ,x _1 ,…,x_l−1 ] refers to the concatenation of the feature-maps produced in layers 0,…,l−1
Composite function H定义为 a composite function of three consecutive operations: batch normalization (BN) , followed by a rectified linear unit (ReLU) and a 3 × 3 convolution (Conv)
Pooling layers 可以改变特征图的尺寸,便于 concatenation
Growth rate:If each function H produces k feature-maps as output,We refer to the hyper-parameter k as the growth rate of the network
Bottleneck layers:尽管每个网络层只输出 k 个特征图,但是同时仍然有太多的输入个数,通常的做法是降维,在进行3×3卷积之前首先用一个 1×1卷积将输入个数降低到 4*k, 也就是在 H的定义中再加入一个 1×1卷积 Although each layer only produces k output feature maps, it typically has many more inputs. It has been noted in [36, 11] that a 1×1 convolution can be introduced as bottleneck layer before each 3×3 convolution to reduce the number of input feature-maps
为什么有太多的输入个数了? If each function H_l produces k feature-maps as output, it follows that the l th layer has k×(l−1)+k0 input feature-maps, where k 0 is the number of channels in the input image.
Compression :为了进一步提升模型的简洁性,我们在 transition layers里 降低特征图数量 To further improve model compactness, we can reduce the number of feature-maps at transition layers. If a dense block contains m feature-maps, we let the following transition layer generate bθmc output feature-maps, where 0 <θ ≤1 is referred to as the compression factor.
DenseNet and ResNet Top-1 (single model and single-crop) 对比:
参数规模还是比较小的
1) Middle: DenseNet-BC requires about 1/3 of the parameters as ResNet to achieve comparable accuracy 2)Right: Training and testing curves of the 1001-layer pre-activation ResNet [12] with more than 10M parameters and a 100-layer DenseNet with only 0.8M parameters
总的来说就是简单的多加几个 shortcut ,效果就好了,计算量少了!