AlexNet论文总结

Introduction

Preconditions

  • ImageNet Objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. ImageNet consists of over 15 million labeled high-resolution images in over 22,000 categories.
  • CNNs To learn about thousands of objects from millions of images, we need a model with a large learning capacity. Convolutional neural networks (CNNs) constitute one such class of models. Their capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies).

Contributions

The specific contributions of this paper are as follows:

  • We trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions and achieved by far the best results ever reported on these datasets.
  • We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly.
  • Our network contains a number of new and unusual features which improve its performance and reduce its training time.
  • The size of our network made overfitting a significant problem, even with 1.2 million labeled training examples, so we used several effective techniques for preventing overfitting.
  • Our final network contains five convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutional layer (each of which contains no more than 1% of the model’s parameters) resulted in inferior performance.

Architecture

The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.

ReLU

Motivation: We are not the first to consider alternatives to traditional neuron models in CNNs.

The standard way to model a neuron’s output f as a function of its input x is with f(x)=tanh(x)f(x)=tanh(x) or f(x)=(1+e−x)−1f(x)=(1+e_{−x})^{−1}. In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity f(x)=max(0,x)f(x)=max(0,x). Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units.

Note: Saturation of Sigmoid f(x) refers to the time when the output is almost zero or 1, where the gradient in the region is almost zero, causing the local gradient to disappear.

A four-layer convolutional neural network with ReLUs (solid line) reaches a 25% training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line).

Training on Multiple GPUs

A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs. The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. Result: This scheme reduces our top-1 and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net.

Local Response Normalization

Motivation: This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels.

We still find that the following local normalization scheme aids generalization. Denoting by aix,ya_{x,y}^i the activity of a neuron computed by applying kernel ii at position (x,y)(x, y) and then applying the ReLU nonlinearity, the response-normalized activity bix,yb^i_{x,y} is given by the expression

bix,y=aix,y/(k+α∑j=max(0,i−n/2)min(N−1,i+n/2)(aix,y)2)β

b^i_{x,y} = a_{x,y}^i / ( k + \alpha \sum _{j = max(0, i-n / 2)} ^{min(N-1, i+n / 2)} (a_{x,y}^i)^2 )^\beta

where the sum runs over n “adjacent” kernel maps at the same spatial position, and N is the total number of kernels in the layer. The constants k, n, α, and β are hyper-parameters whose values are determined using a validation set; we used k = 2, n = 5, α = 0.0001, and β = 0.75.

Result: Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%, respectively.

Overlapping Pooling

A pooling layer can be thought of as consisting of a grid of pooling units spaced ss pixels apart, each summarizing a neighborhood of size z×zz × z centered at the location of the pooling unit. If we set s=zs = z, we obtain traditional local pooling as commonly employed in CNNs. If we set s<zs < z, we obtain overlapping pooling. This is what we use throughout our network, with s=2s = 2 and z=3z = 3.

Result: This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.

Overall Architecture

Reduce Overfitting

Data Augmentation

The first form of data augmentation consists of generating image translations and horizontal reflections.

The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. To each training image, we add multiples of the found principal components, with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel Ixy=[IRxy,IGxy,IBxy]TI_{xy} = [I^R_{xy} , I^G_{xy} , I^B_{xy} ]^T we add the following quantity:

[p1,p2,p3][α1λ1,α2λ2,α3λ3]T

[p_1, p_2, p_3][\alpha_1\lambda_1, \alpha_2\lambda_2, \alpha_3\lambda_3]^T

where p_ip\_i and λi\lambda_i are iith eigenvector and eigenvalue of the 3 × 3 covariance matrix of RGB pixel values, respectively, and αi\alpha_i is the aforementioned random variable.

Result: This scheme reduces the top-1 error rate by over 1%.

Dropout

Motivation: Combining the predictions of many different models is a very successful way to reduce test errors, but it appears to be too expensive for big neural networks that already take several days to train.

There is, however, a very efficient version of model combination that only costs about a factor of two during training. The recently-introduced technique, called “dropout”, consists of setting to zero the output of each hidden neuron with probability 0.5. We use dropout in the first two fully-connected layers. Without dropout, our network exhibits substantial overfitting. Dropout roughly doubles the number of iterations required to converge.

Experimental Results

References

https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks

Supplement material

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏红色石头的机器学习之路

Python机器学习(1)-- 自己设计一个感知机(Perceptron)分类算法

Implementing a perceptron learning algorithm in Python Define a Class import num...

4590
来自专栏算法与数据结构

动态规划--Kin

2804
来自专栏数据结构与算法

计算几何笔记

若向量$(x, y)$旋转角度为$a$,则旋转后的向量为$(xcosa - ysina, y cosa + xsina)$

1022
来自专栏贾志刚-OpenCV学堂

Mat对象与它各种用法

学习OpenCV大家都会遇到一个对象叫做Mat,此对象非常神奇,支持各种操作。很多初学者因此被搞得头晕脑胀,它各种用法太多台杂,搞得初学者应接不暇,感觉有心无力...

3508
来自专栏玉树芝兰

How to Accelerate Your Python Deep Learning with Cloud GPU?

This afternoon, I trained a 3-layers neural network as a regression model to pre...

741
来自专栏数据结构与算法

Day5网络流

算法 无源汇上下界可行流 ?  先强制流过l的流量 从s到每个正权点连流量为l的流量  从每个负权点向t连-l的流量 如果容量为0,则不连边 有源汇上下界最大流...

2889
来自专栏和蔼的张星的图像处理专栏

opencv操作图像像素和通道

老师让我评价一下别人的一个跟踪效果,只有带跟踪框的视频,所以需要检测这个框,用了下投影,最早用matlab写的一个脚本,很简单,转到opencv里反而有些麻烦,...

1141
来自专栏聊聊技术

原 初学算法 - 求凸包的Garham's

43510
来自专栏木子昭的博客

两个玻璃球 测试极限高度

一道有趣的智力题目: 已知,玻璃球从某高楼落到地面会摔碎,楼的最大高度为100层,给你两个玻璃球,请你最快的测出,能使玻璃球摔碎的最低楼层... ? ...

3188
来自专栏LeetCode

LeetCode <dp>62&63.Unique Paths I&II

A robot is located at the top-left corner of a m x n grid (marked 'Start' in the...

1184

扫码关注云+社区