前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >Some Tips/Tricks in Machine Learning

Some Tips/Tricks in Machine Learning

作者头像
GavinZhou
发布2018-01-02 16:27:11
6300
发布2018-01-02 16:27:11
举报

这篇不算是基础系列,只是在网上看见别人写的,感觉很不错,翻译的别人的英文 原文链接: 原文

数据集扩充

对于深度的NN来说,数据集过小会很容易造成Overfitting,扩充数据集的方法通常有以下几种:

horizontally flipping random crops color jittering

预处理(normalization)

常见的有一下三种:

减去均值 zscore 白化(whittening)

效果逐步增强

减去均值

代码语言:javascript
复制
 >> x -= np..mean(X, axis = 0) # zero-center

zscore

就是减去均值,除以标准差

代码语言:javascript
复制
>> X -= np.mean(X, axis = 0) # zero-center
>> X /= np.std(X, axis = 0) # normalize

白化(whitening)

两种白化的操作:

  1. PCA白化
  2. ZCA白化

首先,需要求出协方差矩阵和方向向量(u1,u2),将原始的数据点使用u1u2来表示,得到新的坐标(投影)

PCA whitening

pca白化是指对上面的pca的新坐标X’,每一维的特征做一个标准差归一化处理

PCA whitening form 1
PCA whitening form 1

或者是:

PCA whitening form 2
PCA whitening form 2

ZCA whitening

ZCA白化是在PCA白化的基础上,又进行处理的一个操作。具体的实现是把上面PCA白化的结果,又变换到原来坐标系下的坐标:

zca whitening
zca whitening

权重的初始化

初始化为全0,是错误的做法

一般采用两种方法:

Small Random Numbers

初始化为随机的接近0的小数

代码语言:javascript
复制
>> 0.01 * N(0,1)    #N(0,1)表示均值为0的标准高斯分布

Calibrating the Variances

方差(variance)为2/n

代码语言:javascript
复制
>> w = np.random.randn(n) * sqrt(2.0/n) # current recommendation

Training

Filter size

训练图片的大小是2的倍数的时候,比如32,64,224,512等

it is important to employ a small filter (e.g., 3*3) and small strides (e.g., 1) with zeros-padding, which not only reduces the number of parameters, but improves the accuracy rates of the whole deep network. Meanwhile, a special case mentioned above, i.e., 3*3 filters with stride 1, could preserve the spatial size of images/feature maps. For the pooling layers, the common used pooling size is of 2*2.

Learning rate

推荐使用mini-batch的方式进行训练,初始的lr典型为0.1 对于validation set来说,没什么作用的话,可以将lr/2或者lr/5来试试

Fine-tune on pre-trained models

推荐使用VGG的网络

table 1
table 1

Activation Functions

  • sigmoid很少用,不推荐(kill gradients, not zero-centered)
  • tanhsigmoid要好(is zero-centered)
  • ReLU系列: ReLU,PReLU,Leaky ReLU,RReLU,中推荐使用PReLU and RReLU

Regularizations

L1正则

l1正则
l1正则

L2正则

l2正则
l2正则

l2正则一般情况下优于l1正则

Dropout

0.5的概率值是典型的做法

数据倾斜

常用的解决方法:

sampling techniques

1. duplicating instances(maybe special crops processing) from the minority classes until a balanced distribution is reached (oversampling) 2. removing instances from over-represented classes (undersampling) 3. it is suggested that a combination is the best solution for extremely imbalanced distributions 4. generating new data in minority classes based on the current data

cost sensitive techniques

a higher penalty can be given to the network when it misclassifies the minority classes during training

One-class learning

  1. only provides training data from a single class(每一类去训练,不停的fine-tuning)
  2. firstly fine-tune on the classes which have a large number of training samples (images/crops), and secondly, continue to fine-tune but on the classes with limited number samples
本文参与 腾讯云自媒体分享计划,分享自作者个人站点/博客。
原始发表:2016-09-19 ,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体分享计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
目录
  • 数据集扩充
  • 预处理(normalization)
    • 减去均值
      • zscore
        • 白化(whitening)
          • PCA whitening
          • ZCA whitening
        • 权重的初始化
        • Training
          • Filter size
            • Learning rate
              • Fine-tune on pre-trained models
                • Activation Functions
                  • Regularizations
                    • L1正则
                    • L2正则
                  • Dropout
                    • 数据倾斜
                      • sampling techniques
                      • cost sensitive techniques
                      • One-class learning
                  领券
                  问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档