# batch and minibatch

In practice, we can compute these expectations by randomly sampling a small number of examples from the dataset, then taking the average over only those examples.

Optimization algorithms that use the entire training set are called batch or deterministic gradient methods, because they process all of the training examples simultaneously in a large batch.

Optimization algorithms that use only a single example at a time are sometimes called stochastic or sometimes online methods.

## minibatch SGD

SGD和batch GD是两个极端，一个每次只使用一个训练数据来计算梯度，一个是使用所有的训练数据；batch GD计算成本太高，SGD太具有随机性，所以需要综合下。

using more than one but less than all of the training examples. These were traditionally called minibatch or minibatch stochastic methods and it is now common to simply call them stochastic methods.

It is also crucial that the minibatches be selected randomly. Computing an unbiased estimate of the expected gradient from a set of samples requires that those samples be independent. We also wish for two subsequent gradient estimates to be independent from each other, so two subsequent minibatches of examples should also be independent from each other.

# SGD

SGD和它的变种是最常用的一阶优化算法，具体描述：

## momentum

SGD的问题就是它可能会很慢，所以使用momentum来加速学习过程.

The method of momentum (Polyak, 1964) is designed to accelerate learning, especially in the face of high curvature, small but consistent gradients, or noisy gradients.

momentum的原理:

The momentum algorithm accumulates an exponentially decaying moving average of past gradients and continues to move in their direction.

determines how quickly the contributions of previous gradients exponentially decay.

• how large a sequence of gradients are.
• how aligned a sequence of gradients are.

step_size_with_momentum=lr/(1-α)*||g||，通常典型的α值是0.9

CS231n上有比较形象的解释: 地址在这 中文翻译过来就是：

## Nesterov momentum

Nesterov直接在前向位置(绿色箭头指向的位置)处更新梯度.

## SGD with nesterov momentum

individually adapts the learning rates of all model parameters by scaling them inversely proportional to the square root of the sum of all of their historical squared values.

The parameters with the largest partial derivative of the loss have a correspondingly rapid decrease in their learning rate, while parameters with small partial derivatives have a relatively small decrease in their learning rate.

the accumulation of squared gradients from the beginning of training can result in a premature and excessive decrease in the effective learning rate.

# RMSprop

1. AdaGrad shrinks the learning rate according to the entire history of the squared gradient and may have made the learning rate too small before arriving at such a convex structure.
2. RMSProp uses an exponentially decaying average to discard history from the extreme past so that it can converge rapidly after finding a convex bowl.

## RMSProp with nesterov momentum

1. First, in Adam, momentum is incorporated directly as an estimate of the first order moment (with exponential weighting) of the gradient.
2. Second, Adam includes bias corrections to the estimates of both the first-order moments (the momentum term) and the (uncentered) second-order moments to account for their initialization at the origin.

# 总结

1. 优化算法有一阶和二阶算法
3. 二阶算法由于计算的代价等问题不常用，比如牛顿法, BFGS, L-BFGS等

89 篇文章41 人订阅

0 条评论

## 相关文章

3727

6768

2826

4129

2706

3646

### 代码已公开--StarGAN-多领域图像翻译

github代码：https://github.com/yunjey/StarGAN 应用到多感知信息上面。

1304

### 【干货】基于注意力机制的神经匹配模型用于短文本检索

【导读】在基于检索的问答系统中，很重要的一步是将检索到的答案进行排序得到最佳的答案。在检索到的答案比较短时，对答案进行排序也成为了一个难题。使用深度学习的方法，...

6678

1322

### 【干货长文】神经网络浅讲：从神经元到深度学习

神经网络是一门重要的机器学习技术。它是目前最为火热的研究方向–深度学习的基础。学习神经网络不仅可以让你掌握一门强大的机器学习方法，同时也可以更好地帮助你理解...

2858