blocks|key|2614533|text|我在网上看不到这方面的任何信息，链接的论文肯定没有帮助，所以我看了一下unit+tests+for+tf.train.MomentumOptimizer，从中我可以看到对经典动量和NAG模式的实现的测试。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2614534|摘要|2614535|var+=+var+%2B+accum+*+learning_rate+*+momentum
accum+=+accum+*+momentum+%2B+g
var+=+var+-+learning_rate+*+accum
var+=+var+-+accum+*+learning_rate+*+momentum|code-block|syntax|javascript|2614536|其中accum从0开始，每一步都会更新。上面是单元测试中公式的修改版本，我觉得有点混乱。下面是与我对每个参数所代表的内容的解释相同的一组方程(尽管我可能是错的)：|2614537|average_grad_0+=+accum+#+previous+rolling+average
average_grad_1+=+accum+*+momentum+%2B+g+#+updated+rolling+average
grad_diff+=+average_grad_1+-+average_grad_0
adjustment+=+-learning_rate+*+(grad_diff+*+momentum+%2B+average_grad_1)
var+%2B=+adjustment
accum+=+average_grad_new|2614538|换句话说，在我看来，tensorflow的实现试图通过假设新的梯度将由当前平均梯度加上动量的乘积和平均梯度的变化来猜测NAG中的“调整后的梯度”，我希望看到这方面的证据！|2614539|以下是根据测试如何在tensorflow中实现经典模式和内斯特罗夫模式的更详细信息。|2614540|经典动量模型|2614541|对于use_nesterov=False，基于doTestBasic函数，我们有以下初始参数：|2614542|learning_rate+=+2.0
momentum+=+0.9
var_0+=+1.0+#+at+time+0
grad+=+0.1|2614543|实际上，上面只是grads_0和vars_0数组的第一个元素，但我将只关注一个值。对于随后的时间步长，我们有|2614544|var_1+=+1.0+-+(0.1+*+2.0)
var_2+=+1.0+-+(0.1+*+2.0)+-+((0.9+*+0.1+%2B+0.1)+*+2.0)|2614545|我将把它解释为意义；|2614546|var_1+=+var_0+-+(grad+*+learning_rate)
var_2+=+var_1+-+((momentum+*+grad+%2B+grad)+*+learning_rate)|2614547|如果我们假设单元测试的目的是grad_0+==+grad_1+==+grad，那么这作为经典动量公式是有意义的。|2614548|内斯特罗夫加速梯度(NAG)模式|2614549|对于use_nesterov=True，我查看了_update_nesterov_momentum_numpy函数和testNesterovMomentum测试用例。|2614550|_update_nesterov_momentum_numpy函数具有以下定义：|2614551|++def+_update_nesterov_momentum_numpy(self,+var,+accum,+g,+lr,+momentum):
++++var+=+var+%2B+accum+*+lr+*+momentum
++++accum+=+accum+*+momentum+%2B+g
++++var+=+var+-+lr+*+accum
++++var+=+var+-+accum+*+lr+*+momentum
++++return+var,+accum|2614552|它在单元测试中的调用如下所示：|2614553|++++for+t+in+range(1,+5):
++++++opt_op.run()
++++++var0_np,+accum0_np+=+self._update_nesterov_momentum_numpy(
++++++++++var0_np,+accum0_np,+var0_np+*+10,+2.0,+0.9)|2614554|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/python/training/momentum_test.py^0|1E|Q|Z|15|0|0|0|0|2|5|0|0|A|A|0|A|A|0|0|2|I|N|B|0|0|8|7|G|6|0|0|0|0|E|O|0|0|2|H|O|V|1M|K|0|0|V|0|0|0|0^^$0|@$1|2|3|4|5|6|7|1U|8|@$9|1V|A|1W|B|C]]|D|@$9|1X|A|1Y|1|1Z]]|E|$]]|$1|F|3|G|5|6|7|20|8|@]|D|@]|E|$]]|$1|H|3|I|5|J|7|21|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|22|8|@$9|23|A|24|B|C]]|D|@]|E|$]]|$1|O|3|P|5|J|7|25|8|@]|D|@]|E|$K|L]]|$1|Q|3|R|5|6|7|26|8|@$9|27|A|28|B|C]]|D|@]|E|$]]|$1|S|3|T|5|6|7|29|8|@$9|2A|A|2B|B|C]]|D|@]|E|$]]|$1|U|3|V|5|6|7|2C|8|@]|D|@]|E|$]]|$1|W|3|X|5|6|7|2D|8|@$9|2E|A|2F|B|C]|$9|2G|A|2H|B|C]]|D|@]|E|$]]|$1|Y|3|Z|5|J|7|2I|8|@]|D|@]|E|$K|L]]|$1|10|3|11|5|6|7|2J|8|@$9|2K|A|2L|B|C]|$9|2M|A|2N|B|C]]|D|@]|E|$]]|$1|12|3|13|5|J|7|2O|8|@]|D|@]|E|$K|L]]|$1|14|3|15|5|6|7|2P|8|@]|D|@]|E|$]]|$1|16|3|17|5|J|7|2Q|8|@]|D|@]|E|$K|L]]|$1|18|3|19|5|6|7|2R|8|@$9|2S|A|2T|B|C]]|D|@]|E|$]]|$1|1A|3|1B|5|6|7|2U|8|@]|D|@]|E|$]]|$1|1C|3|1D|5|6|7|2V|8|@$9|2W|A|2X|B|C]|$9|2Y|A|2Z|B|C]|$9|30|A|31|B|C]]|D|@]|E|$]]|$1|1E|3|1F|5|6|7|32|8|@$9|33|A|34|B|C]]|D|@]|E|$]]|$1|1G|3|1H|5|J|7|35|8|@]|D|@]|E|$K|L]]|$1|1I|3|1J|5|6|7|36|8|@]|D|@]|E|$]]|$1|1K|3|1L|5|J|7|37|8|@]|D|@]|E|$K|L]]|$1|1M|3|-4|5|6|7|38|8|@]|D|@]|E|$]]]|1N|$1O|$5|1P|1Q|1R|E|$1S|1T]]]]

I couldn't see any info on this online, and the linked paper certainly wasn't helpful, so I had a look at the <a href="https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/python/training/momentum_test.py" rel="nofollow noreferrer">unit tests for <code>tf.train.MomentumOptimizer</code></a>, from which I can see tests for the implementation of both classic momentum and NAG modes.

<h2>Summary</h2>

<pre><code>var = var + accum * learning_rate * momentum
accum = accum * momentum + g
var = var - learning_rate * accum
var = var - accum * learning_rate * momentum
</code></pre>

where <code>accum</code> starts at 0 and is updated at every step. The above is a modified version of the formulation in the unit test, and I find it a bit confusing. Here is the same set of equations arranged with my interpretation of what each of the parameters represent (I could be wrong though):

<pre><code>average_grad_0 = accum # previous rolling average
average_grad_1 = accum * momentum + g # updated rolling average
grad_diff = average_grad_1 - average_grad_0
adjustment = -learning_rate * (grad_diff * momentum + average_grad_1)
var += adjustment
accum = average_grad_new
</code></pre>

In other words, it seems to me like <code>tensorflow</code>'s implementation attempts to guess the "adjusted gradient" in NAG by assuming that the new gradient will be esimated by the current average gradient plus the product of momentum and the change in the average gradient. I'd love to see a proof for this!

What follows is more detail on how the classic and nesterov modes are implemented in <code>tensorflow</code> as per the tests.

<h2>Classic Momentum mode</h2>

For <code>use_nesterov=False</code>, based on the <code>doTestBasic</code> function, we have the following initial parameters:

<pre><code>learning_rate = 2.0
momentum = 0.9
var_0 = 1.0 # at time 0
grad = 0.1
</code></pre>

Actually, the above are just the first element of the <code>grads_0</code> and <code>vars_0</code> arrays, but I'll just focus on a single value. For the subsequent timesteps, we have

<pre><code>var_1 = 1.0 - (0.1 * 2.0)
var_2 = 1.0 - (0.1 * 2.0) - ((0.9 * 0.1 + 0.1) * 2.0)
</code></pre>

which I'm going to interpret as meaning;

<pre><code>var_1 = var_0 - (grad * learning_rate)
var_2 = var_1 - ((momentum * grad + grad) * learning_rate)
</code></pre>

If we assume that for the purposes of the unit tests <code>grad_0 == grad_1 == grad</code> then this makes sense as a formulation of classic momentum.

<h2>Nesterov's Accelerated Gradient (NAG) mode</h2>

For <code>use_nesterov=True</code>, I had a look at the <code>_update_nesterov_momentum_numpy</code> function and the <code>testNesterovMomentum</code> test case.

The <code>_update_nesterov_momentum_numpy</code> function has the following definition:

<pre><code> def _update_nesterov_momentum_numpy(self, var, accum, g, lr, momentum):
 var = var + accum * lr * momentum
 accum = accum * momentum + g
 var = var - lr * accum
 var = var - accum * lr * momentum
 return var, accum
</code></pre>

and it is called in the unit tests like this:

<pre><code> for t in range(1, 5):
 opt_op.run()
 var0_np, accum0_np = self._update_nesterov_momentum_numpy(
 var0_np, accum0_np, var0_np * 10, 2.0, 0.9)
</code></pre>

The documentation for <a href="https://www.tensorflow.org/api_docs/python/tf/train/MomentumOptimizer" rel="noreferrer"><code>tf.train.MomentumOptimizer</code></a> offers a <code>use_nesterov</code> parameter to utilise Nesterov's Accelerated Gradient (NAG) method.

However, NAG requires the gradient at a location other than that of the current variable to be calculated, and the <code>apply_gradients</code> interface only allows for the current gradient to be passed. So I don't quite understand how the NAG algorithm could be implemented with this interface.

The documentation says the following about the implementation:

<blockquote>
 <code>use_nesterov</code>: If True use Nesterov Momentum. See <a href="http://proceedings.mlr.press/v28/sutskever13.pdf" rel="noreferrer">Sutskever et al.,
 2013</a>. This
 implementation always computes gradients at the value of the
 variable(s) passed to the optimizer. Using Nesterov Momentum makes the
 variable(s) track the values called <code>theta_t + mu*v_t</code> in the paper.
</blockquote>

Having read through the paper in the link, I'm a little unsure about whether this description answers my question or not. How can the NAG algorithm be implemented when the interface doesn't require a gradient function to be provided?

How is Nesterov's Accelerated Gradient Descent implemented in Tensorflow?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

的文档提供了一个use_nesterov参数来使用Nesterov的加速梯度(NAG)方法。但是，NAG要求计算当前变量所在位置以外的位置处的梯度，并且apply_gradients接口只允许传递当前梯度。所以我不太明白NAG算法是如何通过这个接口实现的。文档描述了关于实现的以下内容： use_nesterov：如果为...

Nesterov的加速梯度下降是如何在Tensorflow中实现的？-腾讯云开发者社区-腾讯云

问Nesterov的加速梯度下降是如何在Tensorflow中实现的？
EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Nesterov的加速梯度下降是如何在Tensorflow中实现的？EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Nesterov的加速梯度下降是如何在Tensorflow中实现的？
EN