blocks|key|4451041|text|首先，tf.train.GradientDescentOptimizer被设计为在所有步骤中对所有变量使用恒定的学习率。TensorFlow还提供了开箱即用的自适应优化器，包括tf.train.AdagradOptimizer和tf.train.AdamOptimizer，这些可以作为替代。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|4451042|但是，如果您希望使用普通的梯度下降来控制学习率，则可以利用tf.train.GradientDescentOptimizer+constructor的learning_rate参数可以是Tensor对象这一事实。这允许您在每个步骤中计算不同的学习率值，例如：|4451043|learning_rate+=+tf.placeholder(tf.float32,+shape=[])
#+...
train_step+=+tf.train.GradientDescentOptimizer(
++++learning_rate=learning_rate).minimize(mse)

sess+=+tf.Session()

#+Feed+different+values+for+learning+rate+to+each+training+step.
sess.run(train_step,+feed_dict={learning_rate:+0.1})
sess.run(train_step,+feed_dict={learning_rate:+0.1})
sess.run(train_step,+feed_dict={learning_rate:+0.01})
sess.run(train_step,+feed_dict={learning_rate:+0.01})|code-block|syntax|javascript|4451044|或者，您可以创建保存学习率的标量tf.Variable，并在每次要更改学习率时指定它。|4451045|entityMap|0|LINK|mutability|MUTABLE|url|http://www.tensorflow.org/api_docs/python/train.html#AdagradOptimizer|1|http://www.tensorflow.org/api_docs/python/train.html#AdamOptimizer|2|http://www.tensorflow.org/api_docs/python/train.html#GradientDescentOptimizer.__init__^0|3|X|2G|P|36|M|2G|P|0|36|M|1|0|T|X|23|D|2L|6|T|19|2|0|0|G|B|0^^$0|@$1|2|3|4|5|6|7|10|8|@$9|11|A|12|B|C]|$9|13|A|14|B|C]|$9|15|A|16|B|C]]|D|@$9|17|A|18|1|19]|$9|1A|A|1B|1|1C]]|E|$]]|$1|F|3|G|5|6|7|1D|8|@$9|1E|A|1F|B|C]|$9|1G|A|1H|B|C]|$9|1I|A|1J|B|C]]|D|@$9|1K|A|1L|1|1M]]|E|$]]|$1|H|3|I|5|J|7|1N|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|1O|8|@$9|1P|A|1Q|B|C]]|D|@]|E|$]]|$1|O|3|-4|5|6|7|1R|8|@]|D|@]|E|$]]]|P|$Q|$5|R|S|T|E|$U|V]]|W|$5|R|S|T|E|$U|X]]|Y|$5|R|S|T|E|$U|Z]]]]

First of all, <code>tf.train.GradientDescentOptimizer</code> is designed to use a constant learning rate for all variables in all steps. TensorFlow also provides out-of-the-box adaptive optimizers including the <a href="http://www.tensorflow.org/api_docs/python/train.html#AdagradOptimizer"><code>tf.train.AdagradOptimizer</code></a> and the <a href="http://www.tensorflow.org/api_docs/python/train.html#AdamOptimizer"><code>tf.train.AdamOptimizer</code></a>, and these can be used as drop-in replacements.

However, if you want to control the learning rate with otherwise-vanilla gradient descent, you can take advantage of the fact that the <code>learning_rate</code> argument to the <a href="http://www.tensorflow.org/api_docs/python/train.html#GradientDescentOptimizer.__init__"><code>tf.train.GradientDescentOptimizer</code> constructor</a> can be a <code>Tensor</code> object. This allows you to compute a different value for the learning rate in each step, for example:

<pre><code>learning_rate = tf.placeholder(tf.float32, shape=[])
# ...
train_step = tf.train.GradientDescentOptimizer(
 learning_rate=learning_rate).minimize(mse)

sess = tf.Session()

# Feed different values for learning rate to each training step.
sess.run(train_step, feed_dict={learning_rate: 0.1})
sess.run(train_step, feed_dict={learning_rate: 0.1})
sess.run(train_step, feed_dict={learning_rate: 0.01})
sess.run(train_step, feed_dict={learning_rate: 0.01})
</code></pre>

Alternatively, you could create a scalar <code>tf.Variable</code> that holds the learning rate, and assign it each time you want to change the learning rate.

blocks|key|4450948|text|Tensorflow提供了一个op来自动将指数衰减应用于学习率张量：tf.train.exponential_decay。有关它的使用示例，请参阅this+line+in+the+MNIST+convolutional+model+example。然后使用上面的@mrry建议将这个变量作为learning_rate参数提供给您选择的优化器。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|4450949|要看的关键摘录是：|4450950|#+Optimizer:+set+up+a+variable+that's+incremented+once+per+batch+and
#+controls+the+learning+rate+decay.
batch+=+tf.Variable(0)

learning_rate+=+tf.train.exponential_decay(
++0.01,++++++++++++++++#+Base+learning+rate.
++batch+*+BATCH_SIZE,++#+Current+index+into+the+dataset.
++train_size,++++++++++#+Decay+step.
++0.95,++++++++++++++++#+Decay+rate.
++staircase=True)
#+Use+simple+momentum+for+the+optimization.
optimizer+=+tf.train.MomentumOptimizer(learning_rate,
+++++++++++++++++++++++++++++++++++++0.9).minimize(loss,
+++++++++++++++++++++++++++++++++++++++++++++++++++global_step=batch)|code-block|syntax|javascript|4450951|注意要最小化的global_step=batch参数。这告诉优化器在每次训练时为你增加'batch‘参数。|4450952|entityMap|0|LINK|mutability|MUTABLE|url|http://www.tensorflow.org/api_docs/python/train.html#exponential_decay|1|https://github.com/tensorflow/models/blob/master/tutorials/image/mnist/convolutional.py#L243^0|Y|Q|Y|Q|0|21|1E|1|0|0|0|7|H|0^^$0|@$1|2|3|4|5|6|7|Y|8|@$9|Z|A|10|B|C]]|D|@$9|11|A|12|1|13]|$9|14|A|15|1|16]]|E|$]]|$1|F|3|G|5|6|7|17|8|@]|D|@]|E|$]]|$1|H|3|I|5|J|7|18|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|19|8|@$9|1A|A|1B|B|C]]|D|@]|E|$]]|$1|O|3|-4|5|6|7|1C|8|@]|D|@]|E|$]]]|P|$Q|$5|R|S|T|E|$U|V]]|W|$5|R|S|T|E|$U|X]]]]

Tensorflow provides an op to automatically apply an exponential decay to a learning rate tensor: <a href="http://www.tensorflow.org/api_docs/python/train.html#exponential_decay" rel="noreferrer"><code>tf.train.exponential_decay</code></a>. For an example of it in use, see <a href="https://github.com/tensorflow/models/blob/master/tutorials/image/mnist/convolutional.py#L243" rel="noreferrer">this line in the MNIST convolutional model example</a>. Then use @mrry's suggestion above to supply this variable as the learning_rate parameter to your optimizer of choice.

The key excerpt to look at is:

<pre><code># Optimizer: set up a variable that's incremented once per batch and
# controls the learning rate decay.
batch = tf.Variable(0)

learning_rate = tf.train.exponential_decay(
 0.01, # Base learning rate.
 batch * BATCH_SIZE, # Current index into the dataset.
 train_size, # Decay step.
 0.95, # Decay rate.
 staircase=True)
# Use simple momentum for the optimization.
optimizer = tf.train.MomentumOptimizer(learning_rate,
 0.9).minimize(loss,
 global_step=batch)
</code></pre>

Note the <code>global_step=batch</code> parameter to minimize. That tells the optimizer to helpfully increment the 'batch' parameter for you every time it trains.

blocks|key|4451142|text|来自tensorflow官方文档|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|4451143|global_step+=+tf.Variable(0,+trainable=False)
starter_learning_rate+=+0.1
learning_rate+=+tf.train.exponential_decay(starter_learning_rate,+global_step,
+++++++++++++++++++++++++++++++++++++++100000,+0.96,+staircase=True)

#+Passing+global_step+to+minimize()+will+increment+it+at+each+step.
learning_step+=+(
tf.train.GradientDescentOptimizer(learning_rate)
.minimize(...my+loss...,+global_step=global_step))|code-block|syntax|javascript|4451144|entityMap|0|LINK|mutability|MUTABLE|url|https://www.tensorflow.org/api_docs/python/tf/train/exponential_decay^0|2|A|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

From <a href="https://www.tensorflow.org/api_docs/python/tf/train/exponential_decay" rel="noreferrer">tensorflow</a> official docs

<pre><code>global_step = tf.Variable(0, trainable=False)
starter_learning_rate = 0.1
learning_rate = tf.train.exponential_decay(starter_learning_rate, global_step,
 100000, 0.96, staircase=True)

# Passing global_step to minimize() will increment it at each step.
learning_step = (
tf.train.GradientDescentOptimizer(learning_rate)
.minimize(...my loss..., global_step=global_step))
</code></pre>

blocks|key|4451024|text|如果你想为像0+<+a+<+b+<+c+<+...这样的时间间隔设置特定的学习率。然后，您可以将学习率定义为条件张量，以全局步长为条件，并将其作为正常情况提供给优化器。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|4451025|您可以使用一堆嵌套的tf.cond语句来实现这一点，但递归地构建张量更容易：|4451026|def+make_learning_rate_tensor(reduction_steps,+learning_rates,+global_step):
++++assert+len(reduction_steps)+%2B+1+==+len(learning_rates)
++++if+len(reduction_steps)+==+1:
++++++++return+tf.cond(
++++++++++++global_step+<+reduction_steps[0],
++++++++++++lambda:+learning_rates[0],
++++++++++++lambda:+learning_rates[1]
++++++++)
++++else:
++++++++return+tf.cond(
++++++++++++global_step+<+reduction_steps[0],
++++++++++++lambda:+learning_rates[0],
++++++++++++lambda:+make_learning_rate_tensor(
++++++++++++++++reduction_steps[1:],
++++++++++++++++learning_rates[1:],
++++++++++++++++global_step,)
++++++++++++)|code-block|syntax|javascript|4451027|然后要使用它，你需要知道在单个时期有多少训练步骤，以便我们可以使用全局步骤在正确的时间切换，并最终定义你想要的时期和学习率。因此，如果我想要分别在[0,+19],+[20,+59],+[60,+99],+[100,+\infty]的时期间隔内的学习率[0.1,+0.01,+0.001,+0.0001]，我会这样做：|4451028|global_step+=+tf.train.get_or_create_global_step()
learning_rates+=+[0.1,+0.01,+0.001,+0.0001]
steps_per_epoch+=+225
epochs_to_switch_at+=+[20,+60,+100]
epochs_to_switch_at+=+[x*steps_per_epoch+for+x+in+epochs_to_switch_at+]
learning_rate+=+make_learning_rate_tensor(epochs_to_switch_at+,+learning_rates,+global_step)|4451029|entityMap^0|6|J|0|A|7|0|0|21|16|3H|Q|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@$9|T|A|U|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|V|8|@$9|W|A|X|B|C]]|D|@]|E|$]]|$1|H|3|I|5|J|7|Y|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|Z|8|@$9|10|A|11|B|C]|$9|12|A|13|B|C]]|D|@]|E|$]]|$1|O|3|P|5|J|7|14|8|@]|D|@]|E|$K|L]]|$1|Q|3|-4|5|6|7|15|8|@]|D|@]|E|$]]]|R|$]]

If you want to set specific learning rates for intervals of epochs like <code>0 &lt; a &lt; b &lt; c &lt; ...</code>. Then you can define your learning rate as a conditional tensor, conditional on the global step, and feed this as normal to the optimiser. 

You could achieve this with a bunch of nested <code>tf.cond</code> statements, but its easier to build the tensor recursively:

<pre><code>def make_learning_rate_tensor(reduction_steps, learning_rates, global_step):
 assert len(reduction_steps) + 1 == len(learning_rates)
 if len(reduction_steps) == 1:
 return tf.cond(
 global_step &lt; reduction_steps[0],
 lambda: learning_rates[0],
 lambda: learning_rates[1]
 )
 else:
 return tf.cond(
 global_step &lt; reduction_steps[0],
 lambda: learning_rates[0],
 lambda: make_learning_rate_tensor(
 reduction_steps[1:],
 learning_rates[1:],
 global_step,)
 )
</code></pre>

Then to use it you need to know how many training steps there are in a single epoch, so that we can use the global step to switch at the right time, and finally define the epochs and learning rates you want. So if I want the learning rates <code>[0.1, 0.01, 0.001, 0.0001]</code> during the epoch intervals of <code>[0, 19], [20, 59], [60, 99], [100, \infty]</code> respectively, I would do:

<pre><code>global_step = tf.train.get_or_create_global_step()
learning_rates = [0.1, 0.01, 0.001, 0.0001]
steps_per_epoch = 225
epochs_to_switch_at = [20, 60, 100]
epochs_to_switch_at = [x*steps_per_epoch for x in epochs_to_switch_at ]
learning_rate = make_learning_rate_tensor(epochs_to_switch_at , learning_rates, global_step)
</code></pre>

I am using TensorFlow to train a neural network. This is how I am initializing the <code>GradientDescentOptimizer</code>:

<pre><code>init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

mse = tf.reduce_mean(tf.square(out - out_))
train_step = tf.train.GradientDescentOptimizer(0.3).minimize(mse)
</code></pre>

The thing here is that I don't know how to set an update rule for the learning rate or a decay value for that. 

How can I use an adaptive learning rate here?

How to set adaptive learning rate for GradientDescentOptimizer?

自适应学习

我正在使用TensorFlow来训练神经网络。这就是我初始化GradientDescentOptimizer的方法init = tf.initialize_all_variables()sess = tf.Session()sess.run(init)mse        = tf.reduce_mean(tf.squ...

问如何为GradientDescentOptimizer设置自适应学习率？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何为GradientDescentOptimizer设置自适应学习率？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何为GradientDescentOptimizer设置自适应学习率？
EN