blocks|key|1074768|text|TL，DR:目前还没有已知的方法可以在适当的硬件设置下，在可容忍的时间内从头开始训练初始V3模型。我强烈建议重新训练一个预先训练好的模型来完成你想要的任务。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1074769|在像您这样的小型硬件设置上，将很难实现最大性能。一般来说，对于CNN来说，最好的性能是使用尽可能大的批量。这意味着对于CNN的训练过程通常受到GPU内存中可以容纳的最大批处理大小的限制。|1074770|可用于下载here的初始V3模型在50个GPU上的有效批处理大小为1600时进行了训练--其中每个GPU运行32批处理大小。|offset|length|1074771|考虑到你有限的硬件，我的第一个建议是从上面的链接下载预训练模式，并为你手头的单个任务重新训练模型。这会让你的生活更快乐。|1074772|作为一个思想实验(但几乎不实用)+..如果你觉得特别有必要通过从头开始训练来精确匹配预训练模型的训练性能，你可以在你的1个GPU上执行以下疯狂的过程。也就是说，您可以运行以下过程：|1074773|使用批处理大小为32的|1074774|运行|ordered-list-item|1074775|将run|1074776|中的梯度重复50次。|1074777|使用gradients.|1074778|Repeat|1074779|对来自50个变量的梯度进行平均|1074780|我提到这一点只是为了让您有一个概念性的感觉，即需要完成什么才能实现完全相同的性能。考虑到您提到的速度数字，此过程将需要几个月的时间才能运行。几乎不实用。|1074781|更现实地说，如果你仍然对从头开始训练并尽你所能做到最好感兴趣，这里有一些通用的指导原则：|1074782|1074783|始终以尽可能大的批处理大小运行。看起来你已经在这么做了。很好。|unordered-list-item|1074784|确保你不受CPU的限制。也就是说，确保TensorBoard上显示的输入处理队列总是适度地满。如果没有，请增加预处理线程的数量或使用不同的CPU+If+available.|1074785|Re:学习率。如果您总是运行同步训练(如果您只有一个GPU，则必须是这种情况)，那么批处理大小越大，可容忍的学习率就越高。我会尝试一系列的快速运行(例如，每次几个小时)，以找出不会导致NaN的最高学习率。在找到这样的学习率后，将其降低5-10%25，然后按此运行。|1074786|对于num_epochs_per_decay和decay_rate，有几种策略。每个衰减10个时期，0.001衰减因子突出显示的策略是尽可能长时间地锤击模型，直到评估精度渐近。然后降低学习率。这是一个很好的简单策略。我会验证这是你在你的模型中看到的，监测评估的准确性，并确定它确实是渐近的，然后才允许模型衰减学习率。最后，衰减系数有点特别，但是通过10的幂来降低似乎是一个很好的经验法则。|1074787|1074788|再次注意，这些是一般的指导方针，其他人甚至可能提供不同的建议。我们不能给你更具体的指导的原因是，这种大小的CNN通常不是在适度的硬件设置上从头开始训练的。|1074789|entityMap|0|LINK|mutability|MUTABLE|url|http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz^0|0|0|5|4|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|1P|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|1Q|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|1R|8|@]|9|@$F|1S|G|1T|1|1U]]|A|$]]|$1|H|3|I|5|6|7|1V|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|1W|8|@]|9|@]|A|$]]|$1|L|3|M|5|6|7|1X|8|@]|9|@]|A|$]]|$1|N|3|O|5|P|7|1Y|8|@]|9|@]|A|$]]|$1|Q|3|R|5|P|7|1Z|8|@]|9|@]|A|$]]|$1|S|3|T|5|P|7|20|8|@]|9|@]|A|$]]|$1|U|3|V|5|P|7|21|8|@]|9|@]|A|$]]|$1|W|3|X|5|P|7|22|8|@]|9|@]|A|$]]|$1|Y|3|Z|5|6|7|23|8|@]|9|@]|A|$]]|$1|10|3|11|5|6|7|24|8|@]|9|@]|A|$]]|$1|12|3|13|5|6|7|25|8|@]|9|@]|A|$]]|$1|14|3|-4|5|6|7|26|8|@]|9|@]|A|$]]|$1|15|3|16|5|17|7|27|8|@]|9|@]|A|$]]|$1|18|3|19|5|17|7|28|8|@]|9|@]|A|$]]|$1|1A|3|1B|5|17|7|29|8|@]|9|@]|A|$]]|$1|1C|3|1D|5|17|7|2A|8|@]|9|@]|A|$]]|$1|1E|3|-4|5|6|7|2B|8|@]|9|@]|A|$]]|$1|1F|3|1G|5|6|7|2C|8|@]|9|@]|A|$]]|$1|1H|3|-4|5|6|7|2D|8|@]|9|@]|A|$]]]|1I|$1J|$5|1K|1L|1M|A|$1N|1O]]]]

TL,DR: There is no known method for training an Inception V3 model from scratch in a tolerable amount of time from a modest hardware set up. I would strongly suggest retraining a pre-trained model on your desired task.

On a small hardware set up like yours, it will be difficult to achieve maximum performance. Generally speaking for CNN's, the best performance is with the largest batch sizes possible. This means that for CNN's the training procedure is often limited by the maximum batch size that can fit in GPU memory.

The Inception V3 model available for download <a href="http://download.tensorflow.org/models/image/imagenet/inception-v3-2016-03-01.tar.gz">here</a> was trained with an effective batch size of 1600 across 50 GPU's -- where each GPU ran a batch size of 32.

Given your modest hardware, my number one suggestion would be to download the pre-trained mode from the link above and retrain the model for the individual task you have at hand. This would make your life much happier.

As a thought experiment (but hardly practical) .. if you feel especially compelled to exactly match the training performance of the model from the pre-trained model by training from scratch, you could do the following insane procedure on your 1 GPU. Namely, you could run the following procedure:

<ol>
<li>Run with a batch size of 32</li>
<li>Store the gradients from the run</li>
<li>Repeat this 50 times.</li>
<li>Average the gradients from the 50 batches.</li>
<li>Update all variables with the gradients.</li>
<li>Repeat</li>
</ol>

I am only mentioning this to give you a conceptual sense of what would need to be accomplished to achieve the exact same performance. Given the speed numbers you mentioned, this procedure would take months to run. Hardly practical.

More realistically, if you are still strongly interested in training from scratch and doing the best you can, here are some general guidelines:

<ul>
<li>Always run with the largest batch size possible. It looks like you are already doing that. Great.</li>
<li>Make sure that you are not CPU bound. That is, make sure that the input processing queue's are always modestly full as displayed on TensorBoard. If not, increase the number of preprocessing threads or use a different CPU if available.</li>
<li>Re: learning rate. If you are always running synchronous training (which must be the case if you only have 1 GPU), then the higher batch size, the higher the tolerable learning rate. I would a try a series of several quick runs (e.g. a few hours each) to identify the highest learning possible which does not lead to NaN's. After you find such a learning rate, knock it down by say 5-10% and run with that.</li>
<li>As for num_epochs_per_decay and decay_rate, there are several strategies. The strategy highlighted by 10 epochs per decay, 0.001 decay factor is to hammer the model for as long as possible until the eval accuracy asymptotes. And then lower the learning rate. This is a simple strategy which is nice. I would verify that is what you see in your model monitoring that the eval accuracy and determining that it indeed asymptotes before you allow the model to decay the learning rate. Finally, the decay factor is a bit ad-hoc but lowering by say a power of 10 seems to be a good rule of thumb.</li>
</ul>

Note again that these are general guidelines and others might even offer differing advice. The reason why we can not give you more specific guidance is that CNNs of this size are just not often trained from scratch on a modest hardware setup.

blocks|key|1066291|text|很棒的建议。使用与您的设置类似的设置进行训练是有优先权的。看看这个-+http://3dvision.princeton.edu/pvt/GoogLeNet/这些人训练了GoogleNet，但使用的是咖啡因。尽管如此，研究他们的经验将是有用的。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1066292|entityMap|0|LINK|mutability|MUTABLE|url|http://3dvision.princeton.edu/pvt/GoogLeNet/^0|Z|18|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

Excellent tips. 
There is precedence for training using a similar setup as yours.
Check this out - <a href="http://3dvision.princeton.edu/pvt/GoogLeNet/" rel="nofollow noreferrer">http://3dvision.princeton.edu/pvt/GoogLeNet/</a>
These people trained GoogleNet, but, using Caffe. Still, studying their experience would be useful.

I've been training Inception V3 on a modest machine with a single GPU (GeForce GTX 980 Ti, 6GB). The maximum batch size appears to be around <code>40</code>.

I've used the default learning rate settings specified in the <code>inception_train.py</code> file: <code>initial_learning_rate = 0.1</code>, <code>num_epochs_per_decay = 30</code> and <code>learning_rate_decay_factor = 0.16</code>. After a couple of weeks of training the best accuracy I was able to achieve is as follows (About 500K-1M iterations):

<pre><code>2016-06-06 12:07:52.245005: precision @ 1 = 0.5767 recall @ 5 = 0.8143 [50016 examples]
2016-06-09 22:35:10.118852: precision @ 1 = 0.5957 recall @ 5 = 0.8294 [50016 examples]
2016-06-14 15:30:59.532629: precision @ 1 = 0.6112 recall @ 5 = 0.8396 [50016 examples]
2016-06-20 13:57:14.025797: precision @ 1 = 0.6136 recall @ 5 = 0.8423 [50016 examples]
</code></pre>

I've tried fiddling with the settings towards the end of the training session, but couldn't see any improvements in accuracy. 

I've started a new training session from scratch with <code>num_epochs_per_decay</code> = 10 and <code>learning_rate_decay_factor</code> = 0.001 based on some other posts in this forum, but it's sort of grasping in the dark here. 

Any recommendations on good defaults for a small hardware setup like mine?

Training Tensorflow Inception-v3 Imagenet on modest hardware setup

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我一直在一台普通的机器上训练盗梦空间V3，只有一颗GPU (GeForce GTX980Ti，6 6GB)。最大批处理大小似乎约为40。我使用了inception_train.py文件中指定的默认学习率设置：initial_learning_rate = 0.1、num_epochs_per_decay = 30和le...

问在适度硬件设置上训练Tensorflow Inception-v3 Imagenet
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在适度硬件设置上训练Tensorflow Inception-v3 ImagenetEN