blocks|key|2184316|text|您认为索引不需要任何文件重组，而您描述索引的方式表明右利手指数是可以的。因此，索引似乎被排除为一个问题。当然，正如上面所建议的那样，您可以通过删除索引并重新运行大容量的写操作来明确地排除这一点。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|2184317|除了索引之外，我还会…|2184318|请考虑您的磁盘是否能够跟上您正在持久化的数据量。在蒙戈博士中有更多关于这方面的详细信息|unordered-list-item|2184319|使用侧写了解您的写作过程中发生了什么|2184320|entityMap|0|LINK|mutability|MUTABLE|url|https://docs.mongodb.com/manual/core/write-performance/#document-growth-and-the-mmapv1-storage-engine|1|https://docs.mongodb.com/manual/tutorial/ensure-indexes-fit-ram/#indexing-right-handed|2|https://docs.mongodb.com/manual/core/write-performance/#storage-performance|3|https://docs.mongodb.com/v3.4/administration/analyzing-mongodb-performance/#database-profiling^0|A|4|0|Q|5|1|0|0|P|4|2|0|2|2|3|0^^$0|@$1|2|3|4|5|6|7|Y|8|@]|9|@$A|Z|B|10|1|11]|$A|12|B|13|1|14]]|C|$]]|$1|D|3|E|5|6|7|15|8|@]|9|@]|C|$]]|$1|F|3|G|5|H|7|16|8|@]|9|@$A|17|B|18|1|19]]|C|$]]|$1|I|3|J|5|H|7|1A|8|@]|9|@$A|1B|B|1C|1|1D]]|C|$]]|$1|K|3|-4|5|6|7|1E|8|@]|9|@]|C|$]]]|L|$M|$5|N|O|P|C|$Q|R]]|S|$5|N|O|P|C|$Q|T]]|U|$5|N|O|P|C|$Q|V]]|W|$5|N|O|P|C|$Q|X]]]]

You believe that the indexing does not require any <a href="https://docs.mongodb.com/manual/core/write-performance/#document-growth-and-the-mmapv1-storage-engine" rel="nofollow noreferrer">document reorganisation</a> and the way you described the index suggests that a <a href="https://docs.mongodb.com/manual/tutorial/ensure-indexes-fit-ram/#indexing-right-handed" rel="nofollow noreferrer">right handed index</a> is ok. So, indexing seems to be ruled out as an issue. You could of course - as suggested above - definitively rule this out by dropping the index and re running your bulk writes.

Aside from indexing, I would …

<ul>
<li>Consider whether your disk can keep up with the volume of data you are persisting. More details on this in the <a href="https://docs.mongodb.com/manual/core/write-performance/#storage-performance" rel="nofollow noreferrer">Mongo docs</a></li>
<li>Use <a href="https://docs.mongodb.com/v3.4/administration/analyzing-mongodb-performance/#database-profiling" rel="nofollow noreferrer">profiling</a> to understand what’s happening with your writes</li>
</ul>

blocks|key|176514|text|你的收藏中有索引吗？如果是，则需要时间来构建索引树。|type|ordered-list-item|depth|inlineStyleRanges|entityRanges|data|176515|数据是时间序列吗？如果是，则更多地使用更新而不是插入。请看这个博客。该博客建议，就地更新比插入(https://www.mongodb.com/blog/post/schema-design-for-time-series-data-in-mongodb)更有效。|offset|length|176516|您有能力设置已分割的集合吗？如果是，它将减少时间(在3台带有1500万ip+geo入口记录的切分服务器上进行测试)|176517|unstyled|entityMap|0|LINK|mutability|MUTABLE|url|https://www.mongodb.com/blog/post/schema-design-for-time-series-data-in-mongodb^0|0|1C|27|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|R|8|@]|9|@$D|S|E|T|1|U]]|A|$]]|$1|F|3|G|5|6|7|V|8|@]|9|@]|A|$]]|$1|H|3|-4|5|I|7|W|8|@]|9|@]|A|$]]]|J|$K|$5|L|M|N|A|$O|P]]]]

<ol>
<li>Do have any index in your collection? 
If yes, it has to take time to build index tree.</li>
<li>is data time-series?
if yes, use updates more than inserts. Please read this blog. The blog suggests in-place updates more efficient than inserts (<a href="https://www.mongodb.com/blog/post/schema-design-for-time-series-data-in-mongodb" rel="nofollow noreferrer">https://www.mongodb.com/blog/post/schema-design-for-time-series-data-in-mongodb</a>)</li>
<li>do you have a capability to setup sharded collections?
if yes, it would reduce time (tested it in 3 sharded servers with 15million ip geo entry records)</li>
</ol>

blocks|key|1140984|text|磁盘利用率&+CPU:检查磁盘利用率和CPU，看看其中是否有任何问题正在消失。显然，应该是磁盘造成了这个问题。|type|unordered-list-item|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|1140985|Mongo：，如果1000个大容量查询耗时10秒，那么检查mongo中是否有任何需要时间的插入。如果存在这样的查询，则可以缩小分析范围。|1140986|另一件不清楚的事情是在Mongo实例上发生的查询顺序。插入是唯一发生的操作还是还有其他的查找查询也在运行？如果是，那么您应该考虑扩大任何资源正在最大化。|unstyled|1140987|entityMap^0|0|B|0|0|6|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@$9|N|A|O|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|P|8|@$9|Q|A|R|B|C]]|D|@]|E|$]]|$1|H|3|I|5|J|7|S|8|@]|D|@]|E|$]]|$1|K|3|-4|5|J|7|T|8|@]|D|@]|E|$]]]|L|$]]

<ul>
<li>Disk utilization &amp; CPU: Check the disk utilization and CPU and see if any of these are maxing out. 
Apparently, it should be the disk which is causing this issue for you. </li>
<li>Mongo log:
Also, if a 1000 bulk query is taking 10sec, then check for mongo log if there are any few inserts in the 1000 bulk that are taking time. If there are any such queries, then you can narrow down your analysis</li>
</ul>

Another thing that's not clear is the order of queries that happen on your Mongo instance. Is inserts the only operation that happens or there are other find queries that run too? If yes, then you should look at scaling up whatever resource is maxing out.

I'm using the BulkWriteOperation (java driver) to store data in large chunks. At first it seems to be working fine, but when the collection grows in size, the inserts can take quite a lot of time.

Currently for a collection of 20M documents, bulk insert of 1000 documents could take about 10 seconds.

Is there a way to make inserts independent of collection size?
I don't have any updates or upserts, it's always new data I'm inserting.

Judging from the log, there doesn't seem to be any issue with locks.
Each document has a time field which is indexed, but it's linearly growing so I don't see any need for mongo to take the time to reorganize the indexes.

I'd love to hear some ideas for improving the performance

Thanks

Bulk insert performance in MongoDB for large collections

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

社区新版编辑器体验调研

我使用BulkWriteOperation (java驱动程序)将数据存储在大块中。起初，它似乎工作良好，但是当集合的大小增加时，插入可能会花费相当长的时间。目前，对于2000万个文档的集合，大容量插入1000个文档可能需要大约10秒。是否有一种方法可以使插入与集合大小无关？我没有任何更新或更新，我总是插入新的数据。从...

问大型集合在MongoDB中的大容量插入性能
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问大型集合在MongoDB中的大容量插入性能EN