blocks|key|922621|text|很明显你让读头跳过了整个地方。您的瓶颈是磁盘，而不是处理器。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|922622|换句话来说，CPU可能是并行的，但磁盘不是。|922623|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|G|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|H|8|@]|9|@]|A|$]]]|E|$]]

Well apparently you're causing the read head to skip around all over the place. Your bottleneck is the disk, not the processor.

To re-phrase, the CPU might be parrallel but the disk isn't.

blocks|key|783150|text|您是否看到加速几乎肯定将取决于您正在查看的场景和硬件。关于您的基准方法的更多细节在这里将是有用的。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|783151|在粗略的级别上，当您没有使用i/o控制器的最大吞吐量并且它是缓存时，或者当您将i/o与CPU密集型工作重叠时，当它们被阻止等待对方时，就会出现加速的机会。|783152|您是在比较系统中多个小文件的读取，还是只是按顺序读取几个大文件？您将在这里看到不同的性能特性。|783153|您是否使用过像(免费)+windows性能工具包这样的好的系统分析器来查看基准测试中发生了什么？这几乎是必须的。|offset|length|783154|这类基准测试可以很有趣地编写和分析，不要让一些错误的开始妨碍挖掘和寻找加速。|783155|-Rick|783156|entityMap|0|LINK|mutability|MUTABLE|url|http://msdn.microsoft.com/en-us/performance/default.aspx^0|0|0|0|C|C|0|0|0|0^^$0|@$1|2|3|4|5|6|7|V|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|W|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|X|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|Y|8|@]|9|@$H|Z|I|10|1|11]]|A|$]]|$1|J|3|K|5|6|7|12|8|@]|9|@]|A|$]]|$1|L|3|M|5|6|7|13|8|@]|9|@]|A|$]]|$1|N|3|-4|5|6|7|14|8|@]|9|@]|A|$]]]|O|$P|$5|Q|R|S|A|$T|U]]]]

Whether or not you are seeing speedup will almost assuredly depend on the scenario you are looking at and the hardware. More details on your benchmarking methodology would be useful here.

At a coarse level, the opportunity for a speedup arises when you're not utilizing the maximum throughput of the i/o controller and it's caches or when you are overlapping i/o with CPU intensive work and they are blocked waiting for each other.

Are you comparing doing reads of multiple small files spread out across the system, or just reading a few large files sequentially? You'll see different performance characteristics here.

Have you profiled with a good systems profiler like the (free) <a href="http://msdn.microsoft.com/en-us/performance/default.aspx" rel="nofollow noreferrer">windows performance toolkit</a> to see what is going on in your benchmarks? This is practically a must.

These kind of benchmarks can be a lot of fun to write and profile, don't let a few false starts get in the way of digging in and looking for speedups.

-Rick

blocks|key|783182|text|我认为您关于操作系统优化并发磁盘访问的假设是完全错误的。我设想，当您使用来自单个线程的分散/收集I/O时，它会进行这种重新排序，但是没有实际的方法来以这种方式优化并发请求。任何这样的方案都会在单线程读取中引入不必要的延迟。(如果出现并发请求，操作系统将不得不等待一会儿。)总之，简单的回答是，并发请求导致读取头跳遍了整个位置。操作系统无法对此进行优化。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|783183|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

I think your assumption about the OS optimizing concurrent disk access is simply false. I imagine it does this sort of re-ordering when you use scatter/gather I/O from a single thread, but there's no practical way for it to optimize concurrent requests in this way. Any such scheme would introduce unnecessary latency in single-threaded reads. (The OS would have to wait a bit just in case a concurrent request came in.) Anyway, the short answer is that your concurrent requests are causing the read heads to jump all over the place. The OS cannot optimize this away.

blocks|key|2307488|text|我认为您是在谈论本机命令队列，它可能在您正在测试的系统上启用，也可能不启用。来自维基百科条目|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|2307489|事实上，更新的主流Linux内核本机支持AHCI。即使主机总线适配器上存在AHCI，Windows+XP也需要安装特定于供应商的驱动程序。Windows本机支持AHCI和NCQ。自8.0版以来，FreeBSD完全支持AHCI和NCQ。|blockquote|2307490|另外，我还没有做过任何测试，但是NCQ对于必须访问磁盘上的小文件/inode的目录遍历可能不那么有效。可能是磁盘控制器能够以足够快的速度处理每个请求，从而使队列永远不会建立到重新排序的程度，因此您看不到任何好处。|2307491|entityMap|0|LINK|mutability|MUTABLE|url|http://en.wikipedia.org/wiki/Native_command_queuing^0|14|6|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$]]|$1|G|3|H|5|6|7|V|8|@]|9|@]|C|$]]|$1|I|3|-4|5|6|7|W|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

I think you are talking about native command queuing, which may or may not be enabled on the system you are testing with. From <a href="http://en.wikipedia.org/wiki/Native_command_queuing" rel="nofollow noreferrer">the Wikipedia entry</a>:

<blockquote>
 In fact, newer mainstream Linux kernels support AHCI natively. Windows XP requires the installation of a vendor-specific driver even if AHCI is present on the host bus adapter. Windows Vista natively supports both AHCI and NCQ. FreeBSD fully supports AHCI and NCQ since version 8.0.
</blockquote>

Also, I haven't done any tests, but NCQ may not be that effective for a directory walk that has to access small files/inodes all over the disk. It could be that the disk controller is able to service each request fast enough that a queue is never built up to reorder, thus you don't see any benefit.

blocks|key|783218|text|在这里，将目录或文件信息的读取与该信息的处理分开可能是很重要的。换句话说，磁盘IO在一个线程中，处理和搜索在另一个线程中。将已完成的IO信息传递给具有有界队列的处理线程。通过这样做，您将确保IO线程不会在处理下一个要处理的数据块之前等待结果的处理。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|783219|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

It's probably important here that you split the reading of the directory or file information away from the processing of that information. In other words, disk IO in one thread, processing and searching in another. Pass completed IO information to the processing thread with a bounded queue. By doing this you'll ensure that your IO thread is never waiting on the processing of results before getting busy on the read of the next block of data to process.

blocks|key|2307567|text|解决方案：使用NCQ来提高性能。为此，请将SATA+HDD控制器配置为使用AHCI.。|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|2307568|详情如下：|2307569|在分析特定的应用程序时，我也做过类似的观察。在我的四核系统中，我比较了以下配置：|2307570|仅1核心:相当快|unordered-list-item|2307571|4核启用:慢得多！这让我很惊讶，也让我感到困惑。|2307572|事实证明，该应用程序正在执行大量并发的HDD访问。如果有多个核心(因此也有多个线程)，这将明显地减慢总执行时间。|2307573|我做了一些研究，并了解到一个名为NCQ+(本机命令队列)的特性将对您所指的HDD访问进行优化。|2307574|在SCSI世界中，这一标准已有相当一段时间了。在SATA的世界里，它已经适应了一段时间。要解锁此功能，需要配置您的HDD控制器以AHCI模式操作--这是使用NCQ的先决条件！|2307575|由于现在普通的桌面系统使用车载HDD控制器，这部分配置需要在BIOS设置中完成。对于SATA配置，通常可以在以下操作模式中进行选择：|2307576|兼容/遗留IDE|2307577|AHCI|2307578|我继续并实现了我自己的自定义基准测试，以比较使用以下配置运行的同一个系统：|2307579|启用4个核心，遗留IDE:相当慢|2307580|4核启用，AHCI+/+NCQ:快得多。特定的基准测试部分执行6倍的速度！|2307581|--|2307582|结论|2307583|释放具有并行HDD访问权限的系统的全部能力：|2307584|切换到AHCI+(以便您可以利用NCQ)|ordered-list-item|2307585|不要使用操作系统附带的通用AHCI驱动程序。相反，使用特定于供应商的优化驱动程序。示例:+windows+7附带了一些通用的AHCI驱动程序，这些驱动程序支持大多数常见的HDD控制器。然而，在使用英特尔芯片组时，请确保安装英特尔“矩阵存储管理器”或英特尔“快速存储技术”(例如英特尔RST+11.7)。经过优化的驱动程序已经证明可以进一步提高HDD的性能。|2307586|不这样做将使一些应用程序在使用多个线程而不是单个线程时运行更慢的。这是你需要考虑的令人惊讶的部分。|2307587|2307588|注意:有一个神话说:+NCQ只与“服务器”环境相关(数百个进程并行运行)。我的基准测试结果指向了另一个方向:它也与“桌面”环境相关。每当出现大量并发HDD访问时，都会发生。|2307589|补充说明：|2307590|一些较旧的芯片组/+SATA控制器不支持AHCI模式。但这里没有覆盖。|2307591|有些“旧”操作系统在安装AHCI模式或将系统从IDE模式迁移到AHCI模式时需要特殊操作。但这里没有覆盖。|2307592|entityMap^0|5|11|0|0|0|0|0|0|G|3|0|1S|4|0|U|6|0|0|0|0|0|0|0|0|2|0|0|3|4|G|3|0|0|T|3|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|1U|8|@$9|1V|A|1W|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|1X|8|@]|D|@]|E|$]]|$1|H|3|I|5|6|7|1Y|8|@]|D|@]|E|$]]|$1|J|3|K|5|L|7|1Z|8|@]|D|@]|E|$]]|$1|M|3|N|5|L|7|20|8|@]|D|@]|E|$]]|$1|O|3|P|5|6|7|21|8|@]|D|@]|E|$]]|$1|Q|3|R|5|6|7|22|8|@$9|23|A|24|B|C]]|D|@]|E|$]]|$1|S|3|T|5|6|7|25|8|@$9|26|A|27|B|C]]|D|@]|E|$]]|$1|U|3|V|5|6|7|28|8|@$9|29|A|2A|B|C]]|D|@]|E|$]]|$1|W|3|X|5|L|7|2B|8|@]|D|@]|E|$]]|$1|Y|3|Z|5|L|7|2C|8|@]|D|@]|E|$]]|$1|10|3|11|5|6|7|2D|8|@]|D|@]|E|$]]|$1|12|3|13|5|L|7|2E|8|@]|D|@]|E|$]]|$1|14|3|15|5|L|7|2F|8|@]|D|@]|E|$]]|$1|16|3|17|5|6|7|2G|8|@]|D|@]|E|$]]|$1|18|3|19|5|6|7|2H|8|@$9|2I|A|2J|B|C]]|D|@]|E|$]]|$1|1A|3|1B|5|6|7|2K|8|@]|D|@]|E|$]]|$1|1C|3|1D|5|1E|7|2L|8|@$9|2M|A|2N|B|C]|$9|2O|A|2P|B|C]]|D|@]|E|$]]|$1|1F|3|1G|5|1E|7|2Q|8|@]|D|@]|E|$]]|$1|1H|3|1I|5|6|7|2R|8|@$9|2S|A|2T|B|C]]|D|@]|E|$]]|$1|1J|3|17|5|6|7|2U|8|@]|D|@]|E|$]]|$1|1K|3|1L|5|6|7|2V|8|@]|D|@]|E|$]]|$1|1M|3|1N|5|6|7|2W|8|@]|D|@]|E|$]]|$1|1O|3|1P|5|1E|7|2X|8|@]|D|@]|E|$]]|$1|1Q|3|1R|5|1E|7|2Y|8|@]|D|@]|E|$]]|$1|1S|3|-4|5|6|7|2Z|8|@]|D|@]|E|$]]]|1T|$]]

solution: use NCQ to boost the performance. to do so configure your SATA HDD controller to use AHCI.

additional details below:

i had made similar observations when analyzing a particular application. on my quad-core system i compared the following configurations:

<ul>
<li>1 core only: pretty fast</li>
<li>4 cores enabled: much slower! this was quite surprising and also confusing to me.</li>
</ul>

it turned out the application was doing heavy, concurrent HDD access. in case of multiple cores (and hence multiple threads) this would noticeably slow down total execution time.

i did some research and learned that a feature called NCQ (native command queuing) will do the optimization of HDD access you are referring to.

in SCSI world this has been common standard for quite a while. and in SATA world it has been adapted some time back.
to unlock this feature it's required to configure your HDD controller to operate in AHCI mode - this is a prerequisite to use NCQ!

as regular desktop systems nowadays use on-board HDD controllers, this configuration part needs to be done in BIOS setup. for SATA configuration you can usually choose between the following operational modes:

<ul>
<li>compatible / legacy IDE</li>
<li>AHCI</li>
</ul>

i went ahead and implemented my own custom benchmark to compare one and the same system running with the following configurations:

<ul>
<li>4 cores enabled, legacy IDE: pretty slow</li>
<li>4 cores enabled, AHCI / NCQ: much faster. particular benchmark sections performed 6 times faster!</li>
</ul>

--

conclusion:

to unleash the full power of systems with concurrent HDD access:

<ol>
<li>switch over to AHCI (so you can utilize NCQ)</li>
<li>don't use the generic AHCI drivers that come with the OS. instead, use the vendor-specific, optimized drivers. example: windows 7 comes with some generic AHCI drivers that support most of the common HDD controllers. however, when using an intel chipset, make sure to install the intel "matrix storage manager" or intel "rapid storage technology" (e.g. intel RST 11.7). optimized drivers have shown to additionally boost HDD performance.</li>
</ol>

not doing so will make some applications run slower when using multiple threads instead of a single thread. that's the surprising part you need to consider.

--

note: there's a myth out there that says: NCQ is only relevant for "server" environments (with hundreds of processes running in parallel).
my benchmark results are pointing in a different direction: it's also relevant for "desktop" environments. whenever heavy, concurrent HDD access is happening.

additional notes:

<ol>
<li>some older chipsets / SATA HDD controllers do not support AHCI mode. but that's not covered here.</li>
<li>some "old" OS need special actions when either installing in AHCI mode or migrating an already system from IDE mode to AHCI. but that's not covered here.</li>
</ol>

Once I had the theory that on modern operating systems multithreaded 
read access on the HDD should perform better.

I thought that: 
the operating system queues all read requests,
and rearranges them in such a way, that it could read from the HDD more
sequentially. The more requests it would get, the better it could rearrange them
to optimize the read sequence. 
I was very sure that I read it somewhere few times.

But I did some benchmarking, and had to find out, that multithreaded
read access mostly perform much worst, and never performs better.

I had the experience under Windows and Linux. I benchmarked pure
searching of files using the operating system's tools, and also
had written own little benchmarks.

Am I missing something? 
Can someone explain to me the secrets of this topic? 
Thank you!

Directory walker on modern operating systems slower when it's multi-threaded?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我曾经有过这样的理论:在现代操作系统上，多线程对HDD的访问应该表现得更好。我以为：操作系统对所有读取请求进行排队，并以这样的方式重新排列它们，以便能够更多地从HDD读取。请求越多，就越好地重新排列它们以优化读取顺序。我很确定我在某个地方读过几遍。但我做了一些基准测试，并不得不发现，多线程读取访问大多表现得很糟糕，而且...

问现代操作系统上的目录漫游器，当它是多线程的时候会变慢吗？
EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问现代操作系统上的目录漫游器，当它是多线程的时候会变慢吗？EN

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问现代操作系统上的目录漫游器，当它是多线程的时候会变慢吗？
EN