blocks|key|3252178|text|有人知道如何在|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3252179|中并行化字符集解码操作吗？|blockquote|3252180|3252181|3252182|您可以打开多个输入流来完成此操作(我不确定如何使用NIO完成此操作，但这必须是可能的)。|3252183|这会有多难，取决于你解码的编码。对于目标编码，您需要一个定制的解决方案。如果编码有一个固定的宽度(例如Windows1252)，那么一个字节==一个字符和解码是很容易的。|3252184|现代的可变宽度编码(如UTF-8和UTF-16)包含用于标识字符序列的第一个字节的规则，因此可以跳到文件的中间并开始解码(您必须注意上一个块的结尾，因此首先开始解码文件的结尾是明智的做法)。|3252185|一些传统的可变宽度编码可能没有这么好的设计，因此您将别无选择，只能从数据的开头解码并按顺序读取它。|3252186|如果可以，则将数据生成为UTF-16BE格式。然后你就可以省去解码，直接把两个字节读成一个字符。|3252187|如果文件是Unicode，请注意BOM处理，但我猜您已经熟悉了许多低级细节。|3252188|entityMap^0|0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|V|8|@]|9|@]|A|$]]|$1|E|3|-4|5|6|7|W|8|@]|9|@]|A|$]]|$1|F|3|-4|5|6|7|X|8|@]|9|@]|A|$]]|$1|G|3|H|5|6|7|Y|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|Z|8|@]|9|@]|A|$]]|$1|K|3|L|5|6|7|10|8|@]|9|@]|A|$]]|$1|M|3|N|5|6|7|11|8|@]|9|@]|A|$]]|$1|O|3|P|5|6|7|12|8|@]|9|@]|A|$]]|$1|Q|3|R|5|6|7|13|8|@]|9|@]|A|$]]|$1|S|3|-4|5|6|7|14|8|@]|9|@]|A|$]]]|T|$]]

<blockquote>
 does anyone know how to parallelize the charset decoding operation in Java?
</blockquote>

You might be able to open multiple input streams to do this (I'm not sure how you'd go about this with NIO, but it must be possible).

How difficult this would be depends on the encoding you're decoding from. You will need a bespoke solution for the target encoding. If the encoding has a fixed width (e.g. Windows-1252), then one byte == one character and decoding is easy.

Modern variable-width encodings (like UTF-8 and UTF-16) contain rules for identifying the first byte of a character sequence, so it is possible to jump to the middle of a file and start decoding (you'll have to note the end of the previous chunk, so it is wise to start decoding the end of the file first).

Some legacy variable-width encodings might not be this well designed, so you'll have no option but to decode from the start of the data and read it sequentially.

If it is an option, generate your data as UTF-16BE. Then you can cut out decoding and read two bytes straight to a char.

If the file is Unicode, watch out for BOM handling, but I'm guessing you're already familiar with many of the low-level details.

blocks|key|3252190|text|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3252191|很明显，当前的限制是字符解码(+CharsetDecoder.read(...)+)|blockquote|3252192|3252193|3252194|你怎么知道的?您的监控/分析是否最终显示解码器线程正在使用您的某个核心的100%25？|3252195|另一种可能性是操作系统无法以其理论上的最大速度驱动SSD。|3252196|如果UTF-8解码肯定是瓶颈，那么应该可以并行完成这项任务。但您肯定需要实现自己的解码器才能做到这一点。|3252197|entityMap^0|0|0|0|0|0|0|0^^$0|@$1|2|3|-4|4|5|6|N|7|@]|8|@]|9|$]]|$1|A|3|B|4|C|6|O|7|@]|8|@]|9|$]]|$1|D|3|-4|4|5|6|P|7|@]|8|@]|9|$]]|$1|E|3|-4|4|5|6|Q|7|@]|8|@]|9|$]]|$1|F|3|G|4|5|6|R|7|@]|8|@]|9|$]]|$1|H|3|I|4|5|6|S|7|@]|8|@]|9|$]]|$1|J|3|K|4|5|6|T|7|@]|8|@]|9|$]]|$1|L|3|-4|4|5|6|U|7|@]|8|@]|9|$]]]|M|$]]

<blockquote>
 It is clear that the current limitation is the character decoding ( CharsetDecoder.read(...) )
</blockquote>

How do you know that? Does your monitoring / profiling show conclusively that the decoder thread is using 100% of one of your cores?

Another possibility is that the OS is not capable of driving the SSD at its theoretical maximum speed.

If UTF-8 decoding is definitely the bottleneck then it should be possible to do the task in parallel. But you will certainly need to implement your own decoders to do this.

blocks|key|415088|text|如果您知道编码，并且它是固定大小的，或者不包含重叠的字节序列，那么您可以扫描特殊的序列。在CSV中，换行符的序列可能是有意义的。即使您动态检测编码，您也可以运行前几个字节的传递来确定编码，然后继续并行解码。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|415089|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

If you know the encoding, and it is either fixed size, or does not contain overlapping byte sequences, you could scan for a special sequence. In CSV, a sequence for newlines might make sense. Even if you dynamically detect the encoding, you could run a pass of the first few bytes to determine encoding, and then move on to parallel decoding.

blocks|key|412790|text|另一种(疯狂的)替代方案是将输入分成任意大小的块，忽略解码问题，然后并行解码每个块。但是，您希望确保块重叠(具有参数化的大小)。如果两个线程以相同的方式解码两个块的重叠区域(并且您的重叠对于指定的编码来说足够大)，那么连接结果应该是安全的。重叠越大，需要的处理就越多，出错的概率就越小。此外，如果您知道编码是UTF-8或类似的简单编码，则可以将重叠设置得非常低(对于该客户端)，并且仍然可以保证正确的操作。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|412791|如果第二个块被证明是错误的，你将不得不重做，所以不要并行处理大块是很重要的。如果并行执行两个以上的块，从头到尾进行“修复”将很重要，这样一个未对齐的块就不会导致下一个块(可能正确对齐)无效。|412792|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|G|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|H|8|@]|9|@]|A|$]]]|E|$]]

Another (crazy) alternative would be to just separate the input into chunks of some arbitrary size, ignore the decoding issues and then decode each of the chunks in parallel. However, you want to ensure that the chunks overlap (with a parametrized size). If the overlapping region of the two chunks is decoded the same way by the two threads (and your overlap was big enough for the specified encoding) it should be safe to join the results. The bigger the overlap, the more processing required, and the smaller the probability of error. Furthermore, if you are in a situation where you know the encoding is UTF-8, or a similarly simple encoding, you could set the overlap quite low (for that client) and still be guaranteed correct operation.

If the second chunk turns out to be wrong, you will have to redo it, so it is important to not do to big chunks in parallel. If you do more than two chunks in parallel, it would be important to 'repair' from beginning to end, so that one misaligned block does not result in invalidating the next block (which might be correctly aligned).

I am maintaining a high performance CSV parser and try to get the most out of latest technology to improve the throughput. For this particular tasks this means:

<ul>
<li>Flash memory (We own a relatively inexpensive PCI-Express card, 1 TB of storage that reaches 1 GB/s sustained read performance)</li>
<li>Multiple cores (We own a cheap Nehalem server with 16 hardware threads)</li>
</ul>

The first implementation of the CSV parser was single threaded. File reading, character decoding, field splitting, text parsing, all within the same thread. The result was a throughput of about 50MB/s. Not bad but well below the storage limit...

The second implementation uses one thread to read the file (at the byte level), one thread to decode the characters (from ByteBuffer to CharBuffer), and multiple threads to parse the fields (I mean parsing delimitted text fields into doubles, integers, dates...). This works well faster, close to 400MB/s on our box.

But still well below the performance of our storage. And those SSD will improve again in the future, we are not taking the most out of it in Java. It is clear that the current limitation is the character decoding ( CharsetDecoder.read(...) ). That is the bottleneck, on a powerful Nehalem processor it transforms bytes into chars at 400MB/s, pretty good, but this has to be single threaded. The CharsetDecoder is somewhat stateful, depending on the used charset, and does not support multithreaded decoding.

So my question to the community is (and thank you for reading the post so far): does anyone know how to parallelize the charset decoding operation in Java?

Java: multithreaded character stream decoding

存储性能

服务器

Java

我正在维护一个高性能的CSV解析器，并试图最大限度地利用最新技术来提高吞吐量。对于此特定任务，这意味着：Nehalem闪存(我们拥有相对便宜的PCIExpress卡，1 TB的存储，可达到1 GB/s的持续读取performance)Multiple核心)(我们拥有一台具有16个硬件线程的廉价服务器)CSV解析器的第一...

问Java:多线程字符流解码
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Java:多线程字符流解码EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Java:多线程字符流解码
EN