blocks|key|2287375|text|更新:我刚刚看到你改变了你的问题，以澄清你关心字符长度而不是字节长度。这是一个非常奇怪的约束。我以前从来没听说过。我不知道该怎么理解它。但如果这是您的需求，并且您想要可预测的阻塞行为，那么我认为您的问题非常简单。只需选择可以表示最可能的唯一字符的兼容字符编码，然后在该字符集上映射二进制块，以便每个块都是最长的，并且包含的位数少于字符编码中可表示的字符数。然后，每个这样的块变成单个字符。由于这个约束有点奇怪，我不知道是否有库可以做到这一点。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2287376|UPDATE2:出于对上面提到的内容的好奇，我在谷歌上找到了这个：https://qntm.org/unicodings。如果您的工具和通信通道可以处理UFT-16或UTF-32，那么在寻求使用UTF-16或UTF-32时可能会有所收获。如果是这样的话，我希望这篇文章能帮助你找到你想要的解决方案。我认为这篇文章仍然在优化字节长度和字符长度，所以这可能不会提供最佳的解决方案，但它只能提供帮助(每个字符32位，而不是7位或8位)。我找不到任何仅在字符数上寻求优化的方法，但也许像Base65536这样的UTF32方案是您的答案。查看https://github.com/qntm/base65536。|offset|length|2287377|如果您关心的是字节长度，并且您想坚持使用通常所指的“可打印字符”或“纯可打印文本”，那么这是我最初的答案……|2287378|有一些选项可以从Base85以外的编码中获得更好的“可读文本”编码空间效率。也有理由放弃更多的空间效率，转而使用Base64。在这里，我将说明同时使用Base85和Base64。如果你可以使用Base85，你的二进制文件的膨胀只会有25%25的损失，这样你就省去了很多麻烦。|2287379|如果您试图将任意二进制编码为“纯文本”，那么Base85非常接近您所能做的最好的结果；如果您想要一种“纯文本”编码，并且可以在逻辑上将其分成有意义的、可预测的块，那么它就是您所能做的最好的选择。从理论上讲，您可以使用使用高ASCII范围内可打印字符的字符集，但是经验表明，如果许多工具和通信通道不能处理普通二进制，那么它们就不能很好地处理高ASCII。尝试使用额外的每4个二进制字节5位，或者使用256位high-ASCII与128位ASCII可能使用的额外空间，并不能节省太多空间。|2287380|对于任何BaseXX编码，该算法都会接收传入的二进制位，并使用它所拥有的XX个可打印字符对它们进行尽可能紧密的编码。Base85将比Base64更紧凑，因为它使用了比Base64+(64个字符)更多的可打印字符(85)。|2287381|在标准ASCII中有95个可打印字符。所以有一个Base95，它是使用所有可打印字符的最紧凑的编码。但是，尝试使用所有95位是混乱的，因为这会导致传入位的不均匀阻塞。每4个二进制字节被映射到小于5的某个分数个字符。|2287382|结果表明，将4个字节编码为恰好5个可打印字符需要85个字符。许多人会选择增加大约10%25的额外长度，以实现每4个编码字节恰好导致5个ASCII字符的事实。这只是二进制大小的25%25的膨胀。这一点都不坏，因为它省去了所有令人头疼的问题。因此，这就是Base85背后的动机。|2287383|Base64用于生成更长但问题更少的编码。不使用对各种文本文档造成麻烦的字符，如HTML、XML、JSON等。通过这种方式，Base64在几乎任何上下文中都很有用，而且没有任何转义。您必须更加小心地使用Base85，因为它不会抛出任何这些有问题的字符。为了提高编码/解码效率，它使用范围33+(“！”)通过117+(‘u’)，从33开始，而不是32，只是为了避免经常有问题的空格字符。“u”上面不使用的字符并没有什么特别之处。|2287384|这就是二进制->+ASCII码编码方面的故事。另一个问题是，在将其二进制表示编码为ASCII之前，您可以做些什么来减小所表示内容的大小。您选择使用pickle.dumps()和zlib.compress()。如果这些是你最好的选择留到下一次讨论...|style|CODE|2287385|entityMap|0|LINK|mutability|MUTABLE|url|https://qntm.org/unicodings|1|https://github.com/qntm/base65536^0|0|X|R|0|7E|X|1|0|0|0|0|0|0|0|0|21|E|2G|F|0^^$0|@$1|2|3|4|5|6|7|17|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|18|8|@]|9|@$D|19|E|1A|1|1B]|$D|1C|E|1D|1|1E]]|A|$]]|$1|F|3|G|5|6|7|1F|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|1G|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|1H|8|@]|9|@]|A|$]]|$1|L|3|M|5|6|7|1I|8|@]|9|@]|A|$]]|$1|N|3|O|5|6|7|1J|8|@]|9|@]|A|$]]|$1|P|3|Q|5|6|7|1K|8|@]|9|@]|A|$]]|$1|R|3|S|5|6|7|1L|8|@]|9|@]|A|$]]|$1|T|3|U|5|6|7|1M|8|@$D|1N|E|1O|V|W]|$D|1P|E|1Q|V|W]]|9|@]|A|$]]|$1|X|3|-4|5|6|7|1R|8|@]|9|@]|A|$]]]|Y|$Z|$5|10|11|12|A|$13|14]]|15|$5|10|11|12|A|$13|16]]]]

UPDATE: I just saw that you changed your question to clarify that you care about character length rather than byte length. This is a really strange constraint. I've never heard of it before. I don't know quite what to make of it. But if that's your need, and you want predicable blocking behavior, then I'm thinking that your problem is pretty simple. Just pick the compatible character encoding that can represent the most possible unique characters, and then map blocks of your binary across that character set such that each block is the longest it can be and yet consists of fewer bits than the number of representable characters in your character encoding. Each such block then becomes a single character. Since this constraint is kinda strange, I don't know if there are libraries out there that do this.

UPDATE2: Being curious about the above myself, I just Google'd and found this: <a href="https://qntm.org/unicodings" rel="noreferrer">https://qntm.org/unicodings</a>. If your tools and communication channels can deal with UFT-16 or UTF-32, then you might be onto something in seeking to use that. If so, I hope this article opens up to the solution you're looking for. I think this article is still optimizing for byte length vs character length, so maybe this won't provide the optimal solution, but it can only help (32 potential bits per char rather than 7 or 8). I couldn't find anything seeking to optimize on character count alone, but maybe a UTF-32 scheme like Base65536 is your answer. Check out <a href="https://github.com/qntm/base65536" rel="noreferrer">https://github.com/qntm/base65536</a> .

If it is byte length that you cared about, and you want to stick to using what is usually meant by "printable characters" or "plain printable text", then here's my original answer...

There are options for getting better "readable text" encoding space efficiency from an encoding other than Base85. There's also a case to be made for giving up more space efficiency and going with Base64. Here I'll make the case for using both Base85 and Base64. If you can use Base85, you only take a 25% hit on the inflation of your binary, and you save a whole lot of headaches in doing so.

Base85 is pretty close to the best you're going to do if you seek to encode arbitrary binary to "plain text", and it is the BEST you can do if you want a "plain text" encoding that you can logically break into meaningful, predictable chunks. You can in theory use a character set that uses printable characters in the high-ASCII range, but experience has shown that many tools and communication channels don't deal well with high-ASCII if they can't handle straight binary. You don't get much in additional space savings for trying to use the extra 5 bits per 4 binary bytes or so that could potentially be used by using 256-bit high-ASCII vs 128-bit ASCII. 

For any BaseXX encoding, the algorithm takes incoming binary bits and encodes them as tightly as it can using the XX printable characters it has at its disposal. Base85 will be more compact than Base64 because it uses more of the printable characters (85) than Base64 does (64 characters).

There are 95 printable characters in standard ASCII. So there is a Base95 that is the most compact encoding possible using all the printable characters. But to try to use all 95 bits is messy, because it leads to uneven blockings of the incoming bits. Each 4 binary bytes is mapped to some fractional number of characters less than 5.

It turns out that 85 characters is what you need to encode 4 bytes as exactly 5 printable characters. Many will choose to add about 10% of extra length to attain the fact that every 4 encoded bytes leads to exactly 5 ASCII characters. This is only a 25% inflation in size of the binary. That's not bad at all for all of the headaches it saves. Hence, the motivation behind Base85.

Base64 is used to produce longer, but even less problematic encodings. Characters that cause trouble for various text documents, like HTML, XML, JSON, etc, are not used. In this way, Base64 is useful in almost any context without any escaping. You have to be more careful with Base85, as it doesn't throw out any of these problematic characters. For encoding/decoding efficiency, it uses the range 33 (“!”) through 117 (‘u’), starting at 33 rather than 32 just to avoid the often problematic space character. The characters above 'u' that it doesn't use are nothing special.

So that's the story pretty much on binary -> ASCII encoding side. The other question is what you can do to reduce the size of what you're representing prior to the stage of encoding its binary representation to ASCII. You're choosing to use <code>pickle.dumps()</code> and <code>zlib.compress()</code>. If those are your best choices are left for another discussion...

I have data (mostly a series of numpy arrays) that I want to convert into text that can be copied/pasted/emailed etc.. I created the following formula which does this.

<pre><code>def convert_to_ascii85(x):
 p = pickle.dumps(x)
 p = zlib.compress(p)
 return b64.b85encode(p)
</code></pre>

My issue is that the string it produces is longer than it needs to be because it only uses a subset of letters, numbers, and symbols. If I was able to encode using unicode, I feel like it could produce a shorter string because it would have access to more characters. Is there a way to do this?

Edit to clarify:
My goal is NOT the smallest amount of data/information/bytes. My goal is the smallest number of characters. The reason is that the channel I'm sending the data through is capped by characters (100k to be precise) instead of bytes (strange, I know). I've already tested that I can send 100k unicode characters, I just don't know how to convert my bytes into unicode.

Compress data into smallest amount of text?

JSON

我有一些数据(主要是一系列的numpy数组)想要转换成可以复制/粘贴/通过电子邮件发送的文本。我创建了下面的公式来实现这一点。def convert_to_ascii85(x):    p = pickle.dumps(x)    p = zlib.compress(p)    return b64.b85encode...

问是否将数据压缩为最小数量的文本？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问是否将数据压缩为最小数量的文本？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问是否将数据压缩为最小数量的文本？
EN