用Java压缩，用Python -snappy/redis-py-集群解压缩
用Java压缩，用Python -snappy/redis-py-集群解压缩
提问于 2020-02-17 09:47:05
我正在用python为redis集群编写cron脚本，并且只使用红-py-簇从prod服务器读取数据。一个单独的Java应用程序正在使用、snappy压缩和javaStringcodecutf-8写入redis集群。
我能够读取数据，但无法解码。
from rediscluster import RedisCluster
import snappy
host, port ="127.0.0.1", "30001"
startup_nodes = [{"host": host, "port": port}]
print("Trying connecting to redis cluster host=" + host + ", port=" + str(port))
rc = RedisCluster(startup_nodes=startup_nodes, max_connections=32, decode_responses=True)
print("Connected", rc)
print("Reading all keys, value ...\n\n")
for key in rc.scan_iter("uidx:*"):
value = rc.get(key)
#uncompress = snappy.uncompress(value, decoding="utf-8")
print(key, value)
print('\n')
print("Done. exit()")
exit()decode_responses=False对这条评论的处理很好。然而，更改decode_responses=True是抛出错误。我猜它无法得到正确的解码器。
Traceback (most recent call last):
File "splooks_cron.py", line 22, in <module>
print(key, rc.get(key))
File "/Library/Python/2.7/site-packages/redis/client.py", line 1207, in get
return self.execute_command('GET', name)
File "/Library/Python/2.7/site-packages/rediscluster/utils.py", line 101, in inner
return func(*args, **kwargs)
File "/Library/Python/2.7/site-packages/rediscluster/client.py", line 410, in execute_command
return self.parse_response(r, command, **kwargs)
File "/Library/Python/2.7/site-packages/redis/client.py", line 768, in parse_response
response = connection.read_response()
File "/Library/Python/2.7/site-packages/redis/connection.py", line 636, in read_response
raise e
: 'utf8' codec can't decode byte 0x82 in position 0: invalid start bytePS:取消注释这一行uncompress = snappy.uncompress(value, decoding="utf-8")正因错误而中断
Traceback (most recent call last):
File "splooks_cron.py", line 27, in <module>
uncompress = snappy.uncompress(value, decoding="utf-8")
File "/Library/Python/2.7/site-packages/snappy/snappy.py", line 91, in uncompress
return _uncompress(data).decode(decoding)
snappy.UncompressError: Error while decompressing: invalid input Stack Overflow用户
回答已采纳
发布于 2020-02-17 18:09:43
经过几个小时的调试，我终于解决了这个问题。
我在我的Java代码中使用了xerial/snappy-java压缩器，它正在编写redis集群。有趣的是，在压缩过程中， SnappyOutputStream在压缩数据的开头添加了一些偏移量。在我的例子中，这个看起来像这样
"\x82SNAPPY\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x01\xb6\x8b\x06\\******actual data here*****由于这个原因，减压器无法弄清楚。我修改了下面的代码，并从值中删除偏移量。现在很好用。
for key in rc.scan_iter("uidx:*"):
value = rc.get(key)
#in my case offset was 20 and utf-8 is default ecoder/decoder for snappy
# https://github.com/andrix/python-snappy/blob/master/snappy/snappy.py
uncompress_value = snappy.decompress(value[20:])
print(key, uncompress_value)
print('\n')页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/60259701
复制相关文章
相似问题
腾讯云开发者
