考虑这样一个系统,它需要存储大部分较小的值(1-1000字节),但在某些情况下,它需要存储较大的值(10MB)。它包含一个散列列表(考虑用于EDDSA签名的公钥列表)
在以太坊,存储是通过PATRICIA轮胎完成的,但我不认为这是必要的。有没有人知道用键/存储数据库处理这么大的值的好方法?
发布于 2021-08-15 10:03:15
下面是一些用于存储大型二进制对象的代码(也称为。blobs)和foundationdb:
from uuid import uuid4
from uuid import UUID
from collections import namedtuple
from hashlib import blake2b as hasher
from more_itertools import sliced
import found
class BStoreException(found.BaseFoundException):
pass
BSTORE_SUFFIX_HASH = [b'\x01']
BSTORE_SUFFIX_BLOB = [b'\x02']
BStore = namedtuple('BStore', ('name', 'prefix_hash', 'prefix_blob',))
def make(name, prefix):
prefix = list(prefix)
out = BStore(name, tuple(prefix + BSTORE_SUFFIX_HASH), tuple(prefix + BSTORE_SUFFIX_BLOB))
return out
async def get_or_create(tx, bstore, blob):
hash = hasher(blob).digest()
key = found.pack((bstore.prefix_hash, hash))
maybe_uid = await found.get(tx, key)
if maybe_uid is not None:
return UUID(bytes=maybe_uid)
# Otherwise create the hash entry and store the blob with a new uid
# TODO: Use a counter and implement a garbage collector, and implement
# bstore.delete
uid = uuid4()
found.set(tx, key, uid.bytes)
for index, slice in enumerate(sliced(blob, found.MAX_SIZE_VALUE)):
found.set(tx, found.pack((bstore.prefix_blob, uid, index)), bytes(slice))
return uid
async def get(tx, bstore, uid):
key = found.pack((bstore.prefix_blob, uid))
out = b''
async for _, value in found.query(tx, key, found.next_prefix(key)):
out += value
if out == b'':
raise BStoreException('BLOB should be in database: uid={}'.format(uid))
return out
摘自https://github.com/amirouche/asyncio-foundationdb/blob/main/found/bstore.py
https://stackoverflow.com/questions/52747480
复制相似问题