文章/答案/技术大牛

发布

社区首页 >问答首页 >python中的泡选方法描述符对象

问python中的泡选方法描述符对象
EN

Stack Overflow用户

提问于 2015-12-06 23:44:06

回答 2查看 2.7K关注 0票数 3

我正在尝试腌制一个method_descriptor。

使用pickle或cloudpickle进行腌制失败：

Python 2.7.10 |Continuum Analytics, Inc.| (default, Oct 19 2015, 18:04:42) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org

>>> import pickle, cloudpickle

>>> pickle.dumps(set.union)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/home/pmd/anaconda3/envs/python2/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: cannot pickle method_descriptor objects

>>> cloudpickle.dumps(set.union)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 602, in dumps
    cp.dump(obj)
  File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 111, in dump
    raise pickle.PicklingError(msg)
pickle.PicklingError: Could not pickle object as excessively deep recursion required.

导入dill以某种方式使pickle工作，如下所示：

>>> import dill
>>> pickle.dumps(set.union)
'cdill.dill\n_getattr\np0\n(c__builtin__\nset\np1\nS\'union\'\np2\nS"<method \'union\' of \'set\' objects>"\np3\ntp4\nRp5\n.'

>>> f = pickle.loads(pickle.dumps(set.union))
>>> set.union(set([1,2]), set([3]))
set([1, 2, 3])
>>> f(set([1,2]), set([3]))
set([1, 2, 3])

cloudpickle中的问题即使在dill导入之后仍然存在：

>>> cloudpickle.dumps(set.union)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 602, in dumps
    cp.dump(obj)
  File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 111, in dump
    raise pickle.PicklingError(msg)
pickle.PicklingError: Could not pickle object as excessively deep recursion required.

在我的应用程序中，我依赖cloudpickle来处理全局函数。因此，我的问题是，如何让cloudpickle在Python2.7中为method_descriptor对象工作？

编辑:我注意到在Python3.3中也出现了同样的问题，但是Python3.5中没有这个问题。

python

pickle

回答 2

Stack Overflow用户

回答已采纳

发布于 2015-12-08 18:32:50

经过了很多周旋之后，我想我已经找到了一个清晰的答案，这个问题适用于Python2.7和3.3。请注意，Python3.5一开始没有问题。

在我展示我发现的结果之前，我想把multiprocessing.forking模块归功于这个模块，我就是在这个模块中得到了代码的jist来完成这项工作的。

在下面的内容中，我将使用set.union作为<class 'method_descriptor'>的一个示例。

Python3.5:方法描述符开箱即用

Python 3.5.0 |Continuum Analytics, Inc.| (default, Oct 19 2015, 21:57:25) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> pickle.dumps(set.union)
b'\x80\x03cbuiltins\ngetattr\nq\x00cbuiltins\nset\nq\x01X\x05\x00\x00\x00unionq\x02\x86q\x03Rq\x04.'
>>> f = pickle.loads(pickle.dumps(set.union))
>>> f({1, 2, 3}, {5})
{1, 2, 3, 5}
>>>

Python3.3:使用copyreg为pickle使用method_descriptor提供一种方法

Python 3.3.5 |Continuum Analytics, Inc.| (default, Jun  4 2015, 15:22:11) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> pickle.dumps(set.union)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_pickle.PicklingError: Can't pickle <class 'method_descriptor'>: attribute lookup builtins.method_descriptor failed

set.union的类型是method_descriptor

>>> type(set.union)
<class 'method_descriptor'>

我们为method_descriptor定义了约简函数，并将其注册到copyreg中。

>>> def _reduce_method_descriptor(m):
...     return getattr, (m.__objclass__, m.__name__)
... 
>>> import copyreg
>>> copyreg.pickle(type(set.union), _reduce_method_descriptor)

成功：

>>> pickle.dumps(set.union)
b'\x80\x03cbuiltins\ngetattr\nq\x00cbuiltins\nset\nq\x01X\x05\x00\x00\x00unionq\x02\x86q\x03Rq\x04.'
>>> f = pickle.loads(pickle.dumps(set.union))
>>> f({1, 2, 3}, {5})
{1, 2, 3, 5}

如果我们现在导入cloudpickle，那么注册的酸洗功能仍然有效：

>>> import cloudpickle
>>> cloudpickle.dumps(set.union)
b'\x80\x02c__builtin__\ngetattr\nq\x00c__builtin__\nset\nq\x01X\x05\x00\x00\x00unionq\x02\x86q\x03Rq\x04.'
>>> f = pickle.loads(pickle.dumps(set.union))
>>> f({1, 2, 3}, {5})
{1, 2, 3, 5}
>>>

Python2.7:使用copy_reg为pickle使用method_descriptor提供一种方法

在Python2.7中，用于注册泡菜支持函数的模块称为copy_reg。

Python 2.7.10 |Continuum Analytics, Inc.| (default, Oct 19 2015, 18:04:42) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import pickle
>>> pickle.dumps(set.union)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/home/pmd/anaconda3/envs/python2/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle method_descriptor objects

set.union的类型是method_descriptor

>>> type(set.union)
<type 'method_descriptor'>

我们为method_descriptor定义了约简函数，并将其注册到copyreg中。

>>> def _reduce_method_descriptor(m):
...     return getattr, (m.__objclass__, m.__name__)
... 
>>> import copy_reg
>>> copy_reg.pickle(type(set.union), _reduce_method_descriptor)
>>> pickle.dumps(set.union)
"c__builtin__\ngetattr\np0\n(c__builtin__\nset\np1\nS'union'\np2\ntp3\nRp4\n."

成功：

>>> f = pickle.loads(pickle.dumps(set.union))
>>> f(set([1, 2, 3]), set([5]))

并与cloudpickle一起工作：

set([1, 2, 3, 5])
>>> import cloudpickle
>>> cloudpickle.dumps(set.union)
'\x80\x02c__builtin__\ngetattr\nq\x00c__builtin__\nset\nq\x01U\x05unionq\x02\x86q\x03Rq\x04.'
>>> f = pickle.loads(cloudpickle.dumps(set.union))
>>> f(set([1, 2, 3]), set([5]))
set([1, 2, 3, 5])
>>>

票数 2

Stack Overflow用户

发布于 2015-12-07 14:21:50

我是dill的作者。当您执行import dill时，它将序列化注册表从dill注入pickle (基本上，将dill中的所有copy_reg-type知识放入pickle注册中心)。

>>> import pickle
>>> pickle.Pickler.dispatch
{<type 'function'>: <function save_global at 0x105d0c7d0>, <type 'dict'>: <function save_dict at 0x105d0c668>, <type 'int'>: <function save_int at 0x105d0c230>, <type 'long'>: <function save_long at 0x105d0c2a8>, <type 'list'>: <function save_list at 0x105d0c578>, <type 'str'>: <function save_string at 0x105d0c398>, <type 'unicode'>: <function save_unicode at 0x105d0c410>, <type 'instance'>: <function save_inst at 0x105d0c758>, <type 'type'>: <function save_global at 0x105d0c7d0>, <type 'NoneType'>: <function save_none at 0x105d0c140>, <type 'bool'>: <function save_bool at 0x105d0c1b8>, <type 'tuple'>: <function save_tuple at 0x105d0c488>, <type 'float'>: <function save_float at 0x105d0c320>, <type 'classobj'>: <function save_global at 0x105d0c7d0>, <type 'builtin_function_or_method'>: <function save_global at 0x105d0c7d0>}
>>> import dill
>>> pickle.Pickler.dispatch
{<class '_pyio.BufferedReader'>: <function save_file at 0x106c8b848>, <class '_pyio.TextIOWrapper'>: <function save_file at 0x106c8b848>, <type 'operator.itemgetter'>: <function save_itemgetter at 0x106c8b578>, <type 'weakproxy'>: <function save_weakproxy at 0x106c8c050>, <type 'NoneType'>: <function save_none at 0x105d0c140>, <type 'str'>: <function save_string at 0x105d0c398>, <type 'file'>: <function save_file at 0x106c8b8c0>, <type 'classmethod'>: <function save_classmethod at 0x106c8c230>, <type 'float'>: <function save_float at 0x105d0c320>, <type 'instancemethod'>: <function save_instancemethod0 at 0x106c8ba28>, <type 'cell'>: <function save_cell at 0x106c8bb18>, <type 'member_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'slice'>: <function save_slice at 0x106c8bc80>, <type 'dict'>: <function save_module_dict at 0x106c8b410>, <type 'long'>: <function save_long at 0x105d0c2a8>, <type 'code'>: <function save_code at 0x106c8b320>, <type 'type'>: <function save_type at 0x106c8c0c8>, <type 'xrange'>: <function save_singleton at 0x106c8bde8>, <type 'builtin_function_or_method'>: <function save_builtin_method at 0x106c8b9b0>, <type 'classobj'>: <function save_classobj at 0x106c8b488>, <type 'weakref'>: <function save_weakref at 0x106c8bed8>, <type 'getset_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'weakcallableproxy'>: <function save_weakproxy at 0x106c8c050>, <class '_pyio.BufferedRandom'>: <function save_file at 0x106c8b848>, <type 'int'>: <function save_int at 0x105d0c230>, <type 'list'>: <function save_list at 0x105d0c578>, <type 'functools.partial'>: <function save_functor at 0x106c8b7d0>, <type 'bool'>: <function save_bool at 0x105d0c1b8>, <type 'function'>: <function save_function at 0x106c8b398>, <type 'thread.lock'>: <function save_lock at 0x106c8b500>, <type 'super'>: <function save_functor at 0x106c8b938>, <type 'staticmethod'>: <function save_classmethod at 0x106c8c230>, <type 'module'>: <function save_module at 0x106c8bf50>, <type 'method_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'operator.attrgetter'>: <function save_attrgetter at 0x106c8b5f0>, <type 'wrapper_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'numpy.ufunc'>: <function save_numpy_ufunc at 0x106c8bcf8>, <type 'method-wrapper'>: <function save_instancemethod at 0x106c8baa0>, <type 'instance'>: <function save_inst at 0x105d0c758>, <type 'cStringIO.StringI'>: <function save_stringi at 0x106c8b6e0>, <type 'unicode'>: <function save_unicode at 0x105d0c410>, <class '_pyio.BufferedWriter'>: <function save_file at 0x106c8b848>, <type 'property'>: <function save_property at 0x106c8c140>, <type 'ellipsis'>: <function save_singleton at 0x106c8bde8>, <type 'tuple'>: <function save_tuple at 0x105d0c488>, <type 'cStringIO.StringO'>: <function save_stringo at 0x106c8b758>, <type 'NotImplementedType'>: <function save_singleton at 0x106c8bde8>, <type 'dictproxy'>: <function save_dictproxy at 0x106c8bb90>}

cloudpickle与dill有(稍微)不同的酸洗函数，如果您正在使用cloudpickle，它会将自己的序列化函数推入pickle注册表。如果您想让cloudpickle为您工作，您可能可以设置一个解决方案…。本质上，在应用程序中安装一个执行import dill as cloudpickle (不错的引用：http://blog.dscpl.com.au/2015/03/safely-applying-monkey-patches-in-python.html)…的模块但是，在您的应用程序上下文中，这将将整个cloudpickle的使用替换为dill。您还可以尝试沿着以下路线创建一个猴子区：

>>> #first import dill, which populates itself into pickle's dispatch
>>> import dill
>>> import pickle
>>> # save the MethodDescriptorType from dill
>>> MethodDescriptorType = type(type.__dict__['mro'])
>>> MethodDescriptorWrapper = pickle.Pickler.dispatch[MethodDescriptorType]
>>> # cloudpickle does the same, so let it update the dispatch table
>>> import cloudpickle
>>> # now, put the saved MethodDescriptorType back in
>>> pickle.Pickler.dispatch[MethodDescriptorWrapperType] = MethodDescriptorWrapper

请注意，如果要直接使用cloudpickle.dumps，则必须在cloudpickle中通过在cloudpickle.CloudPickler.dispatch上执行上述monkeypatch来直接重载注册表。

我不保证它能工作，也不保证它不会破坏cloudpickle中的其他对象(本质上，我还没有尝试过)，但是这是一种用来自dill的错误的cloudpickle包装器来替换的潜在途径。

如果您想得到简短的答案，我会说(至少在这种情况下)使用dill。;)

编辑有关copyreg

以下是dill中的内容

def _getattr(objclass, name, repr_str):
    # hack to grab the reference directly
    try:
        attr = repr_str.split("'")[3]
        return eval(attr+'.__dict__["'+name+'"]')
    except:
        attr = getattr(objclass,name)
        if name == '__dict__':
            attr = attr[name]
        return attr

它用于将一个函数注册为一个较低级别的减缩函数(直接在pickler实例上)。obj是要腌制的对象。

pickler.save_reduce(_getattr, (obj.__objclass__, obj.__name__, obj.__repr__()), obj=obj)

我相信这会转化为一种简化方法(在copyreg.pickle中直接使用)，如下所示：

def _reduce_method_descriptor(obj):
    return _getattr, (obj.__objclass__, obj.__name__, obj.__repr__())

票数 4

页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持

原文链接：

https://stackoverflow.com/questions/34124270

复制

相似问题

问python中的泡选方法描述符对象
EN

回答 2

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问python中的泡选方法描述符对象EN

回答 2

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问python中的泡选方法描述符对象
EN