blocks|key|4028176|text|也许您应该使用数据库，但请查看shelve模块|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|4028177|如果搁置对你来说还不够强大，总会有工业优势的|4028178|entityMap|0|LINK|mutability|MUTABLE|url|http://docs.python.org/library/shelve.html#module-shelve^0|F|6|F|6|0|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@$9|Q|A|R|B|C]]|D|@$9|S|A|T|1|U]]|E|$]]|$1|F|3|G|5|6|7|V|8|@]|D|@]|E|$]]|$1|H|3|-4|5|6|7|W|8|@]|D|@]|E|$]]]|I|$J|$5|K|L|M|E|$N|O]]]]

Maybe you should be using a database, but check out the <a href="http://docs.python.org/library/shelve.html#module-shelve" rel="nofollow noreferrer"><code>shelve</code></a> module

If shelve isn't powerful enough for you, there is always the industrial strength <a href="http://www.zodb.org/" rel="nofollow noreferrer">ZODB</a>

blocks|key|4028201|text|一次又一次地处理整个散列必然会遇到与您现在所面临的相同的内存压力--甚至可能更糟，因为所有的数据都会来回编组。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4028202|相反，使用类似于散列的磁盘数据库可能是最好的选择；有关在程序中使用dbm样式的数据库的快速介绍，请参阅此页面：http://docs.python.org/library/dbm|offset|length|4028203|它们的行为足够像散列，这对你来说应该是一个简单的转换。|4028204|entityMap|0|LINK|mutability|MUTABLE|url|http://docs.python.org/library/dbm^0|0|1J|Y|0|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Q|8|@]|9|@$D|R|E|S|1|T]]|A|$]]|$1|F|3|G|5|6|7|U|8|@]|9|@]|A|$]]|$1|H|3|-4|5|6|7|V|8|@]|9|@]|A|$]]]|I|$J|$5|K|L|M|A|$N|O]]]]

Pickling an entire hash over and over again is bound to run into the same memory pressures that you're facing now -- maybe even worse, with all the data marshaling back and forth.

Instead, using an on-disk database that acts like a hash is probably the best bet; see this page for a quick introduction to using dbm-style databases in your program: <a href="http://docs.python.org/library/dbm" rel="nofollow noreferrer">http://docs.python.org/library/dbm</a>

They act enough like hashes that it should be a simple transition for you.

blocks|key|1571834|text|正如@gnibbler推荐的那样，shelve无疑是我会使用的，但要注意两个陷阱:一个简单的陷阱(所有键都必须是字符串)和一个微妙的陷阱(因为值通常不存在于内存中，所以在它们上调用赋值函数可能不会像你预期的那样工作)。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1571835|对于简单的问题，通常很容易找到解决方法(如果你忘记并尝试，例如使用int或其他作为密钥，你会得到一个明显的异常，所以不难记住，你也需要一个解决方法)。|1571836|对于微妙的问题，请考虑以下示例：|1571837|x+=+d['foo']
x.amutatingmethod()
...much+later...
y+=+d['foo']
#+is+y+"mutated"+or+not+now?|code-block|syntax|javascript|1571838|最后一条评论中问题的答案取决于d是真正的dict+(在这种情况下，y将发生变化，实际上与x是同一个对象)还是shelf+(在这种情况下，y将是与x截然不同的对象，并且处于您上次将保存到d['foo']！的完全相同的状态)。|BOLD|1571839|为了让您的突变持久存在，您需要通过执行以下操作来“将它们保存到磁盘”|1571840|d['foo']+=+x|1571841|在x上调用任何你想要的赋值器之后(所以你特别不能这样做|1571842|d['foo'].mutator()|1571843|并期望突变“坚持”，就像如果d是一个字典一样)。|1571844|shelve确实可以选择在内存中缓存所有获取的项，但这当然会再次填满内存，并在最终关闭shelf对象时导致长时间的延迟(因为所有缓存的项都必须保存回磁盘，以防它们发生变化)。这个选项是我最初推动的(作为Python核心提交者)，但后来我改变了主意，我现在为使用它而道歉(啊，好吧，至少它不是默认的！-)，因为它应该用于的情况很少，而且它经常会诱使粗心大意的用户……抱歉的。|1571845|顺便说一句，如果你不知道一个赋值函数或“变异方法”是什么，它是任何改变你调用它的对象的状态的方法--例如，如果对象是一个列表，就是.append，如果对象是任何类型的容器，就是.pop，等等。当然，不必担心对象是否是不可变的(数字、字符串、元组、frozensets等)，因为在这种情况下它没有可变的方法；-)。|1571846|entityMap^0|H|6|0|X|3|0|0|0|F|1|X|1|18|1|1I|5|1W|1|20|1|2K|8|2G|1|0|0|0|1|1|0|0|E|1|0|0|6|17|5|0|1T|7|2G|4|0^^$0|@$1|2|3|4|5|6|7|17|8|@$9|18|A|19|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|1A|8|@$9|1B|A|1C|B|C]]|D|@]|E|$]]|$1|H|3|I|5|6|7|1D|8|@]|D|@]|E|$]]|$1|J|3|K|5|L|7|1E|8|@]|D|@]|E|$M|N]]|$1|O|3|P|5|6|7|1F|8|@$9|1G|A|1H|B|C]|$9|1I|A|1J|B|C]|$9|1K|A|1L|B|C]|$9|1M|A|1N|B|C]|$9|1O|A|1P|B|C]|$9|1Q|A|1R|B|C]|$9|1S|A|1T|B|C]|$9|1U|A|1V|B|Q]]|D|@]|E|$]]|$1|R|3|S|5|6|7|1W|8|@]|D|@]|E|$]]|$1|T|3|U|5|L|7|1X|8|@]|D|@]|E|$M|N]]|$1|V|3|W|5|6|7|1Y|8|@$9|1Z|A|20|B|C]]|D|@]|E|$]]|$1|X|3|Y|5|L|7|21|8|@]|D|@]|E|$M|N]]|$1|Z|3|10|5|6|7|22|8|@$9|23|A|24|B|C]]|D|@]|E|$]]|$1|11|3|12|5|6|7|25|8|@$9|26|A|27|B|C]|$9|28|A|29|B|C]]|D|@]|E|$]]|$1|13|3|14|5|6|7|2A|8|@$9|2B|A|2C|B|C]|$9|2D|A|2E|B|C]]|D|@]|E|$]]|$1|15|3|-4|5|6|7|2F|8|@]|D|@]|E|$]]]|16|$]]

<code>shelve</code>, as @gnibbler recommends, is what I would no doubt be using, but watch out for two traps: a simple one (all keys must be strings) and a subtle one (as the values don't normally exist in memory, calling mutators on them may not work as you expect).

For the simple problem, it's normally easy to find a workaround (and you do get a clear exception if you forget and try e.g. using an <code>int</code> or whatever as the key, so it's not hard t remember that you do need a workaround either).

For the subtle problem, consider for example:

<pre><code>x = d['foo']
x.amutatingmethod()
...much later...
y = d['foo']
# is y "mutated" or not now?
</code></pre>

the answer to the question in the last comment depends on whether <code>d</code> is a real dict (in which case <code>y</code> will be mutated, and in fact exactly the same object as <code>x</code>) or a <code>shelf</code> (in which case <code>y</code> will be a distinct object from <code>x</code>, and in exactly the state you last saved to <code>d['foo']</code>!).

To get your mutations to persist, you need to "save them to disk" by doing

<pre><code>d['foo'] = x
</code></pre>

after calling any mutators you want on <code>x</code> (so in particular you cannot just do

<pre><code>d['foo'].mutator()
</code></pre>

and expect the mutation to "stick", as you would if <code>d</code> were a dict).

<code>shelve</code> does have an option to cache all fetched items in memory, but of course that can fill up the memory again, and result in long delays when you finally close the <code>shelf</code> object (since all the cached items must be saved back to disk then, just in case they had been mutated). That option was something I originally pushed for (as a Python core committer), but I've since changed my mind and I now apologize for getting it in (ah well, at least it's not the default!-), since the cases it should be used in are rare, and it can often trap the unwary user... sorry.

BTW, in case you don't know what a mutator, or "mutating method", is, it's any method that alters the state of the object you call it on -- e.g. <code>.append</code> if the object is a list, <code>.pop</code> if the object is any kind of container, and so on. No need to worry if the object is immutable, of course (numbers, strings, tuples, frozensets, ...), since it doesn't have mutating methods in that case;-).

blocks|key|829181|text|“我有一个大约6000个实例化类的字典，其中每个类都有1000个属性变量，所有类型都是字符串或字符串列表”...我假设你的意思是：“我有一个类，它有大约1000个属性，所有属性的类型都是str或str的list。我有一个字典，将大约6000个未指定类型的键映射到该类的相应实例。”如果这不是一个合理的翻译，请更正它。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|829182|首先，一个类中有1000个属性是令人难以置信的。您必须一般地使用value+=+getattr(obj,+attr_name)和setattr(obj,+attr_name,+value)来处理绝大多数问题。考虑使用字典而不是实例：value+=+obj[attr_name]和obj[attr_name]+=+value。|829183|其次，这600万个属性中有多大百分比是""？如果足够高，您可能会考虑使用__missing__钩子--+docs+here实现一个稀疏字典，该字典在物理上没有这些属性的条目。|829184|entityMap|0|LINK|mutability|MUTABLE|url|http://docs.python.org/library/stdtypes.html#mapping-types-dict^0|2L|3|2P|3|2T|4|0|W|V|1S|U|38|M|3V|M|0|10|B|1G|9|0|0^^$0|@$1|2|3|4|5|6|7|R|8|@$9|S|A|T|B|C]|$9|U|A|V|B|C]|$9|W|A|X|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|Y|8|@$9|Z|A|10|B|C]|$9|11|A|12|B|C]|$9|13|A|14|B|C]|$9|15|A|16|B|C]]|D|@]|E|$]]|$1|H|3|I|5|6|7|17|8|@$9|18|A|19|B|C]]|D|@$9|1A|A|1B|1|1C]]|E|$]]|$1|J|3|-4|5|6|7|1D|8|@]|D|@]|E|$]]]|K|$L|$5|M|N|O|E|$P|Q]]]]

"""I have a dictionary of about 6000 instantiated classes, where each class has 1000 attributed variables all of type string or list of strings""" ... I presume that you mean: """I have a class with about 1000 attributes all of type <code>str</code> or <code>list</code> of <code>str</code>. I have a dictionary mapping about 6000 keys of unspecified type to corresponding instances of that class.""" If that's not a reasonable translation, please correct it.

For a start, 1000 attributes in a class is mindboggling. You must be treating the vast majority generically using <code>value = getattr(obj, attr_name)</code> and <code>setattr(obj, attr_name, value)</code>. Consider using a dict instead of an instance: <code>value = obj[attr_name]</code> and <code>obj[attr_name] = value</code>.

Secondly, what percentage of those 6 million attributes are ""? If sufficiently high, you might like to consider implementing a sparse dict which doesn't physically have entries for those attributes, using the <code>__missing__</code> hook -- <a href="http://docs.python.org/library/stdtypes.html#mapping-types-dict" rel="nofollow noreferrer">docs here</a>.

In my case, I have a dictionary of about 6000 instantiated classes, where each class has 1000 attributed variables all of type string or list of strings. As I build this dictionary up, my RAM goes up super high. Is there a way to write the dictionary as it is being built to the harddrive rather than the RAM so that I can save some memory? I've heard of something called "pickle" but I don't know if this is a feasible method for what I am doing.

Thanks for your help!

I need to free up RAM by storing a Python dictionary on the hard drive, not in RAM. Is it possible?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

在我的例子中，我有一个包含大约6000个实例化类的字典，其中每个类都有1000个属性变量，类型都是字符串或字符串列表。当我建立这本字典的时候，我的RAM增长得非常快。有没有办法编写字典，因为它是建立在硬盘驱动器而不是RAM中，以便我可以节省一些内存？我听说过一种叫“泡菜”的东西，但我不知道这对我正在做的事情是不是一个可...

问我需要通过在硬盘上存储Python字典来释放RAM，而不是在RAM中。有可能吗？
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问我需要通过在硬盘上存储Python字典来释放RAM，而不是在RAM中。有可能吗？EN