blocks|key|2526084|text|我发现使用远程api来做这类事情是最容易的，否则你将不得不将数据存储在blobstore中，然后在完成后导出它。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2526085|远程api不如在appengine上运行它快，但它肯定要容易得多。|2526086|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|G|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|H|8|@]|9|@]|A|$]]]|E|$]]

I find it easiest to do this sort of thing using the remoting api, otherwise you're going to have to store the data in the blobstore and then export it when you're done.

The remoting api isn't as fast as running it on appengine itself, but it's certainly a lot easier.

blocks|key|1455438|text|使用mapreduce+API：https://developers.google.com/appengine/docs/python/dataprocessing/。它还有一个BlobstoreOutputWriter，您可以使用它来创建一个blob，然后下载该blob。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1455439|根据戴夫的建议，下面是一个示例：http://code.google.com/p/appengine-mapreduce/source/browse/trunk/python/demo/main.py#264|1455440|entityMap|0|LINK|mutability|MUTABLE|url|https://developers.google.com/appengine/docs/python/dataprocessing/|1|http://code.google.com/p/appengine-mapreduce/source/browse/trunk/python/demo/main.py#264^0|G|1V|0|0|G|2G|1|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@$A|Q|B|R|1|S]]|C|$]]|$1|D|3|E|5|6|7|T|8|@]|9|@$A|U|B|V|1|W]]|C|$]]|$1|F|3|-4|5|6|7|X|8|@]|9|@]|C|$]]]|G|$H|$5|I|J|K|C|$L|M]]|N|$5|I|J|K|C|$L|O]]]]

Use the mapreduce API: <a href="https://developers.google.com/appengine/docs/python/dataprocessing/" rel="nofollow">https://developers.google.com/appengine/docs/python/dataprocessing/</a>. It also has a BlobstoreOutputWriter which you can use to create a blob and then download that blob.

As per suggestion by Dave, here is an example: <a href="http://code.google.com/p/appengine-mapreduce/source/browse/trunk/python/demo/main.py#264" rel="nofollow">http://code.google.com/p/appengine-mapreduce/source/browse/trunk/python/demo/main.py#264</a>

blocks|key|1454697|text|我会用一种不同的方式来做这件事--如果这里有什么弱点，请告诉我。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1454698|我将使用任务队列和游标。对前1000个结果执行查询，使用实验性的blobstore编程写入API将数据输出到blobstore文件。然后，使用游标重新调度self，并在每次后续迭代中将其附加到文件中，并在游标处选取查询，直到完成为止。|1454699|这可能会很慢--但它不会对一个正在运行的应用程序产生明显的影响，而且与mapreduce不同的是，它不会产生大量的实例，而且可能会花费你的实际成本。它甚至可能不会产生单个额外实例。|1454700|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|H|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|I|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|J|8|@]|9|@]|A|$]]|$1|F|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|G|$]]

I would do this a different way - and somebody please tell me if there's a weakness here.

I would use a task queue and a cursor. Do your query for the first 1000 results or so, output data to a blobstore file using the experimental blobstore programmatic write API. Then, reschedule self with the cursor and keep appending to the file with each subsequent iteration and picking up the query at the cursor, until you're done.

This might be slow - but it won't noticeably effect a running app, and unlike mapreduce it won't spawn a gazillion instances and potentially cost you actual money. It probably won't even result in a single additional instance spawn.

blocks|key|1454710|text|如果您只需要导出到一个文件，并且需要所有实体，则可以使用appengine批量加载器|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1454711|请参阅appcfg.py+download_data|1454712|也就是https://developers.google.com/appengine/docs/python/tools/uploadingdata|offset|length|1454713|它可以处理重试、节流、线程等|1454714|entityMap|0|LINK|mutability|MUTABLE|url|https://developers.google.com/appengine/docs/python/tools/uploadingdata^0|0|0|3|1Z|0|0|0^^$0|@$1|2|3|4|5|6|7|R|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|S|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|T|8|@]|9|@$F|U|G|V|1|W]]|A|$]]|$1|H|3|I|5|6|7|X|8|@]|9|@]|A|$]]|$1|J|3|-4|5|6|7|Y|8|@]|9|@]|A|$]]]|K|$L|$5|M|N|O|A|$P|Q]]]]

if you only need to export to a file and you want all the entities you can use the appengine bulk loader

see appcfg.py download_data

also
<a href="https://developers.google.com/appengine/docs/python/tools/uploadingdata" rel="nofollow">https://developers.google.com/appengine/docs/python/tools/uploadingdata</a>

it handles retries throttling threading etc

blocks|key|1455512|text|通过数据存储备份导出到Google云存储，然后下载：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1455513|http://gbayer.com/big-data/app-engine-datastore-how-to-efficiently-export-your-data/|offset|length|1455514|这看起来比其他方法快得多。我自己还没试过。|1455515|entityMap|0|LINK|mutability|MUTABLE|url^0|0|0|2C|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|P|8|@]|9|@$D|Q|E|R|1|S]]|A|$]]|$1|F|3|G|5|6|7|T|8|@]|9|@]|A|$]]|$1|H|3|-4|5|6|7|U|8|@]|9|@]|A|$]]]|I|$J|$5|K|L|M|A|$N|C]]]]

Export via Datastore Backups to Google Cloud Storage and then download:

<a href="http://gbayer.com/big-data/app-engine-datastore-how-to-efficiently-export-your-data/" rel="nofollow">http://gbayer.com/big-data/app-engine-datastore-how-to-efficiently-export-your-data/</a>

This looks MUCH faster than the other methods. I haven't tried it myself.

I have a model in Google App Engine that has 50,000+ entities. I would like to create a mapreduce or other operation to iterate over all 50,000+ entities and export the results of a method on the model to a text file. Then once I'm done, I want to download the text file.

What is the easiest way to do this in Google App Engine? I just need to iterate though all the entities and write out the results of expert_data() to a common file. 

<pre><code>#Example model
class Car(db.Model):
 color = db.StringProperty()

 def export_data(self):
 return self.color
</code></pre>

What is the simplest method to iterate over 20,000 entities in Google App Engine and export to a text file?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我在Google App Engine中有一个模型，它有50,000+实体。我想创建一个mapreduce或其他操作来迭代所有50,000+实体，并将模型上方法的结果导出到一个文本文件中。完成后，我想下载文本文件。在Google App Engine中，最简单的方法是什么？我只需要遍历所有实体，并将expert_dat...

问在Google App Engine中迭代超过20,000个实体并导出到文本文件的最简单方法是什么？
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问在Google App Engine中迭代超过20,000个实体并导出到文本文件的最简单方法是什么？EN