在Django中流式传输CSV文件
在Django中流式传输CSV文件
提问于 2011-03-01 03:22:11
我正在尝试将csv文件作为附件下载进行流式传输。CSV文件的大小已经达到4MB或更大，我需要一种让用户主动下载文件的方法，而无需等待所有数据都被创建并提交到内存中。
我首先使用了自己的基于Django的FileWrapper类的文件包装器。但失败了。然后，我在这里看到了一个使用生成器流式传输响应的方法：How to stream an HttpResponse with Django
当我在生成器中引发错误时，我可以看到我使用get_row_data()函数创建了正确的数据，但是当我试图返回响应时，它返回为空。我还禁用了Django GZipMiddleware。有人知道我做错了什么吗？
编辑:我遇到的问题是 ConditionalGetMiddleware**.我不得不替换它，代码在下面的答案中。**
视图如下所示：
from django.views.decorators.http import condition
@condition(etag_func=None)
def csv_view(request, app_label, model_name):
""" Based on the filters in the query, return a csv file for the given model """
#Get the model
model = models.get_model(app_label, model_name)
#if there are filters in the query
if request.method == 'GET':
#if the query is not empty
if request.META['QUERY_STRING'] != None:
keyword_arg_dict = {}
for key, value in request.GET.items():
#get the query filters
keyword_arg_dict[str(key)] = str(value)
#generate a list of row objects, based on the filters
objects_list = model.objects.filter(**keyword_arg_dict)
else:
#get all the model's objects
objects_list = model.objects.all()
else:
#get all the model's objects
objects_list = model.objects.all()
#create the reponse object with a csv mimetype
response = HttpResponse(
stream_response_generator(model, objects_list),
mimetype='text/plain',
)
response['Content-Disposition'] = "attachment; filename=foo.csv"
return response下面是我用来流式传输响应的生成器：
def stream_response_generator(model, objects_list):
"""Streaming function to return data iteratively """
for row_item in objects_list:
yield get_row_data(model, row_item)
time.sleep(1)下面是我创建csv行数据的方法：
def get_row_data(model, row):
"""Get a row of csv data from an object"""
#Create a temporary csv handle
csv_handle = cStringIO.StringIO()
#create the csv output object
csv_output = csv.writer(csv_handle)
value_list = []
for field in model._meta.fields:
#if the field is a related field (ForeignKey, ManyToMany, OneToOne)
if isinstance(field, RelatedField):
#get the related model from the field object
related_model = field.rel.to
for key in row.__dict__.keys():
#find the field in the row that matches the related field
if key.startswith(field.name):
#Get the unicode version of the row in the related model, based on the id
try:
entry = related_model.objects.get(
id__exact=int(row.__dict__[key]),
)
except:
pass
else:
value = entry.__unicode__().encode("utf-8")
break
#if it isn't a related field
else:
#get the value of the field
if isinstance(row.__dict__[field.name], basestring):
value = row.__dict__[field.name].encode("utf-8")
else:
value = row.__dict__[field.name]
value_list.append(value)
#add the row of csv values to the csv file
csv_output.writerow(value_list)
#Return the string value of the csv output
return csv_handle.getvalue()回答 3
Stack Overflow用户
回答已采纳
发布于 2011-03-01 04:53:59
下面是一些用于流式传输CSV的简单代码；您可能可以从这里开始执行所需的任何操作：
import cStringIO as StringIO
import csv
def csv(request):
def data():
for i in xrange(10):
csvfile = StringIO.StringIO()
csvwriter = csv.writer(csvfile)
csvwriter.writerow([i,"a","b","c"])
yield csvfile.getvalue()
response = HttpResponse(data(), mimetype="text/csv")
response["Content-Disposition"] = "attachment; filename=test.csv"
return response这只是将每一行写入内存中的文件，读取该行并生成它。
此版本在生成批量数据时效率更高，但在使用它之前，请务必了解以上内容：
import cStringIO as StringIO
import csv
def csv(request):
csvfile = StringIO.StringIO()
csvwriter = csv.writer(csvfile)
def read_and_flush():
csvfile.seek(0)
data = csvfile.read()
csvfile.seek(0)
csvfile.truncate()
return data
def data():
for i in xrange(10):
csvwriter.writerow([i,"a","b","c"])
data = read_and_flush()
yield data
response = HttpResponse(data(), mimetype="text/csv")
response["Content-Disposition"] = "attachment; filename=test.csv"
return responseStack Overflow用户
发布于 2013-09-27 13:41:01
从Django1.5开始，中间件问题已经得到了解决，并引入了StreamingHttpResponse。执行以下操作：
import cStringIO as StringIO
import csv
def csv_view(request):
...
# Assume `rows` is an iterator or lists
def stream():
buffer_ = StringIO.StringIO()
writer = csv.writer(buffer_)
for row in rows:
writer.writerow(row)
buffer_.seek(0)
data = buffer_.read()
buffer_.seek(0)
buffer_.truncate()
yield data
response = StreamingHttpResponse(
stream(), content_type='text/csv'
)
disposition = "attachment; filename=file.csv"
response['Content-Disposition'] = disposition
return response有一些关于how to output csv from Django的文档，但它没有利用StreamingHttpResponse的优势，所以我继续使用opened a ticket in order to track it。
Stack Overflow用户
发布于 2011-03-03 02:35:03
我遇到的问题是ConditionalGetMiddleware。我看到django-piston为ConditionalGetMiddleware开发了一个支持流媒体的替代中间件：
from django.middleware.http import ConditionalGetMiddleware
def compat_middleware_factory(klass):
"""
Class wrapper that only executes `process_response`
if `streaming` is not set on the `HttpResponse` object.
Django has a bad habbit of looking at the content,
which will prematurely exhaust the data source if we're
using generators or buffers.
"""
class compatwrapper(klass):
def process_response(self, req, resp):
if not hasattr(resp, 'streaming'):
return klass.process_response(self, req, resp)
return resp
return compatwrapper
ConditionalMiddlewareCompatProxy = compat_middleware_factory(ConditionalGetMiddleware)因此，您将用您的ConditionalMiddlewareCompatProxy中间件替换ConditionalGetMiddleware，并且在您的观点中(借用了这个问题的巧妙答案中的代码)：
def csv_view(request):
def data():
for i in xrange(10):
csvfile = StringIO.StringIO()
csvwriter = csv.writer(csvfile)
csvwriter.writerow([i,"a","b","c"])
yield csvfile.getvalue()
#create the reponse object with a csv mimetype
response = HttpResponse(
data(),
mimetype='text/csv',
)
#Set the response as an attachment with a filename
response['Content-Disposition'] = "attachment; filename=test.csv"
response.streaming = True
return response页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/5146539
复制相关文章
相似问题
腾讯云开发者
