blocks|key|2352418|text|在这种情况下，不要使用iterparse。iterparse()方法只能发出元素的开始和结束事件，因此当找到结束标记时，元素中的所有文本都会被提供给您。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2352419|相反，请使用SAX+parser+interface。这是XML解析库的通用标准，用于将解析的数据传递给内容处理程序。以块的形式向ContentHandler.characters()+callback传递字符数据(假设实现XML库实际上利用了这种可能性)。这是一个来自Python的低级API，并且ElementTree标准库已经捆绑了Expat解析器来驱动它。|2352420|因此流程就变成了：|2352421|2352422|将传入的请求流包装在GzipFile中以便于解压缩。或者，更好的方法是设置response.raw.decode_content+=+True，并根据服务器设置的内容编码将解压缩留给请求库。|unordered-list-item|2352423|将GzipFile实例或原始流传递给使用xml.sax.make_parser()创建的解析器的.parse()+method。然后，解析器继续以块的形式从流中读取数据。通过使用XML语言，您首先可以启用诸如命名空间处理之类的功能(这样可以确保在make_parser()决定更改用于每个namespace).|2352424|The内容处理程序的短前缀时，代码不会中断。使用make_parser()数据块调用characters()方法；检查元素启动事件是否正确，以便知道何时需要base64数据。您可以一次在chunks+of+(a+multiple+of)+4+characters中解码该base64数据，并将其写入文件。我不会在这里使用base64io，只需自己进行分块即可。|2352425|2352426|简单的内容处理程序可以是：|2352427|from+xml.sax+import+handler
from+base64+import+b64decode

class+AttachmentContentHandler(handler.ContentHandler):
++++types_ns+=+'http://schemas.microsoft.com/exchange/services/2006/types'

++++def+__init__(self,+filename):
++++++++self.filename+=+filename

++++def+startDocument(self):
++++++++self._buffer+=+None
++++++++self._file+=+None

++++def+startElementNS(self,+name,+*args):
++++++++if+name+==+(self.types_ns,+'Content'):
++++++++++++#+we+can+expect+base64+data+next
++++++++++++self._file+=+open(self.filename,+'wb')
++++++++++++self._buffer+=+[]

++++def+endElementNS(self,+name,+*args):
++++++++if+name+==+(self.types_ns,+'Content'):
++++++++++++#+all+attachment+data+received,+close+the+file
++++++++++++try:
++++++++++++++++if+self._buffer:
++++++++++++++++++++raise+ValueError("Incomplete+Base64+data")
++++++++++++finally:
++++++++++++++++self._file.close()
++++++++++++++++self._file+=+self._buffer+=+None

++++def+characters(self,+data):
++++++++if+self._buffer+is+None:
++++++++++++return
++++++++self._buffer.append(data)
++++++++self._decode_buffer()

++++def+_decode_buffer(self):
++++++++remainder+=+''
++++++++for+data+in+self._buffer:
++++++++++++available+=+len(remainder)+%2B+len(data)
++++++++++++overflow+=+available+%25+4
++++++++++++if+remainder:
++++++++++++++++data+=+(remainder+%2B+data)
++++++++++++++++remainder+=+''
++++++++++++if+overflow:
++++++++++++++++remainder,+data+=+data[-overflow:],+data[:-overflow]
++++++++++++if+data:
++++++++++++++++self._file.write(b64decode(data))
++++++++self._buffer+=+[remainder]+if+remainder+else+[]|code-block|syntax|javascript|2352428|你可以这样使用它：|2352429|import+requests
from+xml.sax+import+make_parser,+handler

parser+=+make_parser()
parser.setFeature(handler.feature_namespaces,+True)
parser.setContentHandler(AttachmentContentHandler('foo.txt'))

r+=+requests.post('https://example.com/EWS/Exchange.asmx',+data=...,+stream=True)
r.raw.decode_content+=+True++#+if+content-encoding+is+used,+decompress+as+we+read
parser.parse(r.raw)|2352430|这将解析最大64KB+(默认IncrementalParser+buffer+size)的块，因此附件数据最多解码为48KB的原始数据块。|2352431|我可能会扩展内容处理程序以获取目标目录，然后查找<t:Name>元素来提取文件名，然后使用该元素将数据提取到找到的每个附件的正确文件名。您还需要验证您实际上是在处理一个GetAttachmentResponse文档，并处理错误响应。|2352432|entityMap|0|LINK|mutability|MUTABLE|url|https://docs.python.org/3/library/xml.sax.html#module-xml.sax|1|https://docs.python.org/3/library/xml.sax.handler.html#xml.sax.handler.ContentHandler.characters|2|https://docs.python.org/3/library/xml.sax.html#xml.sax.make_parser|3|https://docs.python.org/3/library/xml.sax.reader.html#xml.sax.xmlreader.XMLReader.parse|4|https://stackoverflow.com/a/22734477/100297|5|https://github.com/python/cpython/blob/a02bc719ebc496bc527e9e46de2f2e83f68bfd77/Lib/xml/sax/xmlreader.py#L111^0|B|9|L|B|0|1T|R|6|K|0|1T|10|1|0|0|0|A|8|11|Y|0|1|8|K|L|1C|8|3F|D|K|L|2|1C|F|3|0|O|D|16|C|4G|8|2L|12|4|0|0|0|0|0|0|E|H|E|T|5|0|O|8|2C|L|0^^$0|@$1|2|3|4|5|6|7|1P|8|@$9|1Q|A|1R|B|C]|$9|1S|A|1T|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|1U|8|@$9|1V|A|1W|B|C]]|D|@$9|1X|A|1Y|1|1Z]|$9|20|A|21|1|22]]|E|$]]|$1|H|3|I|5|6|7|23|8|@]|D|@]|E|$]]|$1|J|3|-4|5|6|7|24|8|@]|D|@]|E|$]]|$1|K|3|L|5|M|7|25|8|@$9|26|A|27|B|C]|$9|28|A|29|B|C]]|D|@]|E|$]]|$1|N|3|O|5|M|7|2A|8|@$9|2B|A|2C|B|C]|$9|2D|A|2E|B|C]|$9|2F|A|2G|B|C]|$9|2H|A|2I|B|C]]|D|@$9|2J|A|2K|1|2L]|$9|2M|A|2N|1|2O]]|E|$]]|$1|P|3|Q|5|M|7|2P|8|@$9|2Q|A|2R|B|C]|$9|2S|A|2T|B|C]|$9|2U|A|2V|B|C]]|D|@$9|2W|A|2X|1|2Y]]|E|$]]|$1|R|3|-4|5|6|7|2Z|8|@]|D|@]|E|$]]|$1|S|3|T|5|6|7|30|8|@]|D|@]|E|$]]|$1|U|3|V|5|W|7|31|8|@]|D|@]|E|$X|Y]]|$1|Z|3|10|5|6|7|32|8|@]|D|@]|E|$]]|$1|11|3|12|5|W|7|33|8|@]|D|@]|E|$X|Y]]|$1|13|3|14|5|6|7|34|8|@$9|35|A|36|B|C]]|D|@$9|37|A|38|1|39]]|E|$]]|$1|15|3|16|5|6|7|3A|8|@$9|3B|A|3C|B|C]|$9|3D|A|3E|B|C]]|D|@]|E|$]]|$1|17|3|-4|5|6|7|3F|8|@]|D|@]|E|$]]]|18|$19|$5|1A|1B|1C|E|$1D|1E]]|1F|$5|1A|1B|1C|E|$1D|1G]]|1H|$5|1A|1B|1C|E|$1D|1I]]|1J|$5|1A|1B|1C|E|$1D|1K]]|1L|$5|1A|1B|1C|E|$1D|1M]]|1N|$5|1A|1B|1C|E|$1D|1O]]]]

Don't use <code>iterparse</code> in this case. The <code>iterparse()</code> method can only issue element start and end events, so any text in an element is given to you when the closing XML tag has been found.

Instead, use a <a href="https://docs.python.org/3/library/xml.sax.html#module-xml.sax" rel="noreferrer">SAX parser interface</a>. This is a general standard for XML parsing libraries, to pass on parsed data to a content handler. The <a href="https://docs.python.org/3/library/xml.sax.handler.html#xml.sax.handler.ContentHandler.characters" rel="noreferrer"><code>ContentHandler.characters()</code> callback</a> is passed character data in chunks (assuming that the implementing XML library actually makes use of this possibility). This is a lower level API from the ElementTree API, and and the Python standard library already bundles the Expat parser to drive it.

So the flow then becomes:

<ul>
<li>wrap the incoming request stream in a <code>GzipFile</code> for easy decompression. Or, better still, set <code>response.raw.decode_content = True</code> and leave decompression to the requests library based on the content-encoding the server has set.</li>
<li>Pass the <code>GzipFile</code> instance or raw stream to the <a href="https://docs.python.org/3/library/xml.sax.reader.html#xml.sax.xmlreader.XMLReader.parse" rel="noreferrer"><code>.parse()</code> method</a> of a parser created with <a href="https://docs.python.org/3/library/xml.sax.html#xml.sax.make_parser" rel="noreferrer"><code>xml.sax.make_parser()</code></a>. The parser then proceeds to read from the stream in chunks. By using <code>make_parser()</code> you first can enable features such as namespace handling (which ensures your code doesn't break if Exchange decides to alter the short prefixes used for each namespace).</li>
<li>The content handler <code>characters()</code> method is called with chunks of XML data; check for the correct element start event, so you know when to expect base64 data. You can decode that base64 data in <a href="https://stackoverflow.com/a/22734477/100297">chunks of (a multiple of) 4 characters</a> at a time, and write it to a file. I'd not use <code>base64io</code> here, just do your own chunking.</li>
</ul>

A simple content handler could be:

<pre><code>from xml.sax import handler
from base64 import b64decode

class AttachmentContentHandler(handler.ContentHandler):
 types_ns = 'http://schemas.microsoft.com/exchange/services/2006/types'

 def __init__(self, filename):
 self.filename = filename

 def startDocument(self):
 self._buffer = None
 self._file = None

 def startElementNS(self, name, *args):
 if name == (self.types_ns, 'Content'):
 # we can expect base64 data next
 self._file = open(self.filename, 'wb')
 self._buffer = []

 def endElementNS(self, name, *args):
 if name == (self.types_ns, 'Content'):
 # all attachment data received, close the file
 try:
 if self._buffer:
 raise ValueError("Incomplete Base64 data")
 finally:
 self._file.close()
 self._file = self._buffer = None

 def characters(self, data):
 if self._buffer is None:
 return
 self._buffer.append(data)
 self._decode_buffer()

 def _decode_buffer(self):
 remainder = ''
 for data in self._buffer:
 available = len(remainder) + len(data)
 overflow = available % 4
 if remainder:
 data = (remainder + data)
 remainder = ''
 if overflow:
 remainder, data = data[-overflow:], data[:-overflow]
 if data:
 self._file.write(b64decode(data))
 self._buffer = [remainder] if remainder else []
</code></pre>

and you'd use it like this:

<pre><code>import requests
from xml.sax import make_parser, handler

parser = make_parser()
parser.setFeature(handler.feature_namespaces, True)
parser.setContentHandler(AttachmentContentHandler('foo.txt'))

r = requests.post('https://example.com/EWS/Exchange.asmx', data=..., stream=True)
r.raw.decode_content = True # if content-encoding is used, decompress as we read
parser.parse(r.raw)
</code></pre>

This will parse the input XML in chunks of up to 64KB (the default <a href="https://github.com/python/cpython/blob/a02bc719ebc496bc527e9e46de2f2e83f68bfd77/Lib/xml/sax/xmlreader.py#L111" rel="noreferrer"><code>IncrementalParser</code> buffer size</a>), so attachment data is decoded in at most 48KB blocks of raw data.

I'd probably extend the content handler to take a target directory and then look for <code>&lt;t:Name&gt;</code> elements to extract the filename, then use that to extract the data to the correct filename for each attachment found. You'd also want to verify that you are actually dealing with a <code>GetAttachmentResponse</code> document, and handle error responses.

I'm trying to consume the Exchange <a href="https://docs.microsoft.com/en-us/exchange/client-developer/web-service-reference/getattachment-operation" rel="nofollow noreferrer">GetAttachment</a> webservice using <a href="http://docs.python-requests.org/en/master/" rel="nofollow noreferrer">requests</a>, <a href="https://lxml.de/" rel="nofollow noreferrer">lxml</a> and <a href="https://github.com/aws/base64io-python" rel="nofollow noreferrer">base64io</a>. This service returns a base64-encoded file in a SOAP XML HTTP response. The file content is contained in a single line in a single XML element. <code>GetAttachment</code> is just an example, but the problem is more general.

I would like to stream the decoded file contents directly to disk without storing the entire contents of the attachment in-memory at any point, since an attachment could be several 100 MB.

I have tried something like this:

<pre><code>r = requests.post('https://example.com/EWS/Exchange.asmx', data=..., stream=True)
with open('foo.txt', 'wb') as f:
 for action, elem in lxml.etree.iterparse(GzipFile(fileobj=r.raw)):
 if elem.tag == 't:Content':
 b64_encoder = Base64IO(BytesIO(elem.text))
 f.write(b64_encoder.read())
</code></pre>

but <code>lxml</code> still stores a copy of the attachment as <code>elem.text</code>. Is there any way I can create a fully streaming XML parser that also streams the content of an element directly from the input stream?

Fully streaming XML parser

服务器

Python

我正在尝试使用、和来使用Exchange。此服务在SOAP XML HTTP响应中返回base64编码的文件。文件内容包含在单个XML元素的一行中。GetAttachment只是一个例子，但问题更普遍。我想直接将解码的文件内容流式传输到磁盘，而不在任何时候将附件的全部内容存储在内存中，因为一个附件可能有几个100MB。...

问全流XML解析器
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问全流XML解析器EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问全流XML解析器
EN