blocks|key|426766|text|如果您是为linux开发的，您可以在单独的进程中运行解压缩，并使用ulimit来限制内存使用。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|426767|import+subprocess
subprocess.Popen("ulimit+-v+%25d;+./decompression_script.py+%25s"+%25+(LIMIT,+FILE))|code-block|syntax|javascript|426768|请记住，在写入磁盘之前，decompression_script.py应该解压缩内存中的整个文件。|426769|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|L|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

If you develop for linux, you can run decompression in separate process and use ulimit to limit the memory usage.

<pre><code>import subprocess
subprocess.Popen("ulimit -v %d; ./decompression_script.py %s" % (LIMIT, FILE))
</code></pre>

Keep in mind that decompression_script.py should decompress the whole file in memory, before writing to disk.

blocks|key|426797|text|我想答案是:没有简单、现成的解决方案。下面是我现在使用的：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|426798|class+SafeUncompressor(object):
++++"""Small+proxy+class+that+enables+external+file+object
++++support+for+uncompressed,+bzip2+and+gzip+files.+Works+transparently,+and
++++supports+a+maximum+size+to+avoid+zipbombs.
++++"""
++++blocksize+=+16+*+1024

++++class+FileTooLarge(Exception):
++++++++pass

++++def+__init__(self,+fileobj,+maxsize=10*1024*1024):
++++++++self.fileobj+=+fileobj
++++++++self.name+=+getattr(self.fileobj,+"name",+None)
++++++++self.maxsize+=+maxsize
++++++++self.init()

++++def+init(self):
++++++++import+bz2
++++++++import+gzip
++++++++self.pos+=+0
++++++++self.fileobj.seek(0)
++++++++self.buf+=+""
++++++++self.format+=+"plain"

++++++++magic+=+self.fileobj.read(2)
++++++++if+magic+==+'\037\213':
++++++++++++self.format+=+"gzip"
++++++++++++self.gzipobj+=+gzip.GzipFile(fileobj+=+self.fileobj,+mode+=+'r')
++++++++elif+magic+==+'BZ':
++++++++++++raise+IOError,+"bzip2+support+in+SafeUncompressor+disabled,+as+self.bz2obj.decompress+is+not+safe"
++++++++++++self.format+=+"bz2"
++++++++++++self.bz2obj+=+bz2.BZ2Decompressor()
++++++++self.fileobj.seek(0)


++++def+read(self,+size):
++++++++b+=+[self.buf]
++++++++x+=+len(self.buf)
++++++++while+x+<+size:
++++++++++++if+self.format+==+'gzip':
++++++++++++++++data+=+self.gzipobj.read(self.blocksize)
++++++++++++++++if+not+data:
++++++++++++++++++++break
++++++++++++elif+self.format+==+'bz2':
++++++++++++++++raw+=+self.fileobj.read(self.blocksize)
++++++++++++++++if+not+raw:
++++++++++++++++++++break
++++++++++++++++#+this+can+already+bomb+here,+to+some+extend.
++++++++++++++++#+so+disable+bzip+support+until+resolved.
++++++++++++++++#+Also+monitor+http://stackoverflow.com/questions/13622706/how-to-protect-myself-from-a-gzip-or-bzip2-bomb+for+ideas
++++++++++++++++data+=+self.bz2obj.decompress(raw)
++++++++++++else:
++++++++++++++++data+=+self.fileobj.read(self.blocksize)
++++++++++++++++if+not+data:
++++++++++++++++++++break
++++++++++++b.append(data)
++++++++++++x+%2B=+len(data)

++++++++++++if+self.pos+%2B+x+>+self.maxsize:
++++++++++++++++self.buf+=+""
++++++++++++++++self.pos+=+0
++++++++++++++++raise+SafeUncompressor.FileTooLarge,+"Compressed+file+too+large"
++++++++self.buf+=+"".join(b)

++++++++buf+=+self.buf[:size]
++++++++self.buf+=+self.buf[size:]
++++++++self.pos+%2B=+len(buf)
++++++++return+buf

++++def+seek(self,+pos,+whence=0):
++++++++if+whence+!=+0:
++++++++++++raise+IOError,+"SafeUncompressor+only+supports+whence=0"
++++++++if+pos+<+self.pos:
++++++++++++self.init()
++++++++self.read(pos+-+self.pos)

++++def+tell(self):
++++++++return+self.pos|code-block|syntax|javascript|426799|它不能很好地用于bzip2，因此部分代码被禁用。原因是bz2.BZ2Decompressor.decompress可能已经产生了大量不需要的数据。|offset|length|style|CODE|426800|entityMap^0|0|0|R|U|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@$I|R|J|S|K|L]]|9|@]|A|$]]|$1|M|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|N|$]]

I guess the answer is: There is no easy, readymade solution. Here is what I use now:

<pre><code>class SafeUncompressor(object):
 """Small proxy class that enables external file object
 support for uncompressed, bzip2 and gzip files. Works transparently, and
 supports a maximum size to avoid zipbombs.
 """
 blocksize = 16 * 1024

 class FileTooLarge(Exception):
 pass

 def __init__(self, fileobj, maxsize=10*1024*1024):
 self.fileobj = fileobj
 self.name = getattr(self.fileobj, "name", None)
 self.maxsize = maxsize
 self.init()

 def init(self):
 import bz2
 import gzip
 self.pos = 0
 self.fileobj.seek(0)
 self.buf = ""
 self.format = "plain"

 magic = self.fileobj.read(2)
 if magic == '\037\213':
 self.format = "gzip"
 self.gzipobj = gzip.GzipFile(fileobj = self.fileobj, mode = 'r')
 elif magic == 'BZ':
 raise IOError, "bzip2 support in SafeUncompressor disabled, as self.bz2obj.decompress is not safe"
 self.format = "bz2"
 self.bz2obj = bz2.BZ2Decompressor()
 self.fileobj.seek(0)


 def read(self, size):
 b = [self.buf]
 x = len(self.buf)
 while x &lt; size:
 if self.format == 'gzip':
 data = self.gzipobj.read(self.blocksize)
 if not data:
 break
 elif self.format == 'bz2':
 raw = self.fileobj.read(self.blocksize)
 if not raw:
 break
 # this can already bomb here, to some extend.
 # so disable bzip support until resolved.
 # Also monitor http://stackoverflow.com/questions/13622706/how-to-protect-myself-from-a-gzip-or-bzip2-bomb for ideas
 data = self.bz2obj.decompress(raw)
 else:
 data = self.fileobj.read(self.blocksize)
 if not data:
 break
 b.append(data)
 x += len(data)

 if self.pos + x &gt; self.maxsize:
 self.buf = ""
 self.pos = 0
 raise SafeUncompressor.FileTooLarge, "Compressed file too large"
 self.buf = "".join(b)

 buf = self.buf[:size]
 self.buf = self.buf[size:]
 self.pos += len(buf)
 return buf

 def seek(self, pos, whence=0):
 if whence != 0:
 raise IOError, "SafeUncompressor only supports whence=0"
 if pos &lt; self.pos:
 self.init()
 self.read(pos - self.pos)

 def tell(self):
 return self.pos
</code></pre>

It does not work well for bzip2, so that part of the code is disabled. The reason is that <code>bz2.BZ2Decompressor.decompress</code> can already produce an unwanted large chunk of data.

blocks|key|208678|text|这将确定gzip流的解压缩大小，同时使用有限的内存：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|208679|#!/usr/bin/python
import+sys
import+zlib
f+=+open(sys.argv[1],+"rb")
z+=+zlib.decompressobj(15%2B16)
total+=+0
while+True:
++++buf+=+z.unconsumed_tail
++++if+buf+==+"":
++++++++buf+=+f.read(1024)
++++++++if+buf+==+"":
++++++++++++break
++++got+=+z.decompress(buf,+4096)
++++if+got+==+"":
++++++++break
++++total+%2B=+len(got)
print+total
if+z.unused_data+!=+""+or+f.read(1024)+!=+"":
++++print+"warning:+more+input+after+end+of+gzip+stream"|code-block|syntax|javascript|208680|解压时，它将返回对tar文件中所有文件所需空间的略微高估。长度包括这些文件，以及tar目录信息。|208681|gzip.py代码不控制解压缩的数据量，除非依靠输入数据的大小。在gzip.py中，它一次读取1024个压缩字节。因此，如果您可以使用高达1056768字节的内存来存储未压缩的数据(1032x1024，其中1032:1是deflate的最大压缩比)，则可以使用gzip.py。这里的解决方案使用带有第二个参数的zlib.decompress，该参数限制了未压缩数据的数量。gzip.py不需要。|offset|length|style|CODE|208682|这将通过解码tar格式准确地确定提取的tar条目的总大小：|208683|#!/usr/bin/python

import+sys
import+zlib

def+decompn(f,+z,+n):
++++"""Return+n+uncompressed+bytes,+or+fewer+if+at+the+end+of+the+compressed
+++++++stream.++This+only+decompresses+as+much+as+necessary,+in+order+to
+++++++avoid+excessive+memory+usage+for+highly+compressed+input.
++++"""
++++blk+=+""
++++while+len(blk)+<+n:
++++++++buf+=+z.unconsumed_tail
++++++++if+buf+==+"":
++++++++++++buf+=+f.read(1024)
++++++++got+=+z.decompress(buf,+n+-+len(blk))
++++++++blk+%2B=+got
++++++++if+got+==+"":
++++++++++++break
++++return+blk

f+=+open(sys.argv[1],+"rb")
z+=+zlib.decompressobj(15%2B16)
total+=+0
left+=+0
while+True:
++++blk+=+decompn(f,+z,+512)
++++if+len(blk)+<+512:
++++++++break
++++if+left+==+0:
++++++++if+blk+==+"\0"*512:
++++++++++++continue
++++++++if+blk[156]+in+["1",+"2",+"3",+"4",+"5",+"6"]:
++++++++++++continue
++++++++if+blk[124]+==+0x80:
++++++++++++size+=+0
++++++++++++for+i+in+range(125,+136):
++++++++++++++++size+<<=+8
++++++++++++++++size+%2B=+blk[i]
++++++++else:
++++++++++++size+=+int(blk[124:136].split()[0].split("\0")[0],+8)
++++++++if+blk[156]+not+in+["x",+"g",+"X",+"L",+"K"]:
++++++++++++++++total+%2B=+size
++++++++left+=+(size+%2B+511)+//+512
++++else:
++++++++left+-=+1
print+total
if+blk+!=+"":
++++print+"warning:+partial+final+block"
if+left+!=+0:
++++print+"warning:+tar+file+ended+in+the+middle+of+an+entry"
if+z.unused_data+!=+""+or+f.read(1024)+!=+"":
++++print+"warning:+more+input+after+end+of+gzip+stream"|208684|您可以使用它的一个变体来扫描tar文件中的炸弹。这样做的好处是，在解压数据之前，就可以在头信息中找到较大的大小。|208685|至于.tar.bz2归档，Python+bz2库(至少在3.3版本中)对于消耗太多内存的bz2炸弹来说不可避免地是不安全的。与zlib.decompress不同，bz2.decompress函数不提供第二个参数。更糟糕的是，由于游程编码，bz2格式的最大压缩比比zlib要高得多。bzip2将1+GB的0压缩为722字节。因此，即使没有第二个参数，也不能像使用zlib.decompress那样通过计量输入来计量bz2.decompress的输出。缺乏对解压缩输出大小的限制是Python接口中的一个根本缺陷。|208686|我查看了3.3中的_bz2module.c，看看是否有未记录的方法来使用它来避免这个问题。这是无计可施的。其中的decompress函数只是不断增加结果缓冲区，直到它可以解压缩所有提供的输入。_bz2module.c需要修复。|208687|entityMap^0|0|0|0|4B|F|0|0|0|0|1R|F|29|E|50|F|5Q|E|0|1K|A|0^^$0|@$1|2|3|4|5|6|7|10|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|11|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|12|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|13|8|@$K|14|L|15|M|N]]|9|@]|A|$]]|$1|O|3|P|5|6|7|16|8|@]|9|@]|A|$]]|$1|Q|3|R|5|D|7|17|8|@]|9|@]|A|$E|F]]|$1|S|3|T|5|6|7|18|8|@]|9|@]|A|$]]|$1|U|3|V|5|6|7|19|8|@$K|1A|L|1B|M|N]|$K|1C|L|1D|M|N]|$K|1E|L|1F|M|N]|$K|1G|L|1H|M|N]]|9|@]|A|$]]|$1|W|3|X|5|6|7|1I|8|@$K|1J|L|1K|M|N]]|9|@]|A|$]]|$1|Y|3|-4|5|6|7|1L|8|@]|9|@]|A|$]]]|Z|$]]

This will determine the uncompressed size of the gzip stream, while using limited memory:

<pre><code>#!/usr/bin/python
import sys
import zlib
f = open(sys.argv[1], "rb")
z = zlib.decompressobj(15+16)
total = 0
while True:
 buf = z.unconsumed_tail
 if buf == "":
 buf = f.read(1024)
 if buf == "":
 break
 got = z.decompress(buf, 4096)
 if got == "":
 break
 total += len(got)
print total
if z.unused_data != "" or f.read(1024) != "":
 print "warning: more input after end of gzip stream"
</code></pre>

It will return a slight overestimate of the space required for all of the files in the tar file in when extracted. The length includes those files, as well as the tar directory information.

The gzip.py code does not control the amount of data decompressed, except by virtue of the size of the input data. In gzip.py, it reads 1024 compressed bytes at a time. So you can use gzip.py if you're ok with up to about 1056768 bytes of memory usage for the uncompressed data (1032 * 1024, where 1032:1 is the maximum compression ratio of deflate). The solution here uses <code>zlib.decompress</code> with the second argument, which limits the amount of uncompressed data. gzip.py does not.

This will accurately determine the total size of the extracted tar entries by decoding the tar format:

<pre><code>#!/usr/bin/python

import sys
import zlib

def decompn(f, z, n):
 """Return n uncompressed bytes, or fewer if at the end of the compressed
 stream. This only decompresses as much as necessary, in order to
 avoid excessive memory usage for highly compressed input.
 """
 blk = ""
 while len(blk) &lt; n:
 buf = z.unconsumed_tail
 if buf == "":
 buf = f.read(1024)
 got = z.decompress(buf, n - len(blk))
 blk += got
 if got == "":
 break
 return blk

f = open(sys.argv[1], "rb")
z = zlib.decompressobj(15+16)
total = 0
left = 0
while True:
 blk = decompn(f, z, 512)
 if len(blk) &lt; 512:
 break
 if left == 0:
 if blk == "\0"*512:
 continue
 if blk[156] in ["1", "2", "3", "4", "5", "6"]:
 continue
 if blk[124] == 0x80:
 size = 0
 for i in range(125, 136):
 size &lt;&lt;= 8
 size += blk[i]
 else:
 size = int(blk[124:136].split()[0].split("\0")[0], 8)
 if blk[156] not in ["x", "g", "X", "L", "K"]:
 total += size
 left = (size + 511) // 512
 else:
 left -= 1
print total
if blk != "":
 print "warning: partial final block"
if left != 0:
 print "warning: tar file ended in the middle of an entry"
if z.unused_data != "" or f.read(1024) != "":
 print "warning: more input after end of gzip stream"
</code></pre>

You could use a variant of this to scan the tar file for bombs. This has the advantage of finding a large size in the header information before you even have to decompress that data.

As for .tar.bz2 archives, the Python bz2 library (at least as of 3.3) is unavoidably unsafe for bz2 bombs consuming too much memory. The <code>bz2.decompress</code> function does not offer a second argument like <code>zlib.decompress</code> does. This is made even worse by the fact that the bz2 format has a much, much higher maximum compression ratio than zlib due to run-length coding. bzip2 compresses 1 GB of zeros to 722 bytes. So you cannot meter the output of <code>bz2.decompress</code> by metering the input as can be done with <code>zlib.decompress</code> even without the second argument. The lack of a limit on the decompressed output size is a fundamental flaw in the Python interface.

I looked in the _bz2module.c in 3.3 to see if there is an undocumented way to use it to avoid this problem. There is no way around it. The <code>decompress</code> function in there just keeps growing the result buffer until it can decompress all of the provided input. _bz2module.c needs to be fixed.

blocks|key|208706|text|您可以使用resource+module来限制您的进程及其子进程可用的资源。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|208707|如果你需要在内存中解压，那么你可以设置resource.RLIMIT_AS+(或RLIMIT_DATA，RLIMIT_STACK)，例如，使用上下文管理器自动将其恢复为以前的值：|208708|import+contextlib
import+resource

@contextlib.contextmanager
def+limit(limit,+type=resource.RLIMIT_AS):
++++soft_limit,+hard_limit+=+resource.getrlimit(type)
++++resource.setrlimit(type,+(limit,+hard_limit))+#+set+soft+limit
++++try:
++++++++yield
++++finally:
++++++++resource.setrlimit(type,+(soft_limit,+hard_limit))+#+restore

with+limit(1+<<+30):+#+1GB+
++++#+do+the+thing+that+might+try+to+consume+all+memory|code-block|syntax|javascript|208709|如果达到限制，则MemoryError将被提升。|208710|entityMap|0|LINK|mutability|MUTABLE|url|http://docs.python.org/2/library/resource.html^0|5|8|5|F|0|0|J|I|14|B|1G|C|0|0|8|B|0^^$0|@$1|2|3|4|5|6|7|W|8|@$9|X|A|Y|B|C]]|D|@$9|Z|A|10|1|11]]|E|$]]|$1|F|3|G|5|6|7|12|8|@$9|13|A|14|B|C]|$9|15|A|16|B|C]|$9|17|A|18|B|C]]|D|@]|E|$]]|$1|H|3|I|5|J|7|19|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|1A|8|@$9|1B|A|1C|B|C]]|D|@]|E|$]]|$1|O|3|-4|5|6|7|1D|8|@]|D|@]|E|$]]]|P|$Q|$5|R|S|T|E|$U|V]]]]

You could use <a href="http://docs.python.org/2/library/resource.html"><code>resource</code> module</a> to limit resources available to your process and its children.

If you need to decompress in memory then you could set <code>resource.RLIMIT_AS</code> (or <code>RLIMIT_DATA</code>, <code>RLIMIT_STACK</code>) e.g., using a context manager to automatically restore it to a previous value:

<pre><code>import contextlib
import resource

@contextlib.contextmanager
def limit(limit, type=resource.RLIMIT_AS):
 soft_limit, hard_limit = resource.getrlimit(type)
 resource.setrlimit(type, (limit, hard_limit)) # set soft limit
 try:
 yield
 finally:
 resource.setrlimit(type, (soft_limit, hard_limit)) # restore

with limit(1 &lt;&lt; 30): # 1GB 
 # do the thing that might try to consume all memory
</code></pre>

If the limit is reached; <code>MemoryError</code> is raised.

blocks|key|208716|text|我还需要处理上传的zipfile中的zip炸弹。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|208717|为此，我创建了一个固定大小的tmpfs，并将其解压缩。如果提取的数据太大，则tmpfs将耗尽空间并给出错误。|208718|下面是创建200M+tmpfs解压到其中的linux命令。|208719|sudo+mkdir+-p+/mnt/ziptmpfs
echo+'tmpfs+++/mnt/ziptmpfs+++++++++tmpfs+++rw,nodev,nosuid,size=200M++++++++++0++0'+%7C+sudo+tee+-a+/etc/fstab|code-block|syntax|javascript|208720|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|N|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|O|8|@]|9|@]|A|$]]|$1|F|3|G|5|H|7|P|8|@]|9|@]|A|$I|J]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

I also need to handle zip bombs in uploaded zipfiles.

I do this by creating a fixed size tmpfs, and unzipping to that. If the extracted data is too large then the tmpfs will run out of space and give an error.

Here is the linux commands to create a 200M tmpfs to unzip to.

<pre><code>sudo mkdir -p /mnt/ziptmpfs
echo 'tmpfs /mnt/ziptmpfs tmpfs rw,nodev,nosuid,size=200M 0 0' | sudo tee -a /etc/fstab
</code></pre>

This is related to the question <a href="https://stackoverflow.com/questions/1459080/how-can-i-protect-myself-from-a-zip-bomb">about zip bombs</a>, but having gzip or bzip2 compression in mind, e.g. a web service accepting <code>.tar.gz</code> files.

Python provides a handy <a href="http://docs.python.org/2/library/tarfile.html" rel="nofollow noreferrer">tarfile module</a> that is convenient to use, but does not seem to provide protection against zipbombs.

In python code using the tarfile module, what would be the most elegant way to detect zip bombs, preferably without duplicating too much logic (e.g. the transparent decompression support) from the tarfile module?

And, just to make it a bit less simple: No real files are involved; the input is a file-like object (provided by the web framework, representing the file a user uploaded).

How to protect myself from a gzip or bzip2 bomb?

Python

这与问题有关，但考虑到了gzip或bzip2压缩，例如接受.tar.gz文件的web服务。Python提供了一个方便使用的，但似乎不能提供对zipbombs的保护。在使用tarfile模块的python代码中，最好不要从tarfile模块复制太多的逻辑(例如，透明的解压缩支持)，那么检测zip炸弹最优雅的方法是什么？而...

问如何保护自己免受gzip或bzip2炸弹的攻击？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何保护自己免受gzip或bzip2炸弹的攻击？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何保护自己免受gzip或bzip2炸弹的攻击？
EN