Numpy的一个小知识

昨天一个同行问了一个问题：

viewdata = np.memmap(path, shape = (), mode = 'r',offset = 0,

dtype = '({})i4,'.format(4))

这个语句中，dtype = '({})i4,'.format(4)很令人费解，到底是啥意思？

根据python 的 format() 函数，dtype = '({})i4,'.format(4) 实际上是dtype = '(4)i4,'

i4是32位有符号整数，但放在一块儿是啥意思呢？

我比较少使用numpy,甚至电脑都没有装它。为了彻底明白它的含义，先看看memmap的定义：

classmemmap(ndarray):

"""Create a memory-map to an array stored in a *binary* file on disk.

Memory-mapped files are used for accessing small segments of large files

on disk, without reading the entire file into memory. NumPy's

memmap's are array-like objects. This differs from Python's ``mmap``

module, which uses file-like objects.

This subclass of ndarray has some unpleasant interactions with

some operations, because it doesn't quite fit properly as a subclass.

An alternative to using this subclass is to create the ``mmap``

object yourself, then create an ndarray with ndarray.__new__ directly,

passing the object created in its 'buffer=' parameter.

This class may at some point be turned into a factory function

which returns a view into an mmap buffer.

Delete the memmap instance to close the memmap file.

Parameters

----------

filename : str, file-like object, or pathlib.Path instance

The file name or file object to be used as the array data buffer.

dtype : data-type, optional

The data-type used to interpret the file contents.

Default is `uint8`.

mode : {'r+', 'r', 'w+', 'c'}, optional

The file is opened in this mode:

+------+-------------------------------------------------------------+

| 'r' | Open existing file for reading only. |

+------+-------------------------------------------------------------+

| 'r+' | Open existing file for reading and writing. |

+------+-------------------------------------------------------------+

| 'w+' | Create or overwrite existing file for reading and writing. |

+------+-------------------------------------------------------------+

| 'c' | Copy-on-write: assignments affect data in memory, but |

| | changes are not saved to disk. The file on disk is |

| | read-only. |

+------+-------------------------------------------------------------+

Default is 'r+'.

offset : int, optional

In the file, array data starts at this offset. Since `offset` is

measured in bytes, it should normally be a multiple of the byte-size

of `dtype`. When ``mode != 'r'``, even positive offsets beyond end of

file are valid; The file will be extended to accommodate the

additional data. By default, ``memmap`` will start at the beginning of

the file, even if ``filename`` is a file pointer ``fp`` and

``fp.tell() != 0``.

shape : tuple, optional

The desired shape of the array. If ``mode == 'r'`` and the number

of remaining bytes after `offset` is not a multiple of the byte-size

of `dtype`, you must specify `shape`. By default, the returned array

will be 1-D with the number of elements determined by file size

and data-type.

order : {'C', 'F'}, optional

Specify the order of the ndarray memory layout:

:term:`row-major`, C-style or :term:`column-major`,

Fortran-style. This only has an effect if the shape is

greater than 1-D. The default order is 'C'.

Attributes

----------

filename : str or pathlib.Path instance

Path to the mapped file.

offset : int

Offset position in the file.

mode : str

File mode.

Methods

-------

flush

Flush any changes in memory to file on disk.

When you delete a memmap object, flush is called first to write

changes to disk before removing the object.

See also

--------

lib.format.open_memmap : Create or load a memory-mapped ``.npy`` file.

Notes

-----

The memmap object can be used anywhere an ndarray is accepted.

Given a memmap ``fp``, ``isinstance(fp, numpy.ndarray)`` returns

``True``.

Memory-mapped files cannot be larger than 2GB on 32-bit systems.

When a memmap causes a file to be created or extended beyond its

current size in the filesystem, the contents of the new part are

unspecified. On systems with POSIX filesystem semantics, the extended

part will be filled with zero bytes.

Examples

--------

>>> data = np.arange(12, dtype='float32')

>>> data.resize((3,4))

This example uses a temporary file so that doctest doesn't write

files to your directory. You would use a 'normal' filename.

>>> from tempfile import mkdtemp

>>> import os.path as path

>>> filename = path.join(mkdtemp(), 'newfile.dat')

Create a memmap with dtype and shape that matches our data:

>>> fp = np.memmap(filename, dtype='float32', mode='w+', shape=(3,4))

memmap([[ 0., 0., 0., 0.],

[ 0., 0., 0., 0.],

[ 0., 0., 0., 0.]], dtype=float32)

Write data to memmap array:

>>> fp[:] = data[:]

memmap([[ 0., 1., 2., 3.],

[ 4., 5., 6., 7.],

[ 8., 9., 10., 11.]], dtype=float32)

>>> fp.filename == path.abspath(filename)

True

Deletion flushes memory changes to disk before removing the object:

>>> del fp

Load the memmap and verify data was stored:

>>> newfp = np.memmap(filename, dtype='float32', mode='r', shape=(3,4))

memmap([[ 0., 1., 2., 3.],

[ 4., 5., 6., 7.],

[ 8., 9., 10., 11.]], dtype=float32)

Read-only memmap:

>>> fpr = np.memmap(filename, dtype='float32', mode='r', shape=(3,4))

False

Copy-on-write memmap:

>>> fpc = np.memmap(filename, dtype='float32', mode='c', shape=(3,4))

True

It's possible to assign to copy-on-write array, but values are only

written into the memory copy of the array, and not written to disk:

memmap([[ 0., 1., 2., 3.],

[ 4., 5., 6., 7.],

[ 8., 9., 10., 11.]], dtype=float32)

>>> fpc[0,:] = 0

memmap([[ 0., 0., 0., 0.],

[ 4., 5., 6., 7.],

[ 8., 9., 10., 11.]], dtype=float32)

File on disk is unchanged:

memmap([[ 0., 1., 2., 3.],

[ 4., 5., 6., 7.],

[ 8., 9., 10., 11.]], dtype=float32)

Offset into a memmap:

>>> fpo = np.memmap(filename, dtype='float32', mode='r', offset=16)

memmap([ 4., 5., 6., 7., 8., 9., 10., 11.], dtype=float32)

"""

__array_priority__ = -100.0

def__new__(subtype, filename, dtype=uint8, mode='r+', offset=0,

shape=None, order='C'):

# Import here to minimize 'import numpy' overhead

importmmap

importos.path

try:

mode = mode_equivalents[mode]

exceptKeyError:

ifmodenot invalid_filemodes:

raiseValueError("mode must be one of %s"%

(valid_filemodes +list(mode_equivalents.keys())))

ifhasattr(filename, 'read'):

fid = filename

own_file =False

elifis_pathlib_path(filename):

fid = filename.open((mode =='c'and'r'ormode)+'b')

own_file =True

else:

fid = open(filename, (mode =='c'and'r'or mode)+'b')

own_file =True

if(mode =='w+')andshapeis None:

raise ValueError("shape must be given")

fid.seek(0, 2)

flen = fid.tell()

descr = dtypedescr(dtype)

_dbytes = descr.itemsize

ifshapeis None:

bytes= flen - offset

if(bytes% _dbytes):

fid.close()

raiseValueError("Size of available data is not a "

"multiple of the data-type size.")

size =bytes// _dbytes

shape = (size,)

else:

if notisinstance(shape,tuple):

shape = (shape,)

size = 1

forkinshape:

size *= k

bytes= long(offset + size*_dbytes)

ifmode =='w+'or(mode =='r+'andflen

fid.seek(bytes- 1, 0)

fid.write(b'\0')

fid.flush()

ifmode =='c':

acc = mmap.ACCESS_COPY

elifmode =='r':

acc = mmap.ACCESS_READ

else:

acc = mmap.ACCESS_WRITE

start = offset - offset % mmap.ALLOCATIONGRANULARITY

bytes-= start

array_offset = offset - start

mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)

self = ndarray.__new__(subtype, shape, dtype=descr, buffer=mm,

offset=array_offset, order=order)

self._mmap = mm

self.offset = offset

self.mode = mode

ifisinstance(filename, basestring):

elifis_pathlib_path(filename):

self.filename = filename.resolve()

# py3 returns int for TemporaryFile().name

elif(hasattr(filename,"name")and

isinstance(filename.name, basestring)):

# same as memmap copies (e.g. memmap + 1)

else:

self.filename =None

ifown_file:

fid.close()

returnself

def__array_finalize__(self, obj):

ifhasattr(obj,'_mmap') and np.may_share_memory(self, obj):

self._mmap = obj._mmap

self.filename = obj.filename

self.offset = obj.offset

self.mode = obj.mode

else:

self._mmap =None

self.filename =None

self.offset =None

self.mode =None

defflush(self):

"""

Write any changes in the array to the file on disk.

For further information, see `memmap`.

Parameters

----------

None

See Also

--------

memmap

"""

ifself.baseis not None andhasattr(self.base, 'flush'):

def__array_wrap__(self, arr, context=None):

arr = super(memmap, self).__array_wrap__(arr, context)

# Return a memmap if a memmap was given as the output of the

# ufunc. Leave the arr class unchanged if self is not a memmap

# to keep original memmap subclasses behavior

ifselfisarrortype(self) is notmemmap:

returnarr

# Return scalar instead of 0d memmap, e.g. for np.sum with

# axis=None

ifarr.shape == ():

returnarr[()]

# Return ndarray otherwise

returnarr.view(np.ndarray)

def__getitem__(self, index):

res =super(memmap, self).__getitem__(index)

iftype(res)ismemmapandres._mmapis None:

return res.view(type=ndarray)

returnres

-------------------------------------

仔细看了上面的定义和举例，并没有得到答案。

然后，搜索了各种网，仍然没有答案......

根据PYTHON的知识来推断：

shape = ()，是个空的元组，说明memmap返回的是个一维数组，

dtype = '(4)i4,'，数组元素是32位有符号整数，'(4)i4,' 这个逗号是用来产生元组的，那么dtype = '(4)i4,'这个4会不会是一维数组的元素个数呢？怎么验证？

实践出真知！

1，下载安装numpy。

2，写个名为“test.py”的文件：“abcdefghijklmnop”

3，写个简单程序：

>>>importnumpyasnp

>>>classNumpytest:

defzwj(self,size):

int_1D_array = np.memmap(

'test.py',shape = (),mode ='r',

offset = 0,

dtype ='({})i4,'.format(size)

)

returnint_1D_array

>>>t = Numpytest()

>>>t.zwj(1)

memmap(1684234849)

>>>t.zwj(2)

memmap([1684234849, 1751606885])

>>>t.zwj(4)

memmap([1684234849, 1751606885, 1818978921, 1886350957])

这完全证明了：dtype = '(4)i4,'这个4是一维数组的元素个数。

（另外，memmap定义中是个类，在使用中完全象一个函数，python中的类和函数没有完全的区别）