blocks|key|1643973|text|在CPython+3.6+requests+a+UTF-8+form+of+the+string+to+work+with中将字符串转换为整数的代码|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1643974|buffer+=+PyUnicode_AsUTF8AndSize(asciidig,+&buflen);|code-block|syntax|javascript|1643975|字符串在第一次被请求时创建UTF-8表示形式，并使用caches+it+on+the+string+object|1643976|if+(PyUnicode_UTF8(unicode)+==+NULL)+{
++++assert(!PyUnicode_IS_COMPACT_ASCII(unicode));
++++bytes+=+_PyUnicode_AsUTF8String(unicode,+NULL);
++++if+(bytes+==+NULL)
++++++++return+NULL;
++++_PyUnicode_UTF8(unicode)+=+PyObject_MALLOC(PyBytes_GET_SIZE(bytes)+%2B+1);
++++if+(_PyUnicode_UTF8(unicode)+==+NULL)+{
++++++++PyErr_NoMemory();
++++++++Py_DECREF(bytes);
++++++++return+NULL;
++++}
++++_PyUnicode_UTF8_LENGTH(unicode)+=+PyBytes_GET_SIZE(bytes);
++++memcpy(_PyUnicode_UTF8(unicode),
++++++++++++++PyBytes_AS_STRING(bytes),
++++++++++++++_PyUnicode_UTF8_LENGTH(unicode)+%2B+1);
++++Py_DECREF(bytes);
}|1643977|额外的3个字节用于UTF-8表示。|1643978|1643979|您可能想知道，当字符串类似于'40'或'plain+ascii+text'时，为什么大小不变。这是因为如果字符串是"compact+ascii"+representation格式的，Python就不会创建单独的UTF8表示。It+returns+the+ASCII+representation+directly，它已经是有效的UTF-8：|style|CODE|1643980|#define+PyUnicode_UTF8(op)++++++++++++++++++++++++++++++\
++++(assert(_PyUnicode_CHECK(op)),++++++++++++++++++++++\
+++++assert(PyUnicode_IS_READY(op)),++++++++++++++++++++\
+++++PyUnicode_IS_COMPACT_ASCII(op)+?+++++++++++++++++++\
+++++++++((char*)((PyASCIIObject*)(op)+%2B+1))+:++++++++++\
+++++++++_PyUnicode_UTF8(op))|1643981|1643982|您可能还想知道为什么像'１'这样的东西的大小不变。这是U%2BFF11+FULLWIDTH数字1，int认为它等同于'1'。这是因为字符串到整数过程中的one+of+the+earlier+steps是|1643983|asciidig+=+_PyUnicode_TransformDecimalAndSpaceToASCII(u);|1643984|它将所有空格字符转换为'+'，并将所有Unicode十进制数字转换为相应的ASCII位。如果最终没有更改任何内容，此转换将返回原始字符串，但当它进行更改时，它将创建一个新字符串，而新字符串将创建一个UTF-8表示。|1643985|1643986|至于在一个字符串上调用int看起来会影响另一个字符串的情况，它们实际上是同一个string对象。在许多情况下，Python将重用字符串，所有这些都与我们到目前为止讨论的所有情况一样坚定地存在于奇怪的实现细节领域。对于'ñ'，之所以会发生重用，是因为这是一个拉丁语-1范围('\x00'-'\xff')中的单字符字符串，其实现是stores+and+reuses+those。|1643987|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/python/cpython/blob/v3.6.3/Objects/longobject.c#L2503|1|https://github.com/python/cpython/blob/v3.6.3/Objects/unicodeobject.c#L4016|2|https://github.com/python/cpython/blob/v3.6.3/Include/unicodeobject.h#L202|3|https://github.com/python/cpython/blob/v3.6.3/Objects/unicodeobject.c#L80|4|https://github.com/python/cpython/blob/v3.6.3/Objects/longobject.c#L2500|5|https://github.com/python/cpython/blob/v3.6.3/Objects/unicodeobject.c#L213^0|D|1C|0|0|0|Q|U|1|0|0|0|0|E|4|J|I|1L|U|2|37|15|3|0|0|0|B|3|1B|3|1K|3|22|O|4|0|0|B|3|0|0|B|3|30|3|3S|6|3Z|6|4J|N|5|0^^$0|@$1|2|3|4|5|6|7|1N|8|@]|9|@$A|1O|B|1P|1|1Q]]|C|$]]|$1|D|3|E|5|F|7|1R|8|@]|9|@]|C|$G|H]]|$1|I|3|J|5|6|7|1S|8|@]|9|@$A|1T|B|1U|1|1V]]|C|$]]|$1|K|3|L|5|F|7|1W|8|@]|9|@]|C|$G|H]]|$1|M|3|N|5|6|7|1X|8|@]|9|@]|C|$]]|$1|O|3|-4|5|6|7|1Y|8|@]|9|@]|C|$]]|$1|P|3|Q|5|6|7|1Z|8|@$A|20|B|21|R|S]|$A|22|B|23|R|S]]|9|@$A|24|B|25|1|26]|$A|27|B|28|1|29]]|C|$]]|$1|T|3|U|5|F|7|2A|8|@]|9|@]|C|$G|H]]|$1|V|3|-4|5|6|7|2B|8|@]|9|@]|C|$]]|$1|W|3|X|5|6|7|2C|8|@$A|2D|B|2E|R|S]|$A|2F|B|2G|R|S]|$A|2H|B|2I|R|S]]|9|@$A|2J|B|2K|1|2L]]|C|$]]|$1|Y|3|Z|5|F|7|2M|8|@]|9|@]|C|$G|H]]|$1|10|3|11|5|6|7|2N|8|@$A|2O|B|2P|R|S]]|9|@]|C|$]]|$1|12|3|-4|5|6|7|2Q|8|@]|9|@]|C|$]]|$1|13|3|14|5|6|7|2R|8|@$A|2S|B|2T|R|S]|$A|2U|B|2V|R|S]|$A|2W|B|2X|R|S]|$A|2Y|B|2Z|R|S]]|9|@$A|30|B|31|1|32]]|C|$]]|$1|15|3|-4|5|6|7|33|8|@]|9|@]|C|$]]]|16|$17|$5|18|19|1A|C|$1B|1C]]|1D|$5|18|19|1A|C|$1B|1E]]|1F|$5|18|19|1A|C|$1B|1G]]|1H|$5|18|19|1A|C|$1B|1I]]|1J|$5|18|19|1A|C|$1B|1K]]|1L|$5|18|19|1A|C|$1B|1M]]]]

The code that converts strings to ints in CPython 3.6 <a href="https://github.com/python/cpython/blob/v3.6.3/Objects/longobject.c#L2503" rel="noreferrer">requests a UTF-8 form of the string to work with</a>:

<pre><code>buffer = PyUnicode_AsUTF8AndSize(asciidig, &amp;buflen);
</code></pre>

and the string creates the UTF-8 representation the first time it's requested and <a href="https://github.com/python/cpython/blob/v3.6.3/Objects/unicodeobject.c#L4016" rel="noreferrer">caches it on the string object</a>:

<pre><code>if (PyUnicode_UTF8(unicode) == NULL) {
 assert(!PyUnicode_IS_COMPACT_ASCII(unicode));
 bytes = _PyUnicode_AsUTF8String(unicode, NULL);
 if (bytes == NULL)
 return NULL;
 _PyUnicode_UTF8(unicode) = PyObject_MALLOC(PyBytes_GET_SIZE(bytes) + 1);
 if (_PyUnicode_UTF8(unicode) == NULL) {
 PyErr_NoMemory();
 Py_DECREF(bytes);
 return NULL;
 }
 _PyUnicode_UTF8_LENGTH(unicode) = PyBytes_GET_SIZE(bytes);
 memcpy(_PyUnicode_UTF8(unicode),
 PyBytes_AS_STRING(bytes),
 _PyUnicode_UTF8_LENGTH(unicode) + 1);
 Py_DECREF(bytes);
}
</code></pre>

The extra 3 bytes are for the UTF-8 representation.

<hr>

You might be wondering why the size doesn't change when the string is something like <code>'40'</code> or <code>'plain ascii text'</code>. That's because if the string is in <a href="https://github.com/python/cpython/blob/v3.6.3/Include/unicodeobject.h#L202" rel="noreferrer">"compact ascii" representation</a>, Python doesn't create a separate UTF-8 representation. It <a href="https://github.com/python/cpython/blob/v3.6.3/Objects/unicodeobject.c#L80" rel="noreferrer">returns the ASCII representation directly</a>, which is already valid UTF-8:

<pre><code>#define PyUnicode_UTF8(op) \
 (assert(_PyUnicode_CHECK(op)), \
 assert(PyUnicode_IS_READY(op)), \
 PyUnicode_IS_COMPACT_ASCII(op) ? \
 ((char*)((PyASCIIObject*)(op) + 1)) : \
 _PyUnicode_UTF8(op))
</code></pre>

<hr>

You also might wonder why the size doesn't change for something like <code>'１'</code>. That's U+FF11 FULLWIDTH DIGIT ONE, which <code>int</code> treats as equivalent to <code>'1'</code>. That's because <a href="https://github.com/python/cpython/blob/v3.6.3/Objects/longobject.c#L2500" rel="noreferrer">one of the earlier steps</a> in the string-to-int process is

<pre><code>asciidig = _PyUnicode_TransformDecimalAndSpaceToASCII(u);
</code></pre>

which converts all whitespace characters to <code>' '</code> and converts all Unicode decimal digits to the corresponding ASCII digits. This conversion returns the original string if it doesn't end up changing anything, but when it does make changes, it creates a new string, and the new string is the one that gets a UTF-8 representation created.

<hr>

As for the cases where calling <code>int</code> on one string looks like it affects another, those are actually the same string object. There are many conditions under which Python will reuse strings, all just as firmly in Weird Implementation Detail Land as everything we've discussed so far. For <code>'ñ'</code>, the reuse happens because this is a single-character string in the Latin-1 range (<code>'\x00'</code>-<code>'\xff'</code>), and the implementation <a href="https://github.com/python/cpython/blob/v3.6.3/Objects/unicodeobject.c#L213" rel="noreferrer">stores and reuses those</a>.

From the <a href="https://twitter.com/dabeaz/status/925787482515533830" rel="noreferrer">tweet here</a>:

<pre><code>import sys
x = 'ñ'
print(sys.getsizeof(x))
int(x) #throws an error
print(sys.getsizeof(x))
</code></pre>

We get 74, then 77 bytes for the two <code>getsizeof</code> calls.

It looks like we are adding 3 bytes to the object, from the failed int call.

Some more examples from twitter (you may need to restart python to reset the size back to 74):

<pre><code>x = 'ñ'
y = 'ñ'
int(x)
print(sys.getsizeof(y))
</code></pre>

77!

<pre><code>print(sys.getsizeof('ñ'))
int('ñ')
print(sys.getsizeof('ñ'))
</code></pre>

74, then 77.

Why does the size of this Python String change on a failed int conversion

Python

从import sysx = 'ñ'print(sys.getsizeof(x))int(x) #throws an errorprint(sys.getsizeof(x))对于两个getsizeof调用，我们得到74个字节，然后是77个字节。看起来我们从失败的int调用中向对象添加了3个字节。来自twitter的更多...

问为什么此Python字符串的大小在int转换失败时发生更改
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问为什么此Python字符串的大小在int转换失败时发生更改EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问为什么此Python字符串的大小在int转换失败时发生更改
EN