blocks|key|760385|text|将unicode编码为UTF-8，然后将URL-+Encode+.|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|760386|entityMap^0|1|7|0^^$0|@$1|2|3|4|5|6|7|H|8|@$9|I|A|J|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|K|8|@]|D|@]|E|$]]]|G|$]]

Encode the <code>unicode</code> to UTF-8, then URL-encode.

blocks|key|1958230|text|严格地说，URI不能包含非ASCII字符；您拥有的是一个IRI。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1958231|若要将IRI转换为普通的ASCII+URI，请执行以下操作：|1958232|地址主机名部分中的非ASCII字符必须使用旁遮普码-based+IDNA算法进行编码；|unordered-list-item|1958233|路径中的非ASCII字符以及地址的大部分其他部分必须按照Ignacio的答案使用UTF-8和%25-编码进行编码。|1958234|所以：|1958235|import+re,+urlparse

def+urlEncodeNonAscii(b):
++++return+re.sub('[\x80-\xFF]',+lambda+c:+'%25%25%2502x'+%25+ord(c.group(0)),+b)

def+iriToUri(iri):
++++parts=+urlparse.urlparse(iri)
++++return+urlparse.urlunparse(
++++++++part.encode('idna')+if+parti==1+else+urlEncodeNonAscii(part.encode('utf-8'))
++++++++for+parti,+part+in+enumerate(parts)
++++)

>>>+iriToUri(u'http://www.a\u0131b.com/a\u0131b')
'http://www.xn--ab-hpa.com/a%25c4%25b1b'|code-block|syntax|javascript|1958236|(从技术上讲，这在一般情况下仍然不够好，因为urlparse不会分割主机名上的任何user:pass@前缀或:port后缀。只有主机名部分应该被编码。在构建URL时使用普通的urllib.quote和.encode('idna')进行编码比将IRI分开要容易得多。)|style|CODE|1958237|entityMap|0|LINK|mutability|MUTABLE|url|http://en.wikipedia.org/wiki/Internationalized_Resource_Identifier|1|http://en.wikipedia.org/wiki/Punycode^0|S|3|0|0|0|L|4|1|0|0|0|0|M|8|15|A|1I|5|2F|C|2S|F|0^^$0|@$1|2|3|4|5|6|7|15|8|@]|9|@$A|16|B|17|1|18]]|C|$]]|$1|D|3|E|5|6|7|19|8|@]|9|@]|C|$]]|$1|F|3|G|5|H|7|1A|8|@]|9|@$A|1B|B|1C|1|1D]]|C|$]]|$1|I|3|J|5|H|7|1E|8|@]|9|@]|C|$]]|$1|K|3|L|5|6|7|1F|8|@]|9|@]|C|$]]|$1|M|3|N|5|O|7|1G|8|@]|9|@]|C|$P|Q]]|$1|R|3|S|5|6|7|1H|8|@$A|1I|B|1J|T|U]|$A|1K|B|1L|T|U]|$A|1M|B|1N|T|U]|$A|1O|B|1P|T|U]|$A|1Q|B|1R|T|U]]|9|@]|C|$]]|$1|V|3|-4|5|6|7|1S|8|@]|9|@]|C|$]]]|W|$X|$5|Y|Z|10|C|$11|12]]|13|$5|Y|Z|10|C|$11|14]]]]

Strictly speaking URIs can't contain non-ASCII characters; what you have there is an <a href="http://en.wikipedia.org/wiki/Internationalized_Resource_Identifier">IRI</a>.

To convert an IRI to a plain ASCII URI:

<ul>
<li>non-ASCII characters in the hostname part of the address have to be encoded using the <a href="http://en.wikipedia.org/wiki/Punycode">Punycode</a>-based IDNA algorithm;</li>
<li>non-ASCII characters in the path, and most of the other parts of the address have to be encoded using UTF-8 and %-encoding, as per Ignacio's answer.</li>
</ul>

So:

<pre><code>import re, urlparse

def urlEncodeNonAscii(b):
 return re.sub('[\x80-\xFF]', lambda c: '%%%02x' % ord(c.group(0)), b)

def iriToUri(iri):
 parts= urlparse.urlparse(iri)
 return urlparse.urlunparse(
 part.encode('idna') if parti==1 else urlEncodeNonAscii(part.encode('utf-8'))
 for parti, part in enumerate(parts)
 )

&gt;&gt;&gt; iriToUri(u'http://www.a\u0131b.com/a\u0131b')
'http://www.xn--ab-hpa.com/a%c4%b1b'
</code></pre>

(Technically this still isn't quite good enough in the general case because <code>urlparse</code> doesn't split away any <code>user:pass@</code> prefix or <code>:port</code> suffix on the hostname. Only the hostname part should be IDNA encoded. It's easier to encode using normal <code>urllib.quote</code> and <code>.encode('idna')</code> at the time you're constructing a URL than to have to pull an IRI apart.)

blocks|key|2315266|text|使用iri2uri方法的httplib2。这和bobin做的一样(他/她是那个的作者吗？)|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2315267|entityMap^0|2|7|C|8|0^^$0|@$1|2|3|4|5|6|7|H|8|@$9|I|A|J|B|C]|$9|K|A|L|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|M|8|@]|D|@]|E|$]]]|G|$]]

Use <code>iri2uri</code> method of <code>httplib2</code>. It makes the same thing as by bobin (is he/she the author of that?)

blocks|key|2315272|text|Python+3有处理这种情况的库。使用urllib.parse.urlsplit将URL拆分为其组件，使用urllib.parse.quote正确引用/转义unicode字符，并使用urllib.parse.urlunsplit将其重新连接起来。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2315273|>>>+import+urllib.parse
>>>+url+=+'http://example.com/unicodè'
>>>+url+=+urllib.parse.urlsplit(url)
>>>+url+=+list(url)
>>>+url[2]+=+urllib.parse.quote(url[2])
>>>+url+=+urllib.parse.urlunsplit(url)
>>>+print(url)
http://example.com/unicod%25C3%25A8|code-block|syntax|javascript|2315274|entityMap^0|K|L|1I|I|2K|N|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@$9|N|A|O|B|C]|$9|P|A|Q|B|C]|$9|R|A|S|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|T|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|U|8|@]|D|@]|E|$]]]|L|$]]

Python 3 has libraries to handle this situation. Use
<code>urllib.parse.urlsplit</code> to split the URL into its components, and
<code>urllib.parse.quote</code> to properly quote/escape the unicode characters
and <code>urllib.parse.urlunsplit</code> to join it back together.

<pre><code>&gt;&gt;&gt; import urllib.parse
&gt;&gt;&gt; url = 'http://example.com/unicodè'
&gt;&gt;&gt; url = urllib.parse.urlsplit(url)
&gt;&gt;&gt; url = list(url)
&gt;&gt;&gt; url[2] = urllib.parse.quote(url[2])
&gt;&gt;&gt; url = urllib.parse.urlunsplit(url)
&gt;&gt;&gt; print(url)
http://example.com/unicod%C3%A8
</code></pre>

blocks|key|1958284|text|在python3中，对非ascii字符串使用urllib.parse.quote函数：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1958285|>>>+from+urllib.request+import+urlopen++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>+from+urllib.parse+import+quote++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>+chinese_wikipedia+=+'http://zh.wikipedia.org/wiki/Wikipedia:'+%2B+quote('首页')
>>>+urlopen(chinese_wikipedia)|code-block|syntax|javascript|1958286|entityMap^0|M|I|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@$9|N|A|O|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|P|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|Q|8|@]|D|@]|E|$]]]|L|$]]

In python3, use the <code>urllib.parse.quote</code> function on the non-ascii string:

<pre><code>&gt;&gt;&gt; from urllib.request import urlopen 
&gt;&gt;&gt; from urllib.parse import quote 
&gt;&gt;&gt; chinese_wikipedia = 'http://zh.wikipedia.org/wiki/Wikipedia:' + quote('首页')
&gt;&gt;&gt; urlopen(chinese_wikipedia)
</code></pre>

blocks|key|1958352|text|对于那些不严格依赖urllib的人来说，一个实用的选择是请求，它可以“开箱即用”来处理IRIs。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1958353|例如，使用http://bücher.ch|style|CODE|1958354|>>>+import+requests
>>>+r+=+requests.get(u'http://b\u00DCcher.ch')
>>>+r.status_code
200|code-block|syntax|javascript|1958355|entityMap|0|LINK|mutability|MUTABLE|url|http://docs.python-requests.org/en/master/^0|S|2|0|0|5|G|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@]|9|@$A|V|B|W|1|X]]|C|$]]|$1|D|3|E|5|6|7|Y|8|@$A|Z|B|10|F|G]]|9|@]|C|$]]|$1|H|3|I|5|J|7|11|8|@]|9|@]|C|$K|L]]|$1|M|3|-4|5|6|7|12|8|@]|9|@]|C|$]]]|N|$O|$5|P|Q|R|C|$S|T]]]]

For those not depending strictly on urllib, one practical alternative is <a href="http://docs.python-requests.org/en/master/" rel="noreferrer">requests</a>, which handles IRIs "out of the box".

For example, with <code>http://bücher.ch</code>:

<pre><code>&gt;&gt;&gt; import requests
&gt;&gt;&gt; r = requests.get(u'http://b\u00DCcher.ch')
&gt;&gt;&gt; r.status_code
200
</code></pre>

blocks|key|1958394|text|这比公认的“波文斯”的回答要复杂得多：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1958395|netloc应该使用IDNA进行编码；|unordered-list-item|1958396|非ascii+URL路径应该编码到UTF-8，然后百分比-转义；|1958397|非ascii查询参数应编码到编码的页面URL被提取(或编码服务器使用)，然后百分比转义。|1958398|这是所有浏览器的工作方式；它是在https://url.spec.whatwg.org/中指定的--参见此示例。可以在w3lib中找到Python实现(这是Scrapy正在使用的库)；请参见字符串|offset|length|1958399|from+w3lib.url+import+safe_url_string
url+=+safe_url_string(u'http://example.org/Ñöñ-ÅŞÇİİ/',+encoding="<page+encoding>")|code-block|syntax|javascript|1958400|检查URL转义实现是否不正确/不完整的一个简单方法是检查它是否提供“页面编码”参数。|1958401|entityMap|0|LINK|mutability|MUTABLE|url|https://url.spec.whatwg.org/|1|https://url.spec.whatwg.org/#query-encoding-example|2|https://github.com/scrapy/w3lib/blob/5918d0e467279195ca41257ad0ba4037c98a34e2/w3lib/url.py#L71^0|0|0|0|0|G|S|0|1H|2|1|2N|3|2|0|0|0^^$0|@$1|2|3|4|5|6|7|15|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|16|8|@]|9|@]|A|$]]|$1|E|3|F|5|D|7|17|8|@]|9|@]|A|$]]|$1|G|3|H|5|D|7|18|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|19|8|@]|9|@$K|1A|L|1B|1|1C]|$K|1D|L|1E|1|1F]|$K|1G|L|1H|1|1I]]|A|$]]|$1|M|3|N|5|O|7|1J|8|@]|9|@]|A|$P|Q]]|$1|R|3|S|5|6|7|1K|8|@]|9|@]|A|$]]|$1|T|3|-4|5|6|7|1L|8|@]|9|@]|A|$]]]|U|$V|$5|W|X|Y|A|$Z|10]]|11|$5|W|X|Y|A|$Z|12]]|13|$5|W|X|Y|A|$Z|14]]]]

It is more complex than the accepted @bobince's answer suggests:

<ul>
<li>netloc should be encoded using IDNA;</li>
<li>non-ascii URL path should be encoded to UTF-8 and then percent-escaped;</li>
<li>non-ascii query parameters should be encoded to the encoding of a page URL was extracted from (or to the encoding server uses), then percent-escaped.</li>
</ul>

This is how all browsers work; it is specified in <a href="https://url.spec.whatwg.org/" rel="noreferrer">https://url.spec.whatwg.org/</a> - see this <a href="https://url.spec.whatwg.org/#query-encoding-example" rel="noreferrer">example</a>. A Python implementation can be found in w3lib (this is the library Scrapy is using); see <a href="https://github.com/scrapy/w3lib/blob/5918d0e467279195ca41257ad0ba4037c98a34e2/w3lib/url.py#L71" rel="noreferrer">w3lib.url.safe_url_string</a>:

<pre><code>from w3lib.url import safe_url_string
url = safe_url_string(u'http://example.org/Ñöñ-ÅŞÇİİ/', encoding="&lt;page encoding&gt;")
</code></pre>

An easy way to check if a URL escaping implementation is incorrect/incomplete is to check if it provides 'page encoding' argument or not.

blocks|key|760603|text|基于@深色猫科动物的答案：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|760604|from+urllib.parse+import+urlsplit,+urlunsplit,+quote

def+iri2uri(iri):
++++"""
++++Convert+an+IRI+to+a+URI+(Python+3).
++++"""
++++uri+=+''
++++if+isinstance(iri,+str):
++++++++(scheme,+netloc,+path,+query,+fragment)+=+urlsplit(iri)
++++++++scheme+=+quote(scheme)
++++++++netloc+=+netloc.encode('idna').decode('utf-8')
++++++++path+=+quote(path)
++++++++query+=+quote(query)
++++++++fragment+=+quote(fragment)
++++++++uri+=+urlunsplit((scheme,+netloc,+path,+query,+fragment))

++++return+uri|code-block|syntax|javascript|760605|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Based on @darkfeline answer:

 

<pre><code>from urllib.parse import urlsplit, urlunsplit, quote

def iri2uri(iri):
 """
 Convert an IRI to a URI (Python 3).
 """
 uri = ''
 if isinstance(iri, str):
 (scheme, netloc, path, query, fragment) = urlsplit(iri)
 scheme = quote(scheme)
 netloc = netloc.encode('idna').decode('utf-8')
 path = quote(path)
 query = quote(query)
 fragment = quote(fragment)
 uri = urlunsplit((scheme, netloc, path, query, fragment))

 return uri
</code></pre>

blocks|key|760625|text|行得通！最后|type|unstyled|depth|inlineStyleRanges|entityRanges|data|760626|我无法逃避这个奇怪的角色，但最后我还是挺过来了。|760627|import+urllib.request
import+os


url+=+"http://www.fourtourismblog.it/le-nuove-tendenze-del-marketing-tenere-docchio/"
with+urllib.request.urlopen(url)+as+file:
++++html+=+file.read()
with+open("marketingturismo.html",+"w",+encoding='utf-8')+as+file:
++++file.write(str(html.decode('utf-8')))
os.system("marketingturismo.html")|code-block|syntax|javascript|760628|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|L|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|M|8|@]|9|@]|A|$G|H]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

<h2>works! finally</h2>

I could not avoid from this strange characters, but at the end I come through it.

<pre><code>import urllib.request
import os


url = "http://www.fourtourismblog.it/le-nuove-tendenze-del-marketing-tenere-docchio/"
with urllib.request.urlopen(url) as file:
 html = file.read()
with open("marketingturismo.html", "w", encoding='utf-8') as file:
 file.write(str(html.decode('utf-8')))
os.system("marketingturismo.html")
</code></pre>

blocks|key|2315588|text|将IRI转换为ASCII+URI的另一个选项是使用furl包：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2315589|gruns/furl:+URL解析和操作变得容易了。-+https://github.com/gruns/furl|2315590|Python的标准urllib和URL解析模块提供了许多与URL相关的函数，但是使用这些函数来执行常见的URL操作却很繁琐。Furl使解析和操作URL变得容易。|blockquote|2315591|示例|2315592|非ASCII域|2315593|http://国立極地研究所.jp/english/+(日本国家极地研究所网站)|2315594|import+furl

url+=+'http://国立極地研究所.jp/english/'
furl.furl(url).tostr()|code-block|syntax|javascript|2315595|'http://xn--vcsoey76a2hh0vtuid5qa.jp/english/'|2315596|非ASCII路径|2315597|https://ja.wikipedia.org/wiki/日本語+(维基百科中的日语文章)|2315598|import+furl

url+=+'https://ja.wikipedia.org/wiki/日本語'
furl.furl(url).tostr()|2315599|'https://ja.wikipedia.org/wiki/%25E6%2597%25A5%25E6%259C%25AC%25E8%25AA%259E'|2315600|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/gruns/furl|1|http://xn--vcsoey76a2hh0vtuid5qa.jp/english/|2|https://ja.wikipedia.org/wiki/%25E6%2597%25A5%25E6%259C%25AC%25E8%25AA%259E^0|P|4|0|S|T|0|0|0|0|0|0|Q|1|0|0|0|0|0|X|2|0|0|0^^$0|@$1|2|3|4|5|6|7|1H|8|@$9|1I|A|1J|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|1K|8|@]|D|@$9|1L|A|1M|1|1N]]|E|$]]|$1|H|3|I|5|J|7|1O|8|@]|D|@]|E|$]]|$1|K|3|L|5|6|7|1P|8|@]|D|@]|E|$]]|$1|M|3|N|5|6|7|1Q|8|@]|D|@]|E|$]]|$1|O|3|P|5|6|7|1R|8|@]|D|@$9|1S|A|1T|1|1U]]|E|$]]|$1|Q|3|R|5|S|7|1V|8|@]|D|@]|E|$T|U]]|$1|V|3|W|5|S|7|1W|8|@]|D|@]|E|$T|U]]|$1|X|3|Y|5|6|7|1X|8|@]|D|@]|E|$]]|$1|Z|3|10|5|6|7|1Y|8|@]|D|@$9|1Z|A|20|1|21]]|E|$]]|$1|11|3|12|5|S|7|22|8|@]|D|@]|E|$T|U]]|$1|13|3|14|5|S|7|23|8|@]|D|@]|E|$T|U]]|$1|15|3|-4|5|6|7|24|8|@]|D|@]|E|$]]]|16|$17|$5|18|19|1A|E|$1B|1C]]|1D|$5|18|19|1A|E|$1B|1E]]|1F|$5|18|19|1A|E|$1B|1G]]]]

Another option to convert an IRI to an ASCII URI is to use <code>furl</code> package:
gruns/furl: URL parsing and manipulation made easy. - <a href="https://github.com/gruns/furl" rel="nofollow noreferrer">https://github.com/gruns/furl</a>
<blockquote>
Python's standard urllib and urlparse modules provide a number of URL
related functions, but using these functions to perform common URL
operations proves tedious. Furl makes parsing and manipulating URLs
easy.
</blockquote>
<h2>Examples</h2>
<h3>Non-ASCII domain</h3>
<a href="http://xn--vcsoey76a2hh0vtuid5qa.jp/english/" rel="nofollow noreferrer">http://国立極地研究所.jp/english/</a> (Japanese National Institute of Polar Research website)
<pre class="lang-py prettyprint-override"><code>import furl

url = 'http://国立極地研究所.jp/english/'
furl.furl(url).tostr()
</code></pre>
<pre><code>'http://xn--vcsoey76a2hh0vtuid5qa.jp/english/'
</code></pre>
<h3>Non-ASCII path</h3>
<a href="https://ja.wikipedia.org/wiki/%E6%97%A5%E6%9C%AC%E8%AA%9E" rel="nofollow noreferrer">https://ja.wikipedia.org/wiki/日本語</a> (&quot;Japanese&quot; article in Wikipedia)
<pre class="lang-py prettyprint-override"><code>import furl

url = 'https://ja.wikipedia.org/wiki/日本語'
furl.furl(url).tostr()
</code></pre>
<pre><code>'https://ja.wikipedia.org/wiki/%E6%97%A5%E6%9C%AC%E8%AA%9E'
</code></pre>

I need to fetch data from a URL with non-ascii characters but urllib2.urlopen refuses to open the resource and raises:

<pre><code>UnicodeEncodeError: 'ascii' codec can't encode character u'\u0131' in position 26: ordinal not in range(128)
</code></pre>

I know the URL is not standards compliant but I have no chance to change it. 

What is the way to access a resource pointed by a URL containing non-ascii characters using Python?

edit: In other words, can / how urlopen open a URL like:

<pre><code>http://example.org/Ñöñ-ÅŞÇİİ/
</code></pre>

How to fetch a non-ascii url with urlopen?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我需要从具有非ascii字符的URL中获取数据，但是urllib2.urlopen拒绝打开资源并引发：UnicodeEncodeError: 'ascii' codec can't encode character u'\u0131' in position 26: ordinal not in range(128)我...

问如何用urlopen获取非ascii url？
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何用urlopen获取非ascii url？EN