blocks|key|210047|text|通过添加更多的头文件，我能够获得数据：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|210048|import+urllib2,cookielib

site=+"http://www.nseindia.com/live_market/dynaContent/live_watch/get_quote/getHistoricalData.jsp?symbol=JPASSOCIAT&fromDate=1-JAN-2012&toDate=1-AUG-2012&datePeriod=unselected&hiddDwnld=true"
hdr+=+{'User-Agent':+'Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.11+(KHTML,+like+Gecko)+Chrome/23.0.1271.64+Safari/537.11',
+++++++'Accept':+'text/html,application/xhtml%2Bxml,application/xml;q=0.9,*/*;q=0.8',
+++++++'Accept-Charset':+'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
+++++++'Accept-Encoding':+'none',
+++++++'Accept-Language':+'en-US,en;q=0.8',
+++++++'Connection':+'keep-alive'}

req+=+urllib2.Request(site,+headers=hdr)

try:
++++page+=+urllib2.urlopen(req)
except+urllib2.HTTPError,+e:
++++print+e.fp.read()

content+=+page.read()
print+content|code-block|syntax|javascript|210049|实际上，它只使用这个额外的头：|210050|'Accept':+'text/html,application/xhtml%2Bxml,application/xml;q=0.9,*/*;q=0.8',|210051|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

By adding a few more headers I was able to get the data:

<pre><code>import urllib2,cookielib

site= "http://www.nseindia.com/live_market/dynaContent/live_watch/get_quote/getHistoricalData.jsp?symbol=JPASSOCIAT&amp;fromDate=1-JAN-2012&amp;toDate=1-AUG-2012&amp;datePeriod=unselected&amp;hiddDwnld=true"
hdr = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
 'Accept-Encoding': 'none',
 'Accept-Language': 'en-US,en;q=0.8',
 'Connection': 'keep-alive'}

req = urllib2.Request(site, headers=hdr)

try:
 page = urllib2.urlopen(req)
except urllib2.HTTPError, e:
 print e.fp.read()

content = page.read()
print content
</code></pre>

Actually, it works with just this one additional header:

<pre><code>'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
</code></pre>

blocks|key|2845902|text|这将在Python+3中运行|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2845903|import+urllib.request

user_agent+=+'Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.9.0.7)+Gecko/2009021910+Firefox/3.0.7'

url+=+"http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers"
headers={'User-Agent':user_agent,}+

request=urllib.request.Request(url,None,headers)+#The+assembled+request
response+=+urllib.request.urlopen(request)
data+=+response.read()+#+The+data+u+need|code-block|syntax|javascript|2845904|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

This will work in Python 3

<pre><code>import urllib.request

user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7'

url = "http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers"
headers={'User-Agent':user_agent,} 

request=urllib.request.Request(url,None,headers) #The assembled request
response = urllib.request.urlopen(request)
data = response.read() # The data u need
</code></pre>

blocks|key|205735|text|NSE网站已更改，较旧的脚本对当前网站是半优化的。此代码片段可以收集安全的日常详细信息。详细信息包括代码、证券类型、前一交易日收盘价、开盘价、高价、低价、均价、交易量、成交额、交易数量、交割数量以及交割与交易的比率。这些都以字典形式的列表形式方便地呈现。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|205736|支持请求和BeautifulSoup的Python3.x版本|205737|from+requests+import+get
from+csv+import+DictReader
from+bs4+import+BeautifulSoup+as+Soup
from+datetime+import+date
from+io+import+StringIO+

SECURITY_NAME="3MINDIA"+#+Change+this+to+get+quote+for+another+stock
START_DATE=+date(2017,+1,+1)+#+Start+date+of+stock+quote+data+DD-MM-YYYY
END_DATE=+date(2017,+9,+14)++#+End+date+of+stock+quote+data+DD-MM-YYYY


BASE_URL+=+"https://www.nseindia.com/products/dynaContent/common/productsSymbolMapping.jsp?symbol={security}&segmentLink=3&symbolCount=1&series=ALL&dateRange=%2B&fromDate={start_date}&toDate={end_date}&dataType=PRICEVOLUMEDELIVERABLE"




def+getquote(symbol,+start,+end):
++++start+=+start.strftime("%25-d-%25-m-%25Y")
++++end+=+end.strftime("%25-d-%25-m-%25Y")

++++hdr+=+{'User-Agent':+'Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.11+(KHTML,+like+Gecko)+Chrome/23.0.1271.64+Safari/537.11',
+++++++++'Accept':+'text/html,application/xhtml%2Bxml,application/xml;q=0.9,*/*;q=0.8',
+++++++++'Referer':+'https://cssspritegenerator.com',
+++++++++'Accept-Charset':+'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
+++++++++'Accept-Encoding':+'none',
+++++++++'Accept-Language':+'en-US,en;q=0.8',
+++++++++'Connection':+'keep-alive'}

++++url+=+BASE_URL.format(security=symbol,+start_date=start,+end_date=end)
++++d+=+get(url,+headers=hdr)
++++soup+=+Soup(d.content,+'html.parser')
++++payload+=+soup.find('div',+{'id':+'csvContentDiv'}).text.replace(':',+'\n')
++++csv+=+DictReader(StringIO(payload))
++++for+row+in+csv:
++++++++print({k:v.strip()+for+k,+v+in+row.items()})


+if+__name__+==+'__main__':
+++++getquote(SECURITY_NAME,+START_DATE,+END_DATE)|code-block|syntax|javascript|205738|此外，这是相对模块化的，并且可以使用snippet。|205739|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|N|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|O|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|P|8|@]|9|@]|A|$]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

NSE website has changed and the older scripts are semi-optimum to current website. This snippet can gather daily details of security. Details include symbol, security type, previous close, open price, high price, low price, average price, traded quantity, turnover, number of trades, deliverable quantities and ratio of delivered vs traded in percentage. These conveniently presented as list of dictionary form.

<h2>Python 3.X version with requests and BeautifulSoup</h2>

<pre><code>from requests import get
from csv import DictReader
from bs4 import BeautifulSoup as Soup
from datetime import date
from io import StringIO 

SECURITY_NAME="3MINDIA" # Change this to get quote for another stock
START_DATE= date(2017, 1, 1) # Start date of stock quote data DD-MM-YYYY
END_DATE= date(2017, 9, 14) # End date of stock quote data DD-MM-YYYY


BASE_URL = "https://www.nseindia.com/products/dynaContent/common/productsSymbolMapping.jsp?symbol={security}&amp;segmentLink=3&amp;symbolCount=1&amp;series=ALL&amp;dateRange=+&amp;fromDate={start_date}&amp;toDate={end_date}&amp;dataType=PRICEVOLUMEDELIVERABLE"




def getquote(symbol, start, end):
 start = start.strftime("%-d-%-m-%Y")
 end = end.strftime("%-d-%-m-%Y")

 hdr = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'Referer': 'https://cssspritegenerator.com',
 'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
 'Accept-Encoding': 'none',
 'Accept-Language': 'en-US,en;q=0.8',
 'Connection': 'keep-alive'}

 url = BASE_URL.format(security=symbol, start_date=start, end_date=end)
 d = get(url, headers=hdr)
 soup = Soup(d.content, 'html.parser')
 payload = soup.find('div', {'id': 'csvContentDiv'}).text.replace(':', '\n')
 csv = DictReader(StringIO(payload))
 for row in csv:
 print({k:v.strip() for k, v in row.items()})


 if __name__ == '__main__':
 getquote(SECURITY_NAME, START_DATE, END_DATE)
</code></pre>

Besides this is relatively modular and ready to use snippet.

blocks|key|2845929|text|import+urllib.request

bank_pdf_list+=+["https://www.hdfcbank.com/content/bbp/repositories/723fb80a-2dde-42a3-9793-7ae1be57c87f/?path=/Personal/Home/content/rates.pdf",
"https://www.yesbank.in/pdf/forexcardratesenglish_pdf",
"https://www.sbi.co.in/documents/16012/1400784/FOREX_CARD_RATES.pdf"]


def+get_pdf(url):
++++user_agent+=+'Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.9.0.7)+Gecko/2009021910+Firefox/3.0.7'
++++
++++#url+=+"https://www.yesbank.in/pdf/forexcardratesenglish_pdf"
++++headers={'User-Agent':user_agent,}+
++++
++++request=urllib.request.Request(url,None,headers)+#The+assembled+request
++++response+=+urllib.request.urlopen(request)
++++#print(response.text)
++++data+=+response.read()
#++++print(type(data))
++++
++++name+=+url.split("www.")[-1].split("//")[-1].split(".")[0]%2B"_FOREX_CARD_RATES.pdf"
++++f+=+open(name,+'wb')
++++f.write(data)
++++f.close()
++++

for+bank_url+in+bank_pdf_list:
++++try:+
++++++++get_pdf(bank_url)
++++except:
++++++++pass|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|2845930|unstyled|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|G|8|@]|9|@]|A|$B|C]]|$1|D|3|-4|5|E|7|H|8|@]|9|@]|A|$]]]|F|$]]

<pre><code>import urllib.request

bank_pdf_list = [&quot;https://www.hdfcbank.com/content/bbp/repositories/723fb80a-2dde-42a3-9793-7ae1be57c87f/?path=/Personal/Home/content/rates.pdf&quot;,
&quot;https://www.yesbank.in/pdf/forexcardratesenglish_pdf&quot;,
&quot;https://www.sbi.co.in/documents/16012/1400784/FOREX_CARD_RATES.pdf&quot;]


def get_pdf(url):
 user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7'
 
 #url = &quot;https://www.yesbank.in/pdf/forexcardratesenglish_pdf&quot;
 headers={'User-Agent':user_agent,} 
 
 request=urllib.request.Request(url,None,headers) #The assembled request
 response = urllib.request.urlopen(request)
 #print(response.text)
 data = response.read()
# print(type(data))
 
 name = url.split(&quot;www.&quot;)[-1].split(&quot;//&quot;)[-1].split(&quot;.&quot;)[0]+&quot;_FOREX_CARD_RATES.pdf&quot;
 f = open(name, 'wb')
 f.write(data)
 f.close()
 

for bank_url in bank_pdf_list:
 try: 
 get_pdf(bank_url)
 except:
 pass
</code></pre>

blocks|key|3058851|text|当您请求的服务器不知道请求来自何处时，通常会出现此错误，服务器这样做是为了避免任何不必要的访问。您可以通过定义一个标头并将其沿urllib.request传递来绕过此错误|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3058852|代码如下：|3058853|#defining+header
header=+{'User-Agent':+'Mozilla/5.0+(X11;+Linux+x86_64)+'+
++++++'AppleWebKit/537.11+(KHTML,+like+Gecko)+'
++++++'Chrome/23.0.1271.64+Safari/537.11',
++++++'Accept':+'text/html,application/xhtml%2Bxml,application/xml;q=0.9,*/*;q=0.8',
++++++'Accept-Charset':+'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
++++++'Accept-Encoding':+'none',
++++++'Accept-Language':+'en-US,en;q=0.8',
++++++'Connection':+'keep-alive'}

#the+URL+where+you+are+requesting+at
req+=+urllib.request.Request(url=your_url,+headers=header)+
page+=+urllib.request.urlopen(req).read()|code-block|syntax|javascript|3058854|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|L|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|M|8|@]|9|@]|A|$G|H]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

This error usually occurs when the server you are requesting doesn't know where the request is coming from, the server does this to avoid any unwanted visit. You could bypass this error by defining a header and passing it along the urllib.request
Heres code:
<pre><code>#defining header
header= {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) ' 
 'AppleWebKit/537.11 (KHTML, like Gecko) '
 'Chrome/23.0.1271.64 Safari/537.11',
 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
 'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
 'Accept-Encoding': 'none',
 'Accept-Language': 'en-US,en;q=0.8',
 'Connection': 'keep-alive'}

#the URL where you are requesting at
req = urllib.request.Request(url=your_url, headers=header) 
page = urllib.request.urlopen(req).read()
</code></pre>

blocks|key|205778|text|有一件事值得一试，那就是更新python版本。几个月前，我的一个爬虫脚本在Windows10上停止了403的工作。任何user_agents都无济于事，我正要放弃这个脚本。今天我在Ubuntu上用Python+(3.8.5+-+64位)尝试了同样的脚本，没有错误。python版本的Windows有点老，只有3.6.2+-+32位。在将Windows10上的python升级到3.9.5+-+64bit之后，我再也看不到403了。如果你试过了，别忘了运行'pip+freeze+>requirements.txt‘来导出包条目。我当然忘了。这篇文章也提醒我，当403以后再来的时候。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|205779|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

There is one thing worth trying is just to update the python version. One of my crawling scripts stopped working with 403 on Windows 10 a few months back. Any user_agents did not help and I was about to give up the script. Today I tried the same script on Ubuntu with Python (3.8.5 - 64 bit) and it worked with no error. The python version of Windows was a bit old as 3.6.2 - 32 bit. After upgrading the python on Windows 10 to 3.9.5 - 64bit, I don't see the 403 any longer. If you give it a try, don't forget to run 'pip freeze &gt; requirements.txt&quot; to export package entries. I forgot it of course.
This post is a reminder for me too when the 403 comes back again in the future.

I am trying to automate download of historic stock data using python. The URL I am trying to open responds with a CSV file, but I am unable to open using urllib2. I have tried changing user agent as specified in few questions earlier, I even tried to accept response cookies, with no luck. Can you please help. 

Note: The same method works for yahoo Finance.

Code:

<pre><code>import urllib2,cookielib

site= "http://www.nseindia.com/live_market/dynaContent/live_watch/get_quote/getHistoricalData.jsp?symbol=JPASSOCIAT&amp;fromDate=1-JAN-2012&amp;toDate=1-AUG-2012&amp;datePeriod=unselected&amp;hiddDwnld=true"

hdr = {'User-Agent':'Mozilla/5.0'}

req = urllib2.Request(site,headers=hdr)

page = urllib2.urlopen(req)
</code></pre>

Error

<blockquote>
 File "C:\Python27\lib\urllib2.py", line 527, in http_error_default
 raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 403: Forbidden
</blockquote>

Thanks for your assistance

urllib2.HTTPError: HTTP Error 403: Forbidden

服务器

Windows

Python

Ubuntu

Linux

我正在尝试使用python自动下载历史股票数据。我尝试打开的网址响应为CSV文件，但我无法使用urllib2打开。我尝试过按照前面几个问题中的说明更改用户代理，我甚至尝试接受响应cookie，但没有成功。你能帮帮我吗。注意:同样的方法也适用于雅虎财经。代码：import urllib2,cookielibsite= "...

问urllib2.HTTPError: HTTP错误403:禁止
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问urllib2.HTTPError: HTTP错误403:禁止EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问urllib2.HTTPError: HTTP错误403:禁止
EN