如何停止重定向并停留在页面上以抓取数据(python3、漂亮的汤、urllib)
我正在尝试从一个网站上获取评论：http://movie.douban.com/subject/2303845/comments
显然，它一直在将自己重定向回主页。我想找到一种方法来停止重定向，并停留在相同的页面上，以获得html中具有div和注释的部分作为其类。
关键词: Python3，美人汤，urllib
我已经读过其他帖子了，下面是我尝试过的：
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys
from bs4 import BeautifulSoup
from urllib.request import urlopen
sys.setrecursionlimit(30000)
try:
import urllib.request as urllib2
import urllib.request as urllib
except ImportError:
import urllib2
class NoRedirectHandler(urllib2.HTTPRedirectHandler):
def http_error_302(self, req, fp, code, msg, headers):
infourl = urllib.addinfourl(fp, headers, req.get_full_url())
infourl.status = code
infourl.code = code
return infourl
http_error_300 = http_error_302
http_error_301 = http_error_302
http_error_303 = http_error_302
http_error_307 = http_error_302
opener = urllib2.build_opener(NoRedirectHandler())
urllib2.install_opener(opener)
response = urllib2.urlopen('http://movie.douban.com/subject/2303845/comments')
soup = BeautifulSoup(response, "html.parser")
comments = soup.findAll('div', {'class': 'comment'})
print (comments)
if response.code in (300, 301, 302, 303, 307):
print ('redirect')由于某种原因，我认为它仍然没有获得数据，我怀疑它从未停止重定向到另一个页面。
回答 2
Stack Overflow用户
发布于 2017-02-20 16:24:25
将cookie添加到标题中，类似于(只需检查实际浏览器设置为豆瓣的内容)：
headers = { "User-Agent": random_user_agent(), "Cookie": 'bid="' + "".join(random.sample(string.ascii_letters + string.digits, 11)) + '"; gr_user_id=42bc6859-9086-4933-9f19-8f2fc412c37a; viewed="5312099_10355142_4742942_3223711_26642036_25850008_4727293_10580248_3726306"; _ga=GA1.2.1107723060.1458721162; ll="108288"; ps=y; ue="276502214@qq.com"; dbcl2="48221338:qY8I57pvNs8"; ck=xvAF; ap=1; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1487574236%2C%22https%3A%2F%2Fwww.douban.com%2Faccounts%2Flogin%3Fredir%3Dhttps%253A%252F%252Fmovie.douban.com%252Fsubject%252F2973543%252F%22%5D; _pk_id.100001.4cf6=ccee4c5d3437a382.1482239227.55.1487574252.1487571075.; _pk_ses.100001.4cf6=*; __utma=30149280.1107723060.1458721162.1487570089.1487574236.345; __utmb=30149280.0.10.1487574236; __utmc=30149280; __utmz=30149280.1487518823.338.100.utmcsr=baidu|utmccn=(organic)|utmcmd=organic; __utmv=30149280.4822; __utma=223695111.1107723060.1458721162.1487570089.1487574236.52; __utmb=223695111.0.10.1487574236; __utmc=223695111; __utmz=223695111.1487298209.35.8.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/accounts/login; push_noty_num=0; push_doumail_num=0; _vwo_uuid_v2=2D161B768A99F5F485C8FFD318CE8704|b630fad39ce1f3cbdb2e7e44cc9411ca' } req = urllib.request.Request( url, data = None, headers = headers )
Stack Overflow用户
发布于 2015-10-20 04:48:29
要避免重定向，请将您的url更改为：
http://movie.douban.com/subject/2303845/?from=showing/comments从…
http://movie.douban.com/subject/2303845/comments我在代码中更改了这一行，并获得了一个包含4个<div class="comment">条目的数组
您还可以使用以下命令打印注释：
for comment in comments:
print(comment.get_text())https://stackoverflow.com/questions/33223038
复制相似问题
腾讯云开发者
