如何使用Scrapy和Splash处理分页，如果按钮的href为javascript:void(0)
如何使用Scrapy和Splash处理分页，如果按钮的href为javascript:void(0)
提问于 2022-05-12 20:48:26
我正试图从这个网站：https://www.topuniversities.com/university-rankings/world-university-rankings/2021中抓取大学的名称和链接，并且在处理分页时遇到了一个问题，因为指向下一页的按钮的href是javascript:void(0)，所以我无法用scrapy.Request()或response.follow()到达下一页，有什么方法可以这样处理分页吗？
此网站的URL不包含params，如果单击下一个页面按钮，URL将保持不变，因此我无法通过更改URL来处理分页。
下面的代码片段只能在第一页和第二页获取大学的名称和链接：
import scrapy
from scrapy_splash import SplashRequest
class UniSpider(scrapy.Spider):
name = 'uni'
allowed_domains = ['www.topuniversities.com']
script = """
function main(splash, args)
splash:set_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36")
splash.private_mode_enabled = false
assert(splash:go(args.url))
assert(splash:wait(3))
return {
html = splash:html()
}
end
"""
next_page = """
function main(splash, args)
splash:set_user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36")
splash.private_mode_enabled = false
assert(splash:go(args.url))
assert(splash:wait(3))
local btn = assert(splash:jsfunc([[
function(){
document.querySelector("#alt-style-pagination a.page-link.next").click()
}
]]))
assert(splash:wait(2))
btn()
splash:set_viewport_full()
assert(splash:wait(3))
return {
html = splash:html()
}
end
"""
def start_requests(self):
yield SplashRequest(
url="https://www.topuniversities.com/university-rankings/world-university-rankings/2021",
callback=self.parse, endpoint="execute",
args={"lua_source": self.script})
def parse(self, response):
for uni in response.css("a.uni-link"):
uni_link = response.urljoin(uni.css("::attr(href)").get())
yield {
"name": uni.css("::text").get(),
"link": uni_link
}
yield SplashRequest(
url=response.url,
callback=self.parse, endpoint="execute",
args={"lua_source": self.next_page}
)回答 1
Stack Overflow用户
回答已采纳
发布于 2022-05-13 07:40:52
你不需要花这个简单的网站。
尝试加载以下链接：
https://www.topuniversities.com/sites/default/files/qs-rankings-data/en/2057712.txt
这有所有的大学，网站只加载这个文件/json一次，然后显示信息与分页。
下面是简短的代码(不使用scrapy)：
from requests import get
from json import loads, dumps
from lxml.html import fromstring
url = "https://www.topuniversities.com/sites/default/files/qs-rankings-data/en/2057712.txt"
html = get(url, stream=True)
## another approach for loading json
# jdata = loads(html.content.decode())
jdata = html.json()
for x in jdata['data']:
core_id = x['core_id']
country = x['country']
city = x['city']
guide = x['guide']
nid = x['nid']
title = x['title']
logo = x['logo']
score = x['score']
rank_display = x['rank_display']
region = x['region']
stars = x['stars']
recm = x['recm']
dagger = x['dagger']
## convert title to text
soup = fromstring(title)
title = soup.xpath(".//a/text()")[0]
print ( title )以上代码打印个别大学的“标题”，尝试将其与其他可用列一起保存在CSV/Excel文件中。结果如下：
Massachusetts Institute of Technology (MIT)
Stanford University
Harvard University
California Institute of Technology (Caltech)
University of Oxford
ETH Zurich - Swiss Federal Institute of Technology
University of Cambridge
Imperial College London页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/72221908
复制相关文章
相似问题
腾讯云开发者
