利用美容汤和代理实现亚马逊数据的__Scrape__
利用美容汤和代理实现亚马逊数据的__Scrape__
提问于 2020-08-10 09:54:38
我正在尝试从亚马逊的列表中抓取。但我的委托书似乎有一些错误。下面是我尝试过的代码：
import requests
from bs4 import BeautifulSoup
url="https://www.amazon.com/Kindle-Paperwhite-Essentials-Bundle-including/dp/B0898ZK226/ref=sr_1_3?dchild=1&keywords=kindle&qid=1597051671&sr=8-3"
headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"}
proxyDict = {
"http": "http://10.10.1.10:3128",
"https": "http://10.10.1.10:1080",
"ftp" : "ftp://10.10.1.10:3128"
}
page = requests.get(url, headers=headers, proxies=proxyDict) #Error here.
print(page.status_code)
soup = BeautifulSoup(page.content, "html.parser")
title = soup.find(id = "productTitle")
if title:
title = title.get_text().strip()
else:
title = "Title: Error 404"
print(title)在没有代理的情况下，代码在输出中显示None。
回答 1
Stack Overflow用户
发布于 2020-08-10 15:07:58
以前的代码不能刮 JavaScript。以下是正确的代码：
from bs4 import BeautifulSoup
import requests
from requests_html import HTMLSession
# create an HTML Session object
session = HTMLSession()
# Use the object above to connect to needed webpage
resp = session.get("https://www.amazon.com/Sceptre-E248W-19203R-Monitor-Speakers-Metallic/dp/B0773ZY26F/ref=sr_1_2?crid=1861TM8A5NDPX&dchild=1&keywords=monitors&qid=1597071906&sprefix=monitors%2Caps%2C364&sr=8-2")
# Run JavaScript code on webpage
resp.html.render()
soup = BeautifulSoup(resp.html.html, "lxml")
title = soup.find(id = "productTitle").get_text().strip()
print (title)页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/63337938
复制相关文章
相似问题
腾讯云开发者
