使用selenium爬取京东商品信息

北山啦

发布于 2022-11-27 11:09:47

3930

发布于 2022-11-27 11:09:47

任务要求：用selenium爬取京东商城某一款华为手机的评论，要求至少爬取2000条用户名和用户评论，网址为https://www.jd.com/，将这些信息存入Excel文件中，文件后缀为.csv。将ipynb文件和后缀为.csv的文件压缩打包

通过博客对selenium的简单介绍，现在开始实战啦，没有看过的，可以先看看

直接上代码

"""
Created on Thu Dec 10 14:15:14 2020

@author: kun
"""
from selenium import webdriver
import pandas as pd
from time import sleep
from random import uniform

driver = webdriver.Chrome(r'M:\chromedriver.exe')
driver.maximize_window()
driver.get("https://www.jd.com/")
driver.implicitly_wait(10)
name,price = [],[]
driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")  
def get_info(thing):
    driver.find_element_by_xpath('//*[@id="key"]').send_keys(thing)
    driver.find_element_by_xpath('//*[@id="search"]/div/div[2]/button').click()
    sleep(uniform(2,3))
    driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")  
    sleep(uniform(2,3))
    try:
        for i in driver.find_elements_by_css_selector('ul[class="gl-warp clearfix"]>li>div>div>a>em'):
            if len(i.text)!=0:
                name.append(i.text)
        for i in driver.find_elements_by_css_selector('ul[class="gl-warp clearfix"]>li>div[class="gl-i-wrap"]>div[class="p-price"]>strong'):
            if len(i.text)!=0:
                price.append(i.text)
    except:
        pass
    """实现翻页"""
    if driver.page_source.find("pn-next disabled") == -1:
        driver.find_element_by_class_name("pn-next").click()

if __name__ == "__main__":
    get_info("5G手机")
    data = pd.DataFrame({'name':name,'price':price})
    data.to_excel("jd.xlsx",index=False)

结果如下：