Python简易爬虫小实例：爬取NBA球队13-14赛季对阵数据

用户7010445

发布于 2020-03-02 16:39:36

7310

发布于 2020-03-02 16:39:36

之前浏览《Python数据挖掘入门与实践》这本书的时候发现了非常有意思的内容——用决策树预测NBA获胜球队，但是书中获得原始数据的方式已经行不通了，所以一直没有能够重复这一章的内容。恰巧最近发现了一个利用Python BeautifulSoup模块抓取NBA选秀数据的教程 Learning Python: Part 1:Scraping and Cleaning the NBA draft。突然意识到是否可以利用这份教程来抓取NBA球队的对阵数据，从而重复利用决策树越策NBA获胜球队的内容。

第一部分

这部分内容来自参考书《Python网络数据采集》第一章的内容基本流程：通过urlopen()函数获得网页的的全部HTML代码；然后通过BeautifulSoup模块解析HTML代码获得我们想要的内容

from urllib.request import urlopenurl = "http://pythonscraping.com/pages/page1.html"html = urlopen(url)print(html.read())

输出结果

b'<html>\n<head>\n<title>A Useful Page</title>\n</head>\n<body>\n<h1>An Interesting Title</h1>\n<div>\nLorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\n</div>\n</body>\n</html>\n'

简易理解html源代码：尖括号<>内是标签，两个尖括号中间是内容 BeautifulSoup解析

from bs4 import BeautifulSoupsoup = BeautifulSoup(html)

如果我们想要获得以上html源代码中title中的内容

soup.titlesoup.findAll("title")soup.title.getText()

第二部分

爬取2013-2014赛季NBA球队的对阵数据

from urllib.request import urlopenfrom bs4 import BeautifulSoupimport pandas as pdimport time
start = time.time()
months = ["october","november","december","january","february","march","april","may","june"]col_header = ["Date","Start(ET)","Visitor/Neutral","PTS","Home/Neutral","PTS","","","Attend","Notes"]url = "https://www.basketball-reference.com/leagues/NBA_2014_games-"
NBA_1314_Schedule_and_results = []for month in months:    urls = url + month + ".html"    html = urlopen(urls)    soup = BeautifulSoup(html,"lxml")    start_1 = time.time()    print(month)    for i in range(len(soup.tbody.findAll("tr"))):        Schedule = []        date = soup.tbody.findAll("tr")[i].findAll("th")[0].getText()        Schedule.append(date)        for j in range(len(soup.findAll("tr")[i].findAll("td"))):            data = soup.findAll("tr")[i].findAll("td")[j].getText()            Schedule.append(data)
        NBA_1314_Schedule_and_results.append(Schedule)    end_1 = time.time()    print(month,round(end_1 - start_1,2),"s")df = pd.DataFrame(NBA_1314_Schedule_and_results,columns = col_header)
end = time.time()print("The total time used:",round(end - start,2),"s")df.to_csv("NBA_2013_2014_Schedule_and_results.csv")

成功

部分结果