我正在使用BeautifulSoup在Python中进行一些webscraping,其中涉及访问大约500个类似的网页,以便将数据放入.txt文件中。
但是,我遇到了一些问题:
def getAnime():
for index in range(2, 502):
# gets anime statistics from HTML container = containers[index] ranking = container.td.text name = container.findAll('td', {'class', 't'}) link = 'https://www.animenewsnetwork.com' + name[0].a['href'] name = name[0].text statistics = container.findAll('td', {'class', 'r'}) rating = statistics[0].text numVotes = statistics[1].text
# prints out anime stats to file currentAnime = Anime(name, ranking, rating, numVotes, link) animeFile.write('\nname: ' + name) animeFile.write('\nlink: ' + link) animeFile.write('\nranking: ' + ranking) animeFile.write('\nrating: ' + rating) animeFile.write('\nvotes: ' + numVotes)
# Goes to the webpage for the current anime animeClient = uReq(link) animeHTML = animeClient.read() animeClient.close() pageSoup = soup(animeHTML, 'html.parser')
# Genres of the current anime try: genreDiv = pageSoup.find(id='infotype-30') genres = genreDiv.findAll('span') genreList = [] for genre in genres: genreList.append(genre.a.text) currentAnime.genres = genreList except: currentAnime.genres = 'unknown'
# Themes of the current anime try: themes = pageSoup.find(id='infotype-31').findAll('span') themeList = [] for theme in themes: themeList.append(theme.a.text) currentAnime.themes = themeList except: currentAnime.themes = 'unknown'
# Premiere date of the current anime try: date = pageSoup.find(id='infotype-9').div.text currentAnime.premiereDate = date except: currentAnime.premiereDate = 'unknown'
# Director of the current anime try: director = pageSoup.find('b', text='Director').parent.a.text currentAnime.director = director except: currentAnime.director = 'unknown'
# Production Studio of the current anime try: productionStudio = pageSoup.find('b', text='Production').parent.a.text currentAnime.studio = productionStudio except: currentAnime.studio = 'unknown'
# Prints the genres animeFile.write('\ngenres: ') for genre in currentAnime.genres: animeFile.write(genre + ', ') # Prints the themes animeFile.write('\nthemes: ') for theme in currentAnime.themes: animeFile.write(theme + ', ') # Prints the premiere date, director, and studio animeFile.write('\npremiere date: ' + currentAnime.premiereDate) animeFile.write('\ndirector: ' + currentAnime.director) animeFile.write('\nproduction studio: ' + currentAnime.studio)
animeFile.write('\n')
发布于 2018-09-20 10:04:50
CTRL-C只是向Python发送键盘中断命令。这意味着在BeautifulSoup中发出HTML请求时,您可能只是提升了一个异常级别。CtrlBreak将完全停止该程序。
您的脚本可能会运行到无响应的网页中。您的CPU 将为 0%,因为它正在等待Web服务。我建议link
在每次uReq
调用之前打印代码中的值以跟踪发生这种情况的位置。
https://stackoverflow.com/questions/-100006141
复制相似问题