使用beautifulSoup和Python抓取h3和div标记中的文本
使用beautifulSoup和Python抓取h3和div标记中的文本
提问于 2017-10-25 21:25:01
我没有使用python，BeautifulSoup，Selenium等的经验，但我渴望从网站上抓取数据，并将其存储为csv文件。我需要的单个数据样本编码如下(单行数据)。
<div class="box effect">
<div class="row">
<div class="col-lg-10">
<h3>HEADING</h3>
<div><i class="fa user"></i> NAME</div>
<div><i class="fa phone"></i> MOBILE</div>
<div><i class="fa mobile-phone fa-2"></i> NUMBER</div>
<div><i class="fa address"></i> XYZ_ADDRESS</div>
<div class="space"> </div>
<div style="padding:10px;padding-left:0px;"><a class="btn btn-primary btn-sm" href="www.link_to_another_page.com"><i class="fa search-plus"></i> more info</a></div>
</div>
<div class="col-lg-2">
</div>
</div>
</div>我需要的输出是Heading,NAME,MOBILE,NUMBER,XYZ_ADDRESS
我发现这些数据还没有id或类作为普通文本出现在网站上。为此，我分别尝试了BeautifulSoup和Python Selenium，在这两种方法中我都坚持提取，因为我看不到任何教程，并引导我从这些和标签中提取文本
我的代码使用BeautifulSoup
import urllib2
from bs4 import BeautifulSoup
import requests
import csv
MAX = 2
'''with open("lg.csv", "a") as f:
w=csv.writer(f)'''
##for i in range(1,MAX+1)
url="http://www.example_site.com"
page=requests.get(url)
soup = BeautifulSoup(page.content,"html.parser")
for h in soup.find_all('h3'):
print(h.get('h3'))我的selenium代码
import csv
from selenium import webdriver
MAX_PAGE_NUM = 2
driver = webdriver.Firefox()
for i in range(1, MAX_PAGE_NUM+1):
url = "http://www.example_site.com"
driver.get(url)
name = driver.find_elements_by_xpath('//div[@class = "col-lg-10"]/h3')
#contact = driver.find_elements_by_xpath('//span[@class="item-price"]')
# phone =
# mobile =
# address =
# print(len(buyers))
# num_page_items = len(buyers)
# with open('res.csv','a') as f:
# for i in range(num_page_items):
# f.write(buyers[i].text + "," + prices[i].text + "\n")
print (name)
driver.close()回答 3
Stack Overflow用户
回答已采纳
发布于 2017-10-25 21:36:54
您可以使用CSS选择器来查找所需的数据。在本例中，div > h3 ~ div将查找直接位于div元素内部并由h3元素处理的所有div元素。
import bs4
page= """
<div class="box effect">
<div class="row">
<div class="col-lg-10">
<h3>HEADING</h3>
<div><i class="fa user"></i> NAME</div>
<div><i class="fa phone"></i> MOBILE</div>
<div><i class="fa mobile-phone fa-2"></i> NUMBER</div>
<div><i class="fa address"></i> XYZ_ADDRESS</div>
</div>
</div>
</div>
"""
soup = bs4.BeautifulSoup(page, 'lxml')
# find all div elements that are inside a div element
# and are proceeded by an h3 element
selector = 'div > h3 ~ div'
# find elements that contain the data we want
found = soup.select(selector)
# Extract data from the found elements
data = [x.text.split(';')[-1].strip() for x in found]
for x in data:
print(x)编辑:刮掉标题中的文本。
heading = soup.find('h3')
heading_data = heading.text
print(heading_data)编辑:或者，您可以使用如下选择器同时获取标题和其他div元素：div.col-lg-10 > *。这将查找属于col-lg-10类的div元素中的所有元素。
soup = bs4.BeautifulSoup(page, 'lxml')
# find all elements inside a div element of class col-lg-10
selector = 'div.col-lg-10 > *'
# find elements that contain the data we want
found = soup.select(selector)
# Extract data from the found elements
data = [x.text.split(';')[-1].strip() for x in found]
for x in data:
print(x)Stack Overflow用户
发布于 2017-10-25 21:37:23
试试这个：
import urllib2
from bs4 import BeautifulSoup
import requests
import csv
MAX = 2
'''with open("lg.csv", "a") as f:
w=csv.writer(f)'''
##for i in range(1,MAX+1)
url="http://www.example_site.com"
page=requests.get(url)
soup = BeautifulSoup(page,"html.parser")
print(soup.text)Stack Overflow用户
发布于 2020-12-05 11:39:21
所以它看起来很不错：
# -*- coding: utf-8 -*-
# by Faguiro #
# run using Python 3.8.6 on Linux#
import requests
from bs4 import BeautifulSoup
# insert your site here
url= input("Enter the url-->")
#use requests
r = requests.get(url)
content = r.content
#soup!
soup = BeautifulSoup(content, "html.parser")
#find all tag in the soup.
heading = soup.find_all("h3")
#print(heading) <--- result...
#...ptonic organization!
n=len(heading)
for x in range(n):
print(str.strip(heading[x].text))依赖关系:在终端(Linux)上：
sudo apt-get install python3-bs4
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/46933679
复制相关文章
相似问题
腾讯云开发者
