在Python语言中,我希望用户在控制台提示符中输入一个URL (接受输入并将其存储在某个变量中),例如,如果网页包含以下内容,则为:
<html>
<head>
</head>
<body>
<div>
<h1 class="class_one">First heading</h1>
<p>Some text</p>
<div class="class_two">
<div class="class_three">
<div class="class_one">
<center class="class_two">
<h3 class="class_three">
</h3>
</center>
<center>
<h3 class="find_first_class">
Some text
</h3>
</center>
</div>
</div>
</div>
<div class="class_two">
<div class="class_three">
<div class="class_one">
<center class="class_two">
<h2 class="find_second_class">
</h2>
</center>
</div>
</div>
</div>
</div>
</body>
</html>
然后,CSV应该包含网页HTML中的每个类的行(因为类可以出现多次,所以任何给定的类都可以有多行)。
现在,我想为页面上出现的所有类生成XPath。到目前为止,我写的是:
import urllib2
from bs4 import BeautifulSoup
result = {}
user_url_list = raw_input("Please enter your urls separated by spaces : \n")
url_list = map(str, user_url_list.split())
for url in url_list:
try:
page = urllib2.urlopen(url)
soup = BeautifulSoup(page, 'html.parser')
user_class_list = raw_input("Please enter the classes to parse for " + url + " separated by spaces : \n")
class_list = map(str, user_class_list.split())
for find_class in class_list:
try:
name_box = soup.find(attrs={'class': find_class})
print(xpath_soup(name_box))
break
except:
print("There was some error getting the xpath of class : " + find_class + " for url : " + url + "\n..trying next class now \n")
continue
except:
print(url + " is not valid, please enter correct full url \n")
continue
print(result)
https://stackoverflow.com/questions/50846570
复制相似问题