编写一个python脚本来根据任何类找到来自HTML的XPath?

内容来源于 Stack Overflow,并遵循CC BY-SA 3.0许可协议进行翻译与使用

  • 回答 (1)
  • 关注 (0)
  • 查看 (60)

在Python中,我希望用户在控制台提示符中输入URL(将输入并将其存储在某个变量中),然后检查该URL是否包含一些HTML。

如果该网页包含一些内容,则解析该网页,并根据类为所有XPath制作CSV。

例如,如果网页包含此HTML:

<html>
<head>
</head>
    <body>
        <div>
            <h1 class="class_one">First heading</h1>
                <p>Some text</p>
            <div class="class_two">
                <div class="class_three">
                    <div class="class_one">
                        <center class="class_two">
                            <h3 class="class_three">
                            </h3>
                        </center>
                        <center>
                            <h3 class="find_first_class">
                                Some text
                            </h3>
                        </center>
                    </div>
                </div>
            </div>
            <div class="class_two">
                <div class="class_three">
                    <div class="class_one">
                        <center class="class_two">
                            <h2 class="find_second_class">
                            </h2>
                        </center>
                    </div>
                </div>
            </div>
        </div>
    </body>
</html>

然后,CSV应该包含网页HTML中每个类的行(因为类可以出现多次,所以对于任何给定的类我们可以有多行)。

示例CSV可能如下所示:

find_first_class,/div[1]/div[1]/div[1]/div[1]/center[2]/h3[1],second path,third path
find_second_class,/div[1]/div[1]/div[1]/div[1]/center[1]/h2[1], second path, thrid path

现在,我想为页面上的所有类生成XPath。我到目前为止写的是:

import urllib2
from bs4 import BeautifulSoup

result = {}
user_url_list = raw_input("Please enter your urls separated by spaces : \n")
url_list = map(str, user_url_list.split())
for url in url_list:
    try:
        page = urllib2.urlopen(url)
        soup = BeautifulSoup(page, 'html.parser')
        user_class_list = raw_input("Please enter the classes to parse for " + url + " separated by spaces : \n")
        class_list = map(str, user_class_list.split())
        for find_class in class_list:
            try:
                name_box = soup.find(attrs={'class': find_class})
                print(xpath_soup(name_box))
                break
            except:
                print("There was some error getting the xpath of class : " + find_class + " for url : " + url + "\n..trying next class now \n")
                continue
    except:
        print(url + " is not valid, please enter correct full url \n")
        continue
print(result)
提问于
用户回答回答于
import csv
import requests
from lxml import etree

target_url = input('Which url is to be scraped?')

page = '''
<html>
<head>
</head>
    <body>
        <div>
            <h1 class="class_one">First heading</h1>
                <p>Some text</p>
            <div class="class_two">
                <div class="class_three">
                    <div class="class_one">
                        <center class="class_two">
                            <h3 class="class_three">
                            </h3>
                        </center>
                        <center>
                            <h3 class="find_first_class">
                                Some text
                            </h3>
                        </center>
                    </div>
                </div>
            </div>
            <div class="class_two">
                <div class="class_three">
                    <div class="class_one">
                        <center class="class_two">
                            <h2 class="find_second_class">
                            </h2>
                        </center>
                    </div>
                </div>
            </div>
        </div>
    </body>
</html>
'''

#response = requests.get(target_url)
#document = etree.parse(response.content)
classes_list = ['find_first_class', 'find_second_class']
expressions = []

document = etree.fromstring(page)

for element in document.xpath('//*'):
    try:
        ele_class = element.xpath("@class")[0]
        print(ele_class)
        if ele_class in classes_list:
            tree = etree.ElementTree(element)
            expressions.append((ele_class, tree.getpath(element)))
    except IndexError:
        print("No class in this element.")
        continue

with open('test.csv', 'w') as f:
    writer = csv.writer(f, delimiter=',')
    writer.writerows(expressions)

扫码关注云+社区

领取腾讯云代金券