xpath库详解xpath入门获取所有节点 //子节点 /父节点 ..属性匹配 @文本获取按序选择节点轴选择

章鱼喵

发布于 2018-09-26 18:03:50

22.7K0

发布于 2018-09-26 18:03:50

文章被收录于专栏：codingcoding

xpath入门

python爬虫抓取网页内容，需要对html或xml结构的数据进行解析，如果用正则，单是写正则表达式就让很多望而生畏了。

这个问题可以用正则表达式处理，于是，一个问题就变成了两个问题

对于我们这些不喜欢写正则的人来说，xpath提供了更方便的解析数据功能。

xpath全称是：XML Path Language, 见名知意，是专门用于解析结构性语言的

xpath常用规则

使用xpath之前要先安装lxml库

pip install lxml

入门示例：

from lxml import etree

text = '''
<div>
    <ul>
        <li class="item-0"><a href="link1.html">first</a></li>
        <li class="item-1"><a href="link2.html">second</a>
        <li class="item-2"><a href="link3.html">third</li>
        <li class="item-3"><a href="link4.html">fourth</a></li>
    </ul>
</div>
'''

html = etree.HTML(text)
result = etree.tostring(html)
print(result.decode('utf-8'))

注意查看代码中的html片段，第二个li没有闭合，第三个li的a标签没有闭合

查看结果：

<html><body><div>
    <ul>
        <li class="item-0"><a href="link1.html">first</a></li>
        <li class="item-1"><a href="link2.html">second</a>
        </li><li class="item-2"><a href="link3.html">third</a></li>
        <li class="item-3"><a href="link4.html">fourth</a></li>
    </ul>
</div>
</body></html>

可以看到，etree模块不仅将缺少的标签闭合了，而且还加上了html、body节点

还可以读取文本内容进行解析

新建 test.html

<div>
    <ul>
        <li class="item-0"><a href="link1.html">first</a></li>
        <li class="item-1"><a href="link2.html">second</a></li>
        <li class="item-2"><a href="link3.html">third</a></li>
        <li class="item-3"><a href="link4.html">fourth</a></li>
    </ul>
</div>

from lxml import etree

html = etree.parse('./test.html', etree.HTMLParser())
result = etree.tostring(html)
print(result.decode('utf-8'))

结果：

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body><div>
    <ul>
        <li class="item-0"><a href="link1.html">first</a></li>
        <li class="item-1"><a href="link2.html">second</a></li>
        <li class="item-2"><a href="link3.html">third</a></li>
        <li class="item-3"><a href="link4.html">fourth</a></li>
    </ul>
</div>
</body></html>

获取所有节点 //

from lxml import etree

html = etree.parse('./test.html', etree.HTMLParser())
result = html.xpath('//*')
print(result)

结果：

[<Element html at 0x1085a5e88>, <Element body at 0x1085a5f88>, <Element div at 0x1085a5fc8>, <Element ul at 0x1085c9048>, <Element li at 0x1085c9088>, <Element a at 0x1085c9108>, <Element li at 0x1085c9148>, <Element a at 0x1085c9188>, <Element li at 0x1085c91c8>, <Element a at 0x1085c90c8>, <Element li at 0x1085c9208>, <Element a at 0x1085c9248>]

//* 表示匹配所有节点

匹配指定节点，如获取所有li节点

from lxml import etree

html = etree.parse('./test.html', etree.HTMLParser())
result = html.xpath('//li')
print(result)  # 所有li节点
print(result[0])  # 第一个li节点

结果：

[<Element li at 0x110115f88>, <Element li at 0x110115fc8>, <Element li at 0x110139048>, <Element li at 0x110139088>]
<Element li at 0x110115f88>

子节点 /

获取li节点的直接子节点

from lxml import etree

html = etree.parse('./test.html', etree.HTMLParser())
result = html.xpath('//li/a') # 获取所有li节点的直接子节点a
print(result)

[<Element a at 0x103c02f88>, <Element a at 0x103c02fc8>, <Element a at 0x103c26048>, <Element a at 0x103c26088>]

改成 // 可以这么写：

from lxml import etree

html = etree.parse('./test.html', etree.HTMLParser())
result = html.xpath('//div//a')  # 获取div的所有后代a节点
print(result)

父节点 ..

from lxml import etree

html = etree.parse('./test.html', etree.HTMLParser())
# 获取href属性为link2.html的a标签的父节点的class名
result = html.xpath('//a[@href="link2.html"]/../@class')

print(result)
# ['item-1']

属性匹配 @

根据属性值匹配节点

from lxml import etree

html = etree.parse('./test.html', etree.HTMLParser())
# 获取属性class值为item-0的li
result = html.xpath('//li[@class="item-0"]')

print(result)
# [<Element li at 0x10c2b1f88>]

获取属性值

from lxml import etree

html = etree.parse('./test.html', etree.HTMLParser())
# 获取所有li的子节点a的属性href
result = html.xpath('//li/a/@href')

print(result)
# ['link1.html', 'link2.html', 'link3.html', 'link4.html']

属性多值匹配

使用contains函数匹配

from lxml import etree

text = '''
<li class="li li-first"><a href="link.html">first item</a></li>
'''

html = etree.HTML(text)
result = html.xpath('//li[@class="li"]/a/text()')
print(result)
# []

result = html.xpath('//li[contains(@class, "li")]/a/text()')
print(result)
# ['first item']

多属性匹配

需要匹配满足多个属性的节点，使用 and 运算符

from lxml import etree

text = '''
<li class="li li-first" name="item"><a href="link.html">first item</a></li>
'''

html = etree.HTML(text)
# 通过class和name两个属性进行匹配
result = html.xpath('//li[contains(@class, "li") and @name="item"]/a/text()')
print(result)
# ['first item']

xpath的运算符介绍

运算符

文本获取

from lxml import etree

html = etree.parse('./test.html', etree.HTMLParser())
# 获取属性class值为item-0的li的子节点a的文本内容
result = html.xpath('//li[@class="item-0"]/a/text()')

print(result)
# ['first']

如果想要获取后代节点内部的所有文本，使用 //text()

from lxml import etree

html = etree.parse('./test.html', etree.HTMLParser())
# 获取所有li的后代节点中的文本
result = html.xpath('//li//text()')

print(result)
# ['first', 'second', 'third', 'fourth']

按序选择

根据节点所在的顺序进行提取

from lxml import etree

html = etree.parse('./test.html', etree.HTMLParser())

# 按索引排序
result = html.xpath('//li[1]/a/text()')
print(result)
# ['first']

# last 最后一个
result = html.xpath('//li[last()]/a/text()')
print(result)
# ['fourth']

# position 位置查找
result = html.xpath('//li[position()<3]/a/text()')
print(result)
# ['first', 'second']

# - 运算符
result = html.xpath('//li[last()-2]/a/text()')
print(result)
# ['second']

节点轴选择

from lxml import etree

html = etree.parse('./test.html', etree.HTMLParser())

# 所有祖先节点
result = html.xpath('//li[1]/ancestor::*')
print(result)
# [<Element html at 0x106e4be88>, <Element body at 0x106e4bf88>, <Element div at 0x106e4bfc8>, <Element ul at 0x106e6f048>]

# 祖先节点中的div
result = html.xpath('//li[1]/ancestor::div')
print(result)
# [<Element div at 0x106ce4fc8>]

# 节点的所有属性
result = html.xpath('//li[1]/attribute::*')
print(result)
# ['item-0']

# 子节点
result = html.xpath('//li[1]/child::a[@href="link1.html"]')
print(result)
# [<Element a at 0x107941fc8>]

# 后代节点中的a
result = html.xpath('//li[1]/descendant::a')
print(result)
# [<Element a at 0x10eeb7fc8>]

# 该节点后面所有节点中的第2个 从1开始计数
result = html.xpath('//li[1]/following::*[2]')
print(result)
# [<Element a at 0x10f188f88>]

# 该节点后面的所有兄弟节点
result = html.xpath('//li[1]/following-sibling::*')
print(result)
# [<Element li at 0x104b7f048>, <Element li at 0x104b7f088>, <Element li at 0x104b7f0c8>]

本文参与腾讯云自媒体分享计划，分享自作者个人站点/博客。

原始发表：2018.09.20 ，如有侵权请联系 cloudcommunity@tencent.com 删除

python