前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >解析网页速度比较(BeautifulSoup、PyQuery、lxml、正则)

解析网页速度比较(BeautifulSoup、PyQuery、lxml、正则)

作者头像
SeanCheney
发布2019-03-01 17:35:10
2.1K0
发布2019-03-01 17:35:10
举报
文章被收录于专栏:SeanCheney的专栏

用标题中的四种方式解析网页,比较其解析速度。复习PyQuery和PySpider,PySpider这个项目有点老了,现在还是使用被淘汰的PhantomJS。

系统配置、Python版本对解析速度也有影响,下面是我的结果(lxml与xpath最快,bs最慢):

代码语言:javascript
复制
==== Python version: 3.6.7 (v3.6.7:6ec5cf24b7, Oct 20 2018, 03:02:14) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] =====

==== Total trials: 10000 =====
bs4 total time: 7.7
pq total time: 0.9
lxml (cssselect) total time: 0.9
lxml (xpath) total time: 0.6
regex total time: 1.0 (doesn't find all p)

拷贝下面代码可以自测:

代码语言:javascript
复制
import re
import sys
import time
import requests
from lxml.html import fromstring
from pyquery import PyQuery as pq
from bs4 import BeautifulSoup as bs

def Timer():
    a = time.time()
    while True:
        c = time.time()
        yield time.time()-a
        a = c
timer = Timer()
url = "http://www.python.org/"
html = requests.get(url).text
num = 10000
print ('\n==== Python version: %s =====' %sys.version)
print ('\n==== Total trials: %s =====' %num)
next(timer)
soup = bs(html, 'lxml')
for x in range(num):
    paragraphs = soup.findAll('p')
t = next(timer)
print ('bs4 total time: %.1f' %t)
d = pq(html)
for x in range(num):
    paragraphs = d('p')
t = next(timer)
print ('pq total time: %.1f' %t)
tree = fromstring(html)
for x in range(num):
    paragraphs = tree.cssselect('p')
t = next(timer)
print ('lxml (cssselect) total time: %.1f' %t)
tree = fromstring(html)
for x in range(num):
    paragraphs = tree.xpath('.//p')
t = next(timer)
print ('lxml (xpath) total time: %.1f' %t)
for x in range(num):
    paragraphs = re.findall('<[p ]>.*?</p>', html)
t = next(timer)
print ('regex total time: %.1f (doesn\'t find all p)\n' %t)

借PyQuery复习CSS选择器。

PyQuery支持下载网页为文本,是通过urllib或Requests实现的:

代码语言:javascript
复制
from pyquery import PyQuery as pq

url = 'https://www.feixiaohao.com/currencies/bitcoin/'
headers = {
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 '
                          '(KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
        }
doc = pq(url=url, headers=headers, method='get')

# 也支持发送表单数据
# pq(your_url, {'q': 'foo'}, method='post', verify=True)

# btc价格
btc_price = doc('.mainPrice span.convert').text()
print(btc_price)

# btc logo
btc_logo = doc('img.coinLogo').attr('src')
print(btc_logo)

也可以直接加载文档字符串或html文档:

代码语言:javascript
复制
from pyquery import PyQuery as pq

html = '''
<div>
    <ul>
         <li class="item-0">first item</li>
         <li class="item-1"><a href="link2.html">second item</a></li>
         <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
         <li class="item-1 active"><a href="link4.html">fourth item</a></li>
         <li class="item-0"><a href="link5.html">fifth item</a></li>
    </ul>
</div>
'''

doc = pq(html)
# doc = pq(filename='demo.html')

# 使用eq可以按次序选择
print(doc('li').eq(1))

查找上下级元素可以通过find(),children(),parent(),parents(),siblings()。CSS选择器举例如下:

Pyspider的选择器是PyQuery。下面的例子是使用PySpider抓取IMDB250信息,fetch_type设为了js,存入MongoDB。

代码语言:javascript
复制
#!/usr/bin/env python
# -*- encoding: utf-8 -*-
# Created on 2019-01-30 16:22:03
# Project: imdb

from pyspider.libs.base_handler import *


class Handler(BaseHandler):
    crawl_config = {
    }

    @every(minutes=5)
    def on_start(self):
        self.crawl('https://www.imdb.com/chart/top', callback=self.index_page)

    @config(age=10 * 24 * 60 * 60)
    def index_page(self, response):
        for each in response.doc('.titleColumn > a').items():
            self.crawl(each.attr.href, callback=self.detail_page, fetch_type='js')

    @config(priority=2)
    def detail_page(self, response):
        return {            
            "title": response.doc('h1').text(),
            "voters": response.doc('[itemprop="aggregateRating"] > a > span').text(),
            "score": response.doc('strong > span').text()
        }

    # 需要再init中定义mongoclient
    def on_result(self, result):
        self.mongo.insert_result(result)
        super(Handler, self).on_result(result)
本文参与 腾讯云自媒体同步曝光计划,分享自作者个人站点/博客。
原始发表:2019.01.30 ,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 作者个人站点/博客 前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体同步曝光计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
相关产品与服务
云数据库 MongoDB
腾讯云数据库 MongoDB(TencentDB for MongoDB)是腾讯云基于全球广受欢迎的 MongoDB 打造的高性能 NoSQL 数据库,100%完全兼容 MongoDB 协议,支持跨文档事务,提供稳定丰富的监控管理,弹性可扩展、自动容灾,适用于文档型数据库场景,您无需自建灾备体系及控制管理系统。
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档