Scrapy1.4最新官方文档总结 2 Tutorial创建项目提取信息XPath简短介绍继续提取名人名言用爬虫提取信息保存数据提取下一页使用爬虫参数更多例子

这是官方文档的Tutorial(https://docs.scrapy.org/en/latest/int

推荐四个Python学习资源:

创建项目

使用命令:

scrapy startproject tutorial

会生成以下文件:

在tutorial/spiders文件夹新建文件quotes_spider.py,它的代码如下:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            'http://quotes.toscrape.com/page/1/',
            'http://quotes.toscrape.com/page/2/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)
        self.log('Saved file %s' % filename)

start_requests方法返回 scrapy.Request对象。每收到一个,就实例化一个Response对象,并调用和request绑定的调回方法(即parse),将response作为参数。

切换到根目录,运行爬虫:

scrapy crawl quotes

输出日志

根目录下会产生两个文件,quotes-1.html和quotes-2.html。

另一种方法是定义一个包含URLs的类,parse( )是Scrapy默认的调回方法,即使没有指明调回,也会执行:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
        'http://quotes.toscrape.com/page/2/',
    ]

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)

提取信息

学习Scrapy提取信息的最好方法是使用Scrapy Shell,win7 shell运行:

scrapy shell "http://quotes.toscrape.com/page/1/"

或者,gitbash运行,注意有单引号和双引号的区别:

scrapy shell 'http://quotes.toscrape.com/page/1/'

输出如下:

利用CSS进行提取:

>>> response.css('title')
[<Selector xpath='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]

只提取标题的文本:

>>> response.css('title::text').extract()
['Quotes to Scrape']

::text表示只提取文本,去掉的话,显示如下:

>>> response.css('title').extract()
['<title>Quotes to Scrape</title>']

因为返回对象是一个列表,只提取第一个的话,使用:

>>> response.css('title::text').extract_first()
'Quotes to Scrape'

或者,使用序号:

>>> response.css('title::text')[0].extract()
'Quotes to Scrape'

前者更好,可以避免潜在的序号错误。

除了使用 extract()和 extract_first(),还可以用正则表达式:

>>> response.css('title::text').re(r'Quotes.*')
['Quotes to Scrape']
>>> response.css('title::text').re(r'Q\w+')
['Quotes']
>>> response.css('title::text').re(r'(\w+) to (\w+)')
['Quotes', 'Scrape']

提取日志

XPath简短介绍

Scrapy还支持XPath:

>>> response.xpath('//title')
[<Selector xpath='//title' data='<title>Quotes to Scrape</title>'>]
>>> response.xpath('//title/text()').extract_first()
'Quotes to Scrape'

其实,CSS是底层转化为XPath的,但XPath的功能更为强大,比如它可以选择包含next page的链接。更多见 using XPath with Scrapy Selectors here

继续提取名人名言

http://quotes.toscrape.com的每个名言的HTML结构如下:

<div class="quote">
    <span class="text">“The world as we have created it is a process of our
    thinking. It cannot be changed without changing our thinking.”</span>
    <span>
        by <small class="author">Albert Einstein</small>
        <a href="/author/Albert-Einstein">(about)</a>
    </span>
    <div class="tags">
        Tags:
        <a class="tag" href="/tag/change/page/1/">change</a>
        <a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>
        <a class="tag" href="/tag/thinking/page/1/">thinking</a>
        <a class="tag" href="/tag/world/page/1/">world</a>
    </div>
</div>

使用:

$ scrapy shell "http://quotes.toscrape.com"

将HTML的元素以列表的形式提取出来:

response.css("div.quote")

只要第一个:

quote = response.css("div.quote")[0]

提取出标题、作者、标签:

>>> title = quote.css("span.text::text").extract_first()
>>> title
'“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
>>> author = quote.css("small.author::text").extract_first()
>>> author
'Albert Einstein'

标签是一组字符串:

>>> tags = quote.css("div.tags a.tag::text").extract()
>>> tags
['change', 'deep-thoughts', 'thinking', 'world']

弄明白了提取每个名言,现在提取所有的:

>>> for quote in response.css("div.quote"):
...     text = quote.css("span.text::text").extract_first()
...     author = quote.css("small.author::text").extract_first()
...     tags = quote.css("div.tags a.tag::text").extract()
...     print(dict(text=text, author=author, tags=tags))
{'tags': ['change', 'deep-thoughts', 'thinking', 'world'], 'author': 'Albert Einstein', 'text': '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'}
{'tags': ['abilities', 'choices'], 'author': 'J.K. Rowling', 'text': '“It is our choices, Harry, that show what we truly are, far more than our abilities.”'}
    ... a few more of these, omitted for brevity
>>>

用爬虫提取信息

使用Python的yield:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
        'http://quotes.toscrape.com/page/2/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('small.author::text').extract_first(),
                'tags': quote.css('div.tags a.tag::text').extract(),
            }

运行爬虫,日志如下:

2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/>
{'tags': ['life', 'love'], 'author': 'André Gide', 'text': '“It is better to be hated for what you are than to be loved for what you are not.”'}
2016-09-19 18:57:19 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/>
{'tags': ['edison', 'failure', 'inspirational', 'paraphrased'], 'author': 'Thomas A. Edison', 'text': "“I have not failed. I've just found 10,000 ways that won't work.”"}

保存数据

最便捷的方式是使用feed export,保存为json,命令如下:

scrapy crawl quotes -o quotes.json

保存为json lines:

scrapy crawl quotes -o quotes.jl

保存为csv:

scrapy crawl quotes -o quotes.csv

提取下一页

首先看下一页的链接:

<ul class="pager">
    <li class="next">
        <a href="/page/2/">Next <span aria-hidden="true">→</span></a>
    </li>
</ul>

提取出来:

>>> response.css('li.next a').extract_first()
'<a href="/page/2/">Next <span aria-hidden="true">→</span></a>'

只要href:

>>> response.css('li.next a::attr(href)').extract_first()
'/page/2/'

利用urljoin生成完整的url,生成下一页的请求,就可以循环抓取了:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('small.author::text').extract_first(),
                'tags': quote.css('div.tags a.tag::text').extract(),
            }

        next_page = response.css('li.next a::attr(href)').extract_first()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            yield scrapy.Request(next_page, callback=self.parse)

更简洁的方式是使用 response.follow:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('span small::text').extract_first(),
                'tags': quote.css('div.tags a.tag::text').extract(),
            }

        next_page = response.css('li.next a::attr(href)').extract_first()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)

直接将参数传递给response.follow:

for href in response.css('li.next a::attr(href)'):
    yield response.follow(href, callback=self.parse)

对于a标签,response.follow可以直接使用它的属性,这样就可以变得更简洁:

for a in response.css('li.next a'):
    yield response.follow(a, callback=self.parse)

下面这个爬虫提取作者信息,使用了调回和自动获取下一页:

import scrapy

class AuthorSpider(scrapy.Spider):
    name = 'author'

    start_urls = ['http://quotes.toscrape.com/']

    def parse(self, response):
        # 作者链接
        for href in response.css('.author + a::attr(href)'):
            yield response.follow(href, self.parse_author)

        # 分页链接
        for href in response.css('li.next a::attr(href)'):
            yield response.follow(href, self.parse)

    def parse_author(self, response):
        def extract_with_css(query):
            return response.css(query).extract_first().strip()

        yield {
            'name': extract_with_css('h3.author-title::text'),
            'birthdate': extract_with_css('.author-born-date::text'),
            'bio': extract_with_css('.author-description::text'),
        }

使用爬虫参数

在命令行中使用参数,只要添加 -a:

scrapy crawl quotes -o quotes-humor.json -a tag=humor

将humor传递给tag:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        url = 'http://quotes.toscrape.com/'
        tag = getattr(self, 'tag', None)
        if tag is not None:
            url = url + 'tag/' + tag
        yield scrapy.Request(url, self.parse)

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('small.author::text').extract_first(),
            }

        next_page = response.css('li.next a::attr(href)').extract_first()
        if next_page is not None:
            yield response.follow(next_page, self.parse)

更多例子

https://github.com/scrapy/quotesbot上有个叫做quotesbot的爬虫,提供了CSS和XPath两种写法:

import scrapy

class ToScrapeCSSSpider(scrapy.Spider):
    name = "toscrape-css"
    start_urls = [
        'http://quotes.toscrape.com/',
    ]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                'text': quote.css("span.text::text").extract_first(),
                'author': quote.css("small.author::text").extract_first(),
                'tags': quote.css("div.tags > a.tag::text").extract()
            }

        next_page_url = response.css("li.next > a::attr(href)").extract_first()
        if next_page_url is not None:
            yield scrapy.Request(response.urljoin(next_page_url))
import scrapy

class ToScrapeSpiderXPath(scrapy.Spider):
    name = 'toscrape-xpath'
    start_urls = [
        'http://quotes.toscrape.com/',
    ]

    def parse(self, response):
        for quote in response.xpath('//div[@class="quote"]'):
            yield {
                'text': quote.xpath('./span[@class="text"]/text()').extract_first(),
                'author': quote.xpath('.//small[@class="author"]/text()').extract_first(),
                'tags': quote.xpath('.//div[@class="tags"]/a[@class="tag"]/text()').extract()
            }

        next_page_url = response.xpath('//li[@class="next"]/a/@href').extract_first()
        if next_page_url is not None:
            yield scrapy.Request(response.urljoin(next_page_url))

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏简书专栏

基于jieba、TfidfVectorizer、LogisticRegression的垃圾邮件分类

jieba中文叫做结巴,是一款中文分词工具,官方文档链接:https://github.com/fxsjy/jieba TfidfVectorizer中文叫做...

23710
来自专栏PPV课数据科学社区

R包之tm:文本挖掘包

简介 安装 数据输入—文集(corpus) 数据输出 查看语料库(corpora) 查看某几条信息 查看单个文档元数据 查看单个文档内容 查看多个文档内容 变换...

40980
来自专栏落影的专栏

OpenGLES进阶教程8-obj文件和mtl文件解析

教程 距离上一篇教程已经有两个月了,这两个月详细阅读GPUImage的源码,并写了详细解析,发现对OpenGLES的深入了解很有帮助。 上周一个简书的朋友问我...

45470
来自专栏Python小屋

Win10系统配置Python3.6+OpenGL环境详细步骤

1、首先登录https://www.opengl.org/resources/libraries/glut/,下载下图箭头所指的文件 ? 2、解压缩,如下图所示...

46170
来自专栏开源FPGA

基于FPGA驱动VGA显示图片的小问题

         学习VGA显示图片的过程中,遇到了一个小问题,我在显示屏上开了一个60x60的框,放了一张图片进去显示,但是最终的结果如下图所示。 ?   出...

28790
来自专栏为数不多的Android技巧

ASCII Art:使用纯文本流程图

我们使用纯文本写代码,有了Markdown又可以使用纯文本写文档,那么对于更直观的信息表达方式——图片,能不能使用纯文本描述呢?

41020
来自专栏生信宝典

R语言学习 - 箱线图一步法

箱线图 - 一步绘制 绘图时通常会碰到两个头疼的问题: 有时需要绘制很多的图,唯一的不同就是输入文件,其它都不需要修改。如果用R脚本,需要反复替换文件名,繁琐又...

38650
来自专栏玩转JavaEE

Spring Cloud中的负载均衡策略

在上篇博客(Spring Cloud中负载均衡器概览)中,我们大致的了解了一下Spring Cloud中有哪些负载均衡器,但是对于负载均衡策略我们并没有去详细了...

35850
来自专栏计算机视觉与深度学习基础

Leetcode 85 Maximal Rectangle 推荐!

Given a 2D binary matrix filled with 0's and 1's, find the largest rectangle co...

24950
来自专栏猿人谷

memcpy和memmove的区别

memcpy()和memmove()都是C语言中的库函数,在头文件string.h中,其原型分别如下: void *memcpy(void *dst, con...

36750

扫码关注云+社区

领取腾讯云代金券