Scrapy 入门教程

scrapy startproject tutorial

(scrapy) localhost:scrapy stanley$ scrapy startproject tutorial
New Scrapy project 'tutorial', using template directory '/Users/stanley/virtualenv/scrapy/lib/python2.7/site-packages/scrapy/templates/project', created in:
    /Users/stanley/virtualenv/scrapy/tutorial

You can start your first spider with:
    cd tutorial
    scrapy genspider example example.com

|____scrapy.cfg   #部署配置文件
|____tutorial     #项目的python模块
| |______init__.py
| |____items.py        #item文件
| |____middlewares.py  #middlewares文件
| |____pipelines.py    #pipeline文件
| |____settings.py     #项目配置
| |____spiders         #存放爬虫的目录
| | |______init__.py

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            'http://quotes.toscrape.com/page/1/',
            'http://quotes.toscrape.com/page/2/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)
        self.log('Saved file %s' % filename)

scrapy crawl quotes

...

2017-08-08 07:17:01 [scrapy.core.engine] INFO: Spider opened
2017-08-08 07:17:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-08-08 07:17:01 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-08-08 07:17:03 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)
2017-08-08 07:17:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
2017-08-08 07:17:03 [quotes] DEBUG: Saved file quotes-1.html
2017-08-08 07:17:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/2/> (referer: None)
2017-08-08 07:17:03 [quotes] DEBUG: Saved file quotes-2.html
2017-08-08 07:17:03 [scrapy.core.engine] INFO: Closing spider (finished)
2017-08-08 07:17:03 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 675,
 'downloader/request_count': 3,
 'downloader/request_method_count/GET': 3,
 'downloader/response_bytes': 5976,
 'downloader/response_count': 3,
 'downloader/response_status_count/200': 2,
 'downloader/response_status_count/404': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 8, 7, 23, 17, 3, 793776),
 'log_count/DEBUG': 6,
 'log_count/INFO': 7,
 'memusage/max': 43855872,
 'memusage/startup': 43855872,
 'response_received_count': 3,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'start_time': datetime.datetime(2017, 8, 7, 23, 17, 1, 659265)}
2017-08-08 07:17:03 [scrapy.core.engine] INFO: Spider closed (finished)

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
        'http://quotes.toscrape.com/page/2/',
    ]

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)

scrapy shell 'http://quotes.toscrape.com/page/1/'

2017-08-08 11:45:41 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: tutorial)
2017-08-08 11:45:41 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'tutorial.spiders', 'ROBOTSTXT_OBEY': True, 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter', 'SPIDER_MODULES': ['tutorial.spiders'], 'BOT_NAME': 'tutorial', 'LOGSTATS_INTERVAL': 0}

...

2017-08-08 11:45:41 [scrapy.core.engine] INFO: Spider opened
2017-08-08 11:45:42 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)
2017-08-08 11:45:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   crawler    <scrapy.crawler.Crawler object at 0x10fc21190>
[s]   item       {}
[s]   request    <GET http://quotes.toscrape.com/page/1/>
[s]   response   <200 http://quotes.toscrape.com/page/1/>
[s]   settings   <scrapy.settings.Settings object at 0x10fc21a90>
[s]   spider     <DefaultSpider 'default' at 0x10feb3650>
[s] Useful shortcuts:
[s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
[s]   fetch(req)                  Fetch a scrapy.Request and update local objects 
[s]   shelp()           Shell help (print this help)
[s]   view(response)    View response in a browser
>>> 

>>> response.css('title')
[<Selector xpath=u'descendant-or-self::title' data=u'<title>Quotes to Scrape</title>'>]

>>> response.css('title::text').extract()
[u'Quotes to Scrape']

>>> response.css('title').extract()
[u'<title>Quotes to Scrape</title>']

>>> response.css('title::text').extract_first()
u'Quotes to Scrape'

>>> response.css('title::text').re(r'Quotes.*')
['Quotes to Scrape']
>>> response.css('title::text').re(r'Q\w+')
['Quotes']
>>> response.css('title::text').re(r'(\w+) to (\w+)')
['Quotes', 'Scrape']

view(response)

>>> response.xpath('//title')
[<Selector xpath='//title' data='<title>Quotes to Scrape</title>'>]
>>> response.xpath('//title/text()').extract_first()
'Quotes to Scrape'

<div class="quote">
    <span class="text">“The world as we have created it is a process of our
    thinking. It cannot be changed without changing our thinking.”</span>
    <span>
        by <small class="author">Albert Einstein</small>
        <a href="/author/Albert-Einstein">(about)</a>
    </span>
    <div class="tags">
        Tags:
        <a class="tag" href="/tag/change/page/1/">change</a>
        <a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>
        <a class="tag" href="/tag/thinking/page/1/">thinking</a>
        <a class="tag" href="/tag/world/page/1/">world</a>
    </div>
</div>

>>> response.css("div.quote")
[<Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data=u'<div class="quote" itemscope itemtype="h'>, <Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data=u'<div class="quote" itemscope itemtype="h'>, <Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data=u'<div class="quote" itemscope itemtype="h'>, <Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data=u'<div class="quote" itemscope itemtype="h'>, <Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data=u'<div class="quote" itemscope itemtype="h'>, <Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data=u'<div class="quote" itemscope itemtype="h'>, <Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data=u'<div class="quote" itemscope itemtype="h'>, <Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data=u'<div class="quote" itemscope itemtype="h'>, <Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data=u'<div class="quote" itemscope itemtype="h'>, <Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data=u'<div class="quote" itemscope itemtype="h'>]
>>> 

>>> response.css("div.quote")[0]
<Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' quote ')]" data=u'<div class="quote" itemscope itemtype="h'>

>>> quote = response.css("div.quote")[0]
>>> title = quote.css("span.text::text").extract_first()
>>> title
u'\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d'
>>> author = quote.css("small.author::text").extract_first()
>>> author
u'Albert Einstein'
>>> 

>>> tags = quote.css("div.tags a.tag::text").extract()
>>> tags
[u'change', u'deep-thoughts', u'thinking', u'world']

>>> for quote in response.css("div.quote"):
...   text = quote.css("span.text::text").extract_first()
...   author = quote.css("small.author::text").extract_first()
...   tags = quote.css("div.tags a.tag::text").extract()
...   print(dict(text=text, author=author, tags=tags))
... 
{'text': u'\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d', 'tags': [u'change', u'deep-thoughts', u'thinking', u'world'], 'author': u'Albert Einstein'}
{'text': u'\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d', 'tags': [u'abilities', u'choices'], 'author': u'J.K. Rowling'}
{'text': u'\u201cThere are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.\u201d', 'tags': [u'inspirational', u'life', u'live', u'miracle', u'miracles'], 'author': u'Albert Einstein'}

...

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
        'http://quotes.toscrape.com/page/2/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('small.author::text').extract_first(),
                'tags': quote.css('div.tags a.tag::text').extract(),
            }

>>>scrapy crawl quotes

...

{'text': u'\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d', 'tags': [u'change', u'deep-thoughts', u'thinking', u'world'], 'author': u'Albert Einstein'}
2017-08-08 13:00:58 [scrapy.core.scraper] DEBUG: Scraped from <200 http://quotes.toscrape.com/page/1/>
{'text': u'\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d', 'tags': [u'abilities', u'choices'], 'author': u'J.K. Rowling'}
...

scrapy crawl quotes -o quotes.json

[
{"text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d", "author": "Albert Einstein", "tags": ["change", "deep-thoughts", "thinking", "world"]},
{"text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d", "author": "J.K. Rowling", "tags": ["abilities", "choices"]},
...

<ul class="pager">
    <li class="next">
        <a href="/page/2/">Next <span aria-hidden="true">&rarr;</span></a>
    </li>
</ul>

>>> response.css('li.next a::attr(href)').extract_first()
u'/page/2/'

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('small.author::text').extract_first(),
                'tags': quote.css('div.tags a.tag::text').extract(),
            }

        next_page = response.css('li.next a::attr(href)').extract_first()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            yield scrapy.Request(next_page, callback=self.parse)

>>> next_page = response.css('li.next a::attr(href)').extract_first()
>>> next_page
u'/page/2/'
>>> response.url
'http://quotes.toscrape.com/page/1/'
>>> response.urljoin(next_page)
u'http://quotes.toscrape.com/page/2/'

...
next_page = response.css('li.next a::attr(href)').extract_first()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)
...

for href in response.css('li.next a::attr(href)'):
    yield response.follow(href, callback=self.parse)

for a in response.css('li.next a'):
    yield response.follow(a, callback=self.parse)

import scrapy


class AuthorSpider(scrapy.Spider):
    name = 'author'

    start_urls = ['http://quotes.toscrape.com/']

    def parse(self, response):
        # follow links to author pages
        for href in response.css('.author + a::attr(href)'):
            yield response.follow(href, self.parse_author)

        # follow pagination links
        for href in response.css('li.next a::attr(href)'):
            yield response.follow(href, self.parse)

    def parse_author(self, response):
        def extract_with_css(query):
            return response.css(query).extract_first().strip()

        yield {
            'name': extract_with_css('h3.author-title::text'),
            'birthdate': extract_with_css('.author-born-date::text'),
            'bio': extract_with_css('.author-description::text'),
        }

scrapy crawl author -o author.json

[
{"bio": "Marilyn Monroe (born Norma Jeane Mortenson; June 1, 1926 \u2013 August 5, 1962) was an American actress, model, and singer, who became a major sex symbol, starring in a number of commercially successful motion pictures during the 1950s and early 1960s.After spending much of her childhood in foster homes, Monroe began a career as a model, which led to a film contract in 1946 with Twentieth Century-Fox. Her early film appearances were minor, but her performances in The Asphalt Jungle and All About Eve (both 1950), drew attention. By 1952 she had her first leading role in Don't Bother to Knock and 1953 brought a lead in Niagara, a melodramatic film noir that dwelt on her seductiveness. Her \"dumb blonde\" persona was used to comic effect in subsequent films such as Gentlemen Prefer Blondes (1953), How to Marry a Millionaire (1953) and The Seven Year Itch (1955). Limited by typecasting, Monroe studied at the Actors Studio to broaden her range. Her dramatic performance in Bus Stop (1956) was hailed by critics and garnered a Golden Globe nomination. Her production company, Marilyn Monroe Productions, released The Prince and the Showgirl (1957), for which she received a BAFTA Award nomination and won a David di Donatello award. She received a Golden Globe Award for her performance in Some Like It Hot (1959). Monroe's last completed film was The Misfits, co-starring Clark Gable with screenplay by her then-husband, Arthur Miller.Marilyn was a passionate reader, owning four hundred books at the time of her death, and was often photographed with a book.The final years of Monroe's life were marked by illness, personal problems, and a reputation for unreliability and being difficult to work with. The circumstances of her death, from an overdose of barbiturates, have been the subject of conjecture. Though officially classified as a \"probable suicide\", the possibility of an accidental overdose, as well as of homicide, have not been ruled out. In 1999, Monroe was ranked as the sixth greatest female star of all time by the American Film Institute. In the decades following her death, she has often been cited as both a pop and a cultural icon as well as the quintessential American sex symbol.", "name": "Marilyn Monroe", "birthdate": "June 01, 1926"},
{"bio": "Anna Eleanor Roosevelt was an American political leader who used her influence as an active First Lady from 1933 to 1945 to promote the New Deal policies of her husband, President Franklin D. Roosevelt, as well as taking a prominent role as an advocate for civil rights. After her husband's death in 1945, she continued to be an internationally prominent author and speaker for the New Deal coalition. She was a suffragist who worked to enhance the status of working women, although she opposed the Equal Rights Amendment because she believed it would adversely affect women. In the 1940s, she was one of the co-founders of Freedom House and supported the formation of the United Nations. Eleanor Roosevelt founded the UN Association of the United States in 1943 to advance support for the formation of the UN. She was a delegate to the UN General Assembly from 1945 and 1952, a job for which she was appointed by President Harry S. Truman and confirmed by the United States Congress. During her time at the United Nations chaired the committee that drafted and approved the Universal Declaration of Human Rights. President Truman called her the \"First Lady of the World\" in tribute to her human rights achievements.She was one of the most admired persons of the 20th century, according to Gallup's List of Widely Admired People.", "name": "Eleanor Roosevelt", "birthdate": "October 11, 1884"},
...
]

scrapy crawl quotes -o quotes-humor.json -a tag=humor

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        url = 'http://quotes.toscrape.com/'
        tag = getattr(self, 'tag', None)
        if tag is not None:
            url = url + 'tag/' + tag
        yield scrapy.Request(url, self.parse)

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').extract_first(),
                'author': quote.css('small.author::text').extract_first(),
            }

        next_page = response.css('li.next a::attr(href)').extract_first()
        if next_page is not None:
            yield response.follow(next_page, self.parse)