Scrapy框架下第一个爬虫

Stanley Sun

发布于 2019-09-23 15:00:27

3630

发布于 2019-09-23 15:00:27

1. 安装scrapy

pip install scrapy

2. 第一个爬虫代码myspider.py

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('h2.entry-title'):
            yield {'title': title.css('a ::text').extract_first()}

        for next_page in response.css('div.prev-post > a'):
            yield response.follow(next_page, self.parse)

3. 运行

scrapy runspider myspider.py -o result.json

4. 运行结果

017-08-06 17:44:56 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: scrapybot)
2017-08-06 17:44:56 [scrapy.utils.log] INFO: Overridden settings: {'SPIDER_LOADER_WARN_ONLY': True}
2017-08-06 17:44:56 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.memusage.MemoryUsage',

...

2017-08-06 17:44:56 [scrapy.core.engine] INFO: Spider opened
2017-08-06 17:44:56 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-08-06 17:44:56 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-08-06 17:45:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://blog.scrapinghub.com> (referer: None)
2017-08-06 17:45:01 [scrapy.core.scraper] DEBUG: Scraped from <200 https://blog.scrapinghub.com>
{'title': u'Scraping the Steam Game Store with Scrapy'}
2017-08-06 17:45:01 [scrapy.core.scraper] DEBUG: Scraped from <200 https://blog.scrapinghub.com>
{'title': u'Do Androids Dream of Electric Sheep?'}

...

2017-08-06 17:45:12 [scrapy.core.scraper] DEBUG: Scraped from <200 https://blog.scrapinghub.com/page/11/>
{'title': u'Hello, world'}
2017-08-06 17:45:12 [scrapy.core.engine] INFO: Closing spider (finished)
2017-08-06 17:45:12 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 2933,
 'downloader/request_count': 11,
 'downloader/request_method_count/GET': 11,
 'downloader/response_bytes': 123915,
 'downloader/response_count': 11,
 'downloader/response_status_count/200': 11,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 8, 6, 9, 45, 12, 168728),
 'item_scraped_count': 105,
 'log_count/DEBUG': 117,
 'log_count/INFO': 7,
 'memusage/max': 43909120,
 'memusage/startup': 43909120,
 'request_depth_max': 10,
 'response_received_count': 11,
 'scheduler/dequeued': 11,
 'scheduler/dequeued/memory': 11,
 'scheduler/enqueued': 11,
 'scheduler/enqueued/memory': 11,
 'start_time': datetime.datetime(2017, 8, 6, 9, 44, 56, 752503)}
2017-08-06 17:45:12 [scrapy.core.engine] INFO: Spider closed (finished)

生成结果的文件result.json

[
{"title": "Scraping the Steam Game Store with Scrapy"},
{"title": "Do Androids Dream of Electric Sheep?"},
{"title": "Deploy your Scrapy Spiders from GitHub"},
{"title": "Looking Back at 2016"},
{"title": "How to Increase Sales with Online Reputation Management"},
{"title": "How to Build your own Price Monitoring Tool"},

...

{"title": "Spoofing your Scrapy bot IP using tsocks"},
{"title": "Hello, world"}
]

解析运行过程

当你运行下面命令时，Scrapy框架会启动爬虫引擎，根据myspider.py中的逻辑进行抓取网页，然后把结果存到result.json中。

scrapy runspider myspider.py -o result.json

第一步：爬虫先请求start_urls中定义到URLs。本例中，只有一个URL。请求该URL返回内容，如下所示。

...

<h2 class="entry-title">
	<a href="https://blog.scrapinghub.com/2017/07/07/scraping-the-steam-game-store-with-scrapy/" rel="bookmark">Scraping the Steam Game Store with Scrapy</a>
</h2>

...

<h2 class="entry-title">
  <a href="https://blog.scrapinghub.com/2016/10/06/interview-how-up-hail-uses-scrapy-to-increase-transparency/" rel="bookmark">Interview: How Up Hail uses Scrapy to Increase Transparency</a>
</h2>

...


<div class="col-md-6 prev-post">		
  <a href="https://blog.scrapinghub.com/page/2/">OLDER POST<i class="fa fa-angle-double-right"></i></a>				
</div>

...

第二步：调用callback方法parse，解析返回到内容。在回调函数parse中，爬虫循环使用CSS选择器 h2.entry-title，找出网页中的<h2 class="entry-title">元素，并这些元素中的text找出来，生成Python的dict格式数据。

当前页中所有的h2.entry-title元素找完以后，爬虫会执行下面的代码

for next_page in response.css('div.prev-post > a'):
            yield response.follow(next_page, self.parse)

它会找div.prev-post元素，然后再次递归执行parse方法。从网页上可以看到，div.prev-post元素是翻页链接，所以，爬虫会不断翻页，知道抓取所有的网页。

第三步：生成的字典数据存到result.json文件中

scrapy的任务是异步执行的，也就是说，它不用等一个请求返回以后才发送另一个请求，而是可以同时进行的。这可以加快运行速度。我们也可以对Scrapy进行设置，比如每一个请求延迟一段时间，等等。

本文参与腾讯云自媒体分享计划，分享自作者个人站点/博客。

如有侵权请联系 cloudcommunity@tencent.com 删除

javascript

本文分享自作者个人站点/博客前往查看

如有侵权，请联系 cloudcommunity@tencent.com 删除。

本文参与腾讯云自媒体分享计划，欢迎热爱写作的你一起参与！

登录后参与评论

0 条评论

热度