(配置型爬虫)
什么是爬虫框架?
格式
scrapy <command> [options] [args]
** 常用命令 **
命令 | 说明 | 格式 |
---|---|---|
startproject | 创建一个新工程 | scrapy startproject [dir] |
genspider | 创建一个爬虫 | scrapy genspider [options] [domain] |
settings | 获得爬虫配置信息 | scrapy settings [options] |
crawl | 运行一个爬虫 | scrapy crawl |
list | 列出工程中所有的爬虫 | scrapy list |
shell | 启动URL调试命令行 | scrapy shell [url] |
scrapy startproject python123demo
scrapy genspider demo python123.io
//生成了一个名为demo的spider
//在spider目录下增加代码文件demo.py(该文件也可以手工生成)
** demo.py文件 **
# -*- coding: utf-8 -*-
import scrapy
class DemoSpider(scrapy.Spider):
name = 'demo'
allowed_domains = ['python123.io']
start_urls = ['http://python123.io/']
def parse(self, response):
pass
# -*- coding: utf-8 -*-
import scrapy
class DemoSpider(scrapy.Spider):
name = 'demo'
#allowed_domains = ['python123.io']
start_urls = ['http://python123.io/ws/demo.html']
def parse(self, response):
#存储文件名demo.html
file_name = response.url.split('/')[-1]
with open(file_name,"wb") as f:
f.write(response.body)
self.log('Saved file %s' % file_name)#日志
*** 另一个版本 **
# -*- coding: utf-8 -*-
import scrapy
class DemoSpider(scrapy.Spider):
name = 'demo'
#allowed_domains = ['python123.io']
#start_urls = ['http://python123.io/ws/demo.html']
def start_requests(self):
urls = [
'http://python123.io/ws/demo.html'
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
#存储文件名demo.html
file_name = response.url.split('/')[-1]
with open(file_name,"wb") as f:
f.write(response.body)
self.log('Saved file %s' % file_name)#日志
scrapy crawl demo
class scrapy.http.Request()
属性 | 方法 |
---|---|
.url | Requests对应的请求URL地址 |
.method | 对应的请求方法,’GEt’、’POST’等 |
.headers | 字典类型风格的请求头 |
.body | 请求内容主体,字符串类型 |
.meta | 用户添加的扩展信息,在Scrapy内部模块间传递信息使用 |
.copy | 复制该请求 |
class scrapy.http.Response()
属性或方法 | 说明 |
---|---|
.url | Response对应的URL地址 |
.status | HTTP状态码,默认是200 |
.headers | Response对应的头部信息 |
.body | Response对应的内容信息,字符串类型 |
.flags | 一组标记 |
.request | 产生Response类型对应的Request对象 |
.copy() | 复制该响应 |
class scrapy.item.Item()