blocks|key|498264|text|在the+solution+from+Pablo+Hoffman的基础上，您可以在管道对象的process_item方法上使用以下修饰器，以便它检查爬行器的pipeline属性，以确定是否应该执行它。例如：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|498265|def+check_spider_pipeline(process_item_method):

++++@functools.wraps(process_item_method)
++++def+wrapper(self,+item,+spider):

++++++++#+message+template+for+debugging
++++++++msg+=+'%25%25s+%25s+pipeline+step'+%25+(self.__class__.__name__,)

++++++++#+if+class+is+in+the+spider's+pipeline,+then+use+the
++++++++#+process_item+method+normally.
++++++++if+self.__class__+in+spider.pipeline:
++++++++++++spider.log(msg+%25+'executing',+level=log.DEBUG)
++++++++++++return+process_item_method(self,+item,+spider)

++++++++#+otherwise,+just+return+the+untouched+item+(skip+this+step+in
++++++++#+the+pipeline)
++++++++else:
++++++++++++spider.log(msg+%25+'skipping',+level=log.DEBUG)
++++++++++++return+item

++++return+wrapper|code-block|syntax|javascript|498266|要使此装饰器正常工作，爬行器必须具有一个pipeline属性，其中包含您要用于处理项目的Pipeline对象的容器，例如：|498267|class+MySpider(BaseSpider):

++++pipeline+=+set([
++++++++pipelines.Save,
++++++++pipelines.Validate,
++++])

++++def+parse(self,+response):
++++++++#+insert+scrapy+goodness+here
++++++++return+item|498268|然后在pipelines.py文件中：|498269|class+Save(object):

++++@check_spider_pipeline
++++def+process_item(self,+item,+spider):
++++++++#+do+saving+here
++++++++return+item

class+Validate(object):

++++@check_spider_pipeline
++++def+process_item(self,+item,+spider):
++++++++#+do+validating+here
++++++++return+item|498270|所有管道对象仍然应该在ITEM_PIPELINES中的设置中定义(按照正确的顺序--最好更改一下，以便也可以在爬行器上指定顺序)。|498271|entityMap|0|LINK|mutability|MUTABLE|url|http://groups.google.com/group/scrapy-users/browse_thread/thread/9ac290ed469887f1^0|1A|C|26|8|1|V|0|0|0|0|0|3|C|0|0|0^^$0|@$1|2|3|4|5|6|7|12|8|@$9|13|A|14|B|C]|$9|15|A|16|B|C]]|D|@$9|17|A|18|1|19]]|E|$]]|$1|F|3|G|5|H|7|1A|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|1B|8|@]|D|@]|E|$]]|$1|M|3|N|5|H|7|1C|8|@]|D|@]|E|$I|J]]|$1|O|3|P|5|6|7|1D|8|@$9|1E|A|1F|B|C]]|D|@]|E|$]]|$1|Q|3|R|5|H|7|1G|8|@]|D|@]|E|$I|J]]|$1|S|3|T|5|6|7|1H|8|@]|D|@]|E|$]]|$1|U|3|-4|5|6|7|1I|8|@]|D|@]|E|$]]]|V|$W|$5|X|Y|Z|E|$10|11]]]]

Building on <a href="http://groups.google.com/group/scrapy-users/browse_thread/thread/9ac290ed469887f1" rel="noreferrer">the solution from Pablo Hoffman</a>, you can use the following decorator on the <code>process_item</code> method of a Pipeline object so that it checks the <code>pipeline</code> attribute of your spider for whether or not it should be executed. For example:

<pre><code>def check_spider_pipeline(process_item_method):

 @functools.wraps(process_item_method)
 def wrapper(self, item, spider):

 # message template for debugging
 msg = '%%s %s pipeline step' % (self.__class__.__name__,)

 # if class is in the spider's pipeline, then use the
 # process_item method normally.
 if self.__class__ in spider.pipeline:
 spider.log(msg % 'executing', level=log.DEBUG)
 return process_item_method(self, item, spider)

 # otherwise, just return the untouched item (skip this step in
 # the pipeline)
 else:
 spider.log(msg % 'skipping', level=log.DEBUG)
 return item

 return wrapper
</code></pre>

For this decorator to work correctly, the spider must have a pipeline attribute with a container of the Pipeline objects that you want to use to process the item, for example:

<pre><code>class MySpider(BaseSpider):

 pipeline = set([
 pipelines.Save,
 pipelines.Validate,
 ])

 def parse(self, response):
 # insert scrapy goodness here
 return item
</code></pre>

And then in a <code>pipelines.py</code> file:

<pre><code>class Save(object):

 @check_spider_pipeline
 def process_item(self, item, spider):
 # do saving here
 return item

class Validate(object):

 @check_spider_pipeline
 def process_item(self, item, spider):
 # do validating here
 return item
</code></pre>

All Pipeline objects should still be defined in ITEM_PIPELINES in settings (in the correct order -- would be nice to change so that the order could be specified on the Spider, too).

blocks|key|713794|text|您可以在管道中使用爬行器的name属性|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|713795|class+CustomPipeline(object)

++++def+process_item(self,+item,+spider)
+++++++++if+spider.name+==+'spider1':
+++++++++++++#+do+something
+++++++++++++return+item
+++++++++return+item|code-block|syntax|javascript|713796|以这种方式定义所有管道可以实现您想要的功能。|713797|entityMap^0|D|4|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@$9|P|A|Q|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|R|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|S|8|@]|D|@]|E|$]]|$1|M|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|N|$]]

You can use the <code>name</code> attribute of the spider in your pipeline

<pre><code>class CustomPipeline(object)

 def process_item(self, item, spider)
 if spider.name == 'spider1':
 # do something
 return item
 return item
</code></pre>

Defining all pipelines this way can accomplish what you want.

blocks|key|498367|text|只需从主设置中删除所有管道，并使用此内部爬行器。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|498368|这将定义到每个爬行器的用户的管道|498369|class+testSpider(InitSpider):
++++name+=+'test'
++++custom_settings+=+{
++++++++'ITEM_PIPELINES':+{
++++++++++++'app.MyPipeline':+400
++++++++}
++++}|code-block|syntax|javascript|498370|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|L|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|M|8|@]|9|@]|A|$G|H]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

Just remove all pipelines from main settings and use this inside spider.

This will define the pipeline to user per spider

<pre><code>class testSpider(InitSpider):
 name = 'test'
 custom_settings = {
 'ITEM_PIPELINES': {
 'app.MyPipeline': 400
 }
 }
</code></pre>

blocks|key|498407|text|我使用了两个管道，一个用于图像下载(MyImagesPipeline)，另一个用于在mongodb中保存数据(MongoPipeline)。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|498408|假设我们有许多爬行器(+spider1，spider2，...........)，在我的示例中，spider1和spider5不能使用MyImagesPipeline|498409|settings.py|498410|ITEM_PIPELINES+=+{'scrapycrawler.pipelines.MyImagesPipeline'+:+1,'scrapycrawler.pipelines.MongoPipeline'+:+2}
IMAGES_STORE+=+'/var/www/scrapycrawler/dowload'|code-block|syntax|javascript|498411|和下面的完整代码的管道|498412|import+scrapy
import+string
import+pymongo
from+scrapy.pipelines.images+import+ImagesPipeline

class+MyImagesPipeline(ImagesPipeline):
++++def+process_item(self,+item,+spider):
++++++++if+spider.name+not+in+['spider1',+'spider5']:
++++++++++++return+super(ImagesPipeline,+self).process_item(item,+spider)
++++++++else:
+++++++++++return+item+

++++def+file_path(self,+request,+response=None,+info=None):
++++++++image_name+=+string.split(request.url,+'/')[-1]
++++++++dir1+=+image_name[0]
++++++++dir2+=+image_name[1]
++++++++return+dir1+%2B+'/'+%2B+dir2+%2B+'/'+%2Bimage_name

class+MongoPipeline(object):

++++collection_name+=+'scrapy_items'
++++collection_url='snapdeal_urls'

++++def+__init__(self,+mongo_uri,+mongo_db):
++++++++self.mongo_uri+=+mongo_uri
++++++++self.mongo_db+=+mongo_db

++++@classmethod
++++def+from_crawler(cls,+crawler):
++++++++return+cls(
++++++++++++mongo_uri=crawler.settings.get('MONGO_URI'),
++++++++++++mongo_db=crawler.settings.get('MONGO_DATABASE',+'scraping')
++++++++)

++++def+open_spider(self,+spider):
++++++++self.client+=+pymongo.MongoClient(self.mongo_uri)
++++++++self.db+=+self.client[self.mongo_db]

++++def+close_spider(self,+spider):
++++++++self.client.close()

++++def+process_item(self,+item,+spider):
++++++++#self.db[self.collection_name].insert(dict(item))
++++++++collection_name=item.get(+'collection_name',+self.collection_name+)
++++++++self.db[collection_name].insert(dict(item))
++++++++data+=+{}
++++++++data['base_id']+=+item['base_id']
++++++++self.db[self.collection_url].update({
++++++++++++'base_id':+item['base_id']
++++++++},+{
++++++++++++'$set':+{
++++++++++++'image_download':+1
++++++++++++}
++++++++},+upsert=False,+multi=True)
++++++++return+item|498413|entityMap^0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|R|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|S|8|@]|9|@]|A|$]]|$1|F|3|G|5|H|7|T|8|@]|9|@]|A|$I|J]]|$1|K|3|L|5|6|7|U|8|@]|9|@]|A|$]]|$1|M|3|N|5|H|7|V|8|@]|9|@]|A|$I|J]]|$1|O|3|-4|5|6|7|W|8|@]|9|@]|A|$]]]|P|$]]

I am using two pipelines, one for image download (MyImagesPipeline) and second for save data in mongodb (MongoPipeline).

suppose we have many spiders(spider1,spider2,...........),in my example spider1 and spider5 can not use MyImagesPipeline

settings.py

<pre><code>ITEM_PIPELINES = {'scrapycrawler.pipelines.MyImagesPipeline' : 1,'scrapycrawler.pipelines.MongoPipeline' : 2}
IMAGES_STORE = '/var/www/scrapycrawler/dowload'
</code></pre>

And bellow complete code of pipeline

<pre><code>import scrapy
import string
import pymongo
from scrapy.pipelines.images import ImagesPipeline

class MyImagesPipeline(ImagesPipeline):
 def process_item(self, item, spider):
 if spider.name not in ['spider1', 'spider5']:
 return super(ImagesPipeline, self).process_item(item, spider)
 else:
 return item 

 def file_path(self, request, response=None, info=None):
 image_name = string.split(request.url, '/')[-1]
 dir1 = image_name[0]
 dir2 = image_name[1]
 return dir1 + '/' + dir2 + '/' +image_name

class MongoPipeline(object):

 collection_name = 'scrapy_items'
 collection_url='snapdeal_urls'

 def __init__(self, mongo_uri, mongo_db):
 self.mongo_uri = mongo_uri
 self.mongo_db = mongo_db

 @classmethod
 def from_crawler(cls, crawler):
 return cls(
 mongo_uri=crawler.settings.get('MONGO_URI'),
 mongo_db=crawler.settings.get('MONGO_DATABASE', 'scraping')
 )

 def open_spider(self, spider):
 self.client = pymongo.MongoClient(self.mongo_uri)
 self.db = self.client[self.mongo_db]

 def close_spider(self, spider):
 self.client.close()

 def process_item(self, item, spider):
 #self.db[self.collection_name].insert(dict(item))
 collection_name=item.get( 'collection_name', self.collection_name )
 self.db[collection_name].insert(dict(item))
 data = {}
 data['base_id'] = item['base_id']
 self.db[self.collection_url].update({
 'base_id': item['base_id']
 }, {
 '$set': {
 'image_download': 1
 }
 }, upsert=False, multi=True)
 return item
</code></pre>

blocks|key|713960|text|我们可以在管道中使用一些条件，如下所示|type|unstyled|depth|inlineStyleRanges|entityRanges|data|713961|++++#+-*-+coding:+utf-8+-*-
from+scrapy_app.items+import+x

class+SaveItemPipeline(object):
++++def+process_item(self,+item,+spider):
++++++++if+isinstance(item,+x,):
++++++++++++item.save()
++++++++return+item|code-block|syntax|javascript|713962|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

we can use some conditions in pipeline as this

<pre><code> # -*- coding: utf-8 -*-
from scrapy_app.items import x

class SaveItemPipeline(object):
 def process_item(self, item, spider):
 if isinstance(item, x,):
 item.save()
 return item
</code></pre>

blocks|key|498511|text|您可以像下面这样在爬行器中设置项目管道设置：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|498512|class+CustomSpider(Spider):
++++name+=+'custom_spider'
++++custom_settings+=+{
++++++++'ITEM_PIPELINES':+{
++++++++++++'__main__.PagePipeline':+400,
++++++++++++'__main__.ProductPipeline':+300,
++++++++},
++++++++'CONCURRENT_REQUESTS_PER_DOMAIN':+2
++++}|code-block|syntax|javascript|498513|然后，我可以通过向加载器/返回项添加一个值来拆分管道(甚至使用多个管道)，该值标识爬行器的哪个部分发送了项。这样我就不会得到任何KeyError异常，并且我知道哪些项目应该是可用的。|498514|++++...
++++def+scrape_stuff(self,+response):
++++++++pageloader+=+PageLoader(
++++++++++++++++PageItem(),+response=response)

++++++++pageloader.add_xpath('entire_page',+'/html//text()')
++++++++pageloader.add_value('item_type',+'page')
++++++++yield+pageloader.load_item()

++++++++productloader+=+ProductLoader(
++++++++++++++++ProductItem(),+response=response)

++++++++productloader.add_xpath('product_name',+'//span[contains(text(),+"Example")]')
++++++++productloader.add_value('item_type',+'product')
++++++++yield+productloader.load_item()

class+PagePipeline:
++++def+process_item(self,+item,+spider):
++++++++if+item['item_type']+==+'product':
++++++++++++#+do+product+stuff

++++++++if+item['item_type']+==+'page':
++++++++++++#+do+page+stuff|498515|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

You can just set the item pipelines settings inside of the spider like this:

<pre><code>class CustomSpider(Spider):
 name = 'custom_spider'
 custom_settings = {
 'ITEM_PIPELINES': {
 '__main__.PagePipeline': 400,
 '__main__.ProductPipeline': 300,
 },
 'CONCURRENT_REQUESTS_PER_DOMAIN': 2
 }
</code></pre>

I can then split up a pipeline (or even use multiple pipelines) by adding a value to the loader/returned item that identifies which part of the spider sent items over. This way I won’t get any KeyError exceptions and I know which items should be available. 

<pre><code> ...
 def scrape_stuff(self, response):
 pageloader = PageLoader(
 PageItem(), response=response)

 pageloader.add_xpath('entire_page', '/html//text()')
 pageloader.add_value('item_type', 'page')
 yield pageloader.load_item()

 productloader = ProductLoader(
 ProductItem(), response=response)

 productloader.add_xpath('product_name', '//span[contains(text(), "Example")]')
 productloader.add_value('item_type', 'product')
 yield productloader.load_item()

class PagePipeline:
 def process_item(self, item, spider):
 if item['item_type'] == 'product':
 # do product stuff

 if item['item_type'] == 'page':
 # do page stuff
</code></pre>

blocks|key|498548|text|简单但仍然有用的解决方案。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|498549|爬行器代码|498550|++++def+parse(self,+response):
++++++++item+=+{}
++++++++...+do+parse+stuff
++++++++item['info']+=+{'spider':+'Spider2'}|code-block|syntax|javascript|498551|流水线代码|498552|++++def+process_item(self,+item,+spider):
++++++++if+item['info']['spider']+==+'Spider1':
++++++++++++logging.error('Spider1+pipeline+works')
++++++++elif+item['info']['spider']+==+'Spider2':
++++++++++++logging.error('Spider2+pipeline+works')
++++++++elif+item['info']['spider']+==+'Spider3':
++++++++++++logging.error('Spider3+pipeline+works')|498553|希望这能为某些人节省一些时间！|498554|entityMap^0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|R|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|S|8|@]|9|@]|A|$G|H]]|$1|I|3|J|5|6|7|T|8|@]|9|@]|A|$]]|$1|K|3|L|5|F|7|U|8|@]|9|@]|A|$G|H]]|$1|M|3|N|5|6|7|V|8|@]|9|@]|A|$]]|$1|O|3|-4|5|6|7|W|8|@]|9|@]|A|$]]]|P|$]]

Simple but still useful solution.

Spider code

<pre><code> def parse(self, response):
 item = {}
 ... do parse stuff
 item['info'] = {'spider': 'Spider2'}
</code></pre>

pipeline code

<pre><code> def process_item(self, item, spider):
 if item['info']['spider'] == 'Spider1':
 logging.error('Spider1 pipeline works')
 elif item['info']['spider'] == 'Spider2':
 logging.error('Spider2 pipeline works')
 elif item['info']['spider'] == 'Spider3':
 logging.error('Spider3 pipeline works')
</code></pre>

Hope this save some time for somebody!

blocks|key|714067|text|最简单有效的解决方案是在每个爬行器本身中设置自定义设置。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|714068|custom_settings+=+{'ITEM_PIPELINES':+{'project_name.pipelines.SecondPipeline':+300}}|code-block|syntax|javascript|714069|之后，您需要在settings.py文件中设置它们|714070|ITEM_PIPELINES+=+{
+++'project_name.pipelines.FistPipeline':+300,
+++'project_name.pipelines.SecondPipeline':+400
}|714071|通过这种方式，每个爬行器将使用各自的管道。|714072|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|R|8|@]|9|@]|A|$E|F]]|$1|K|3|L|5|6|7|S|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|N|$]]

The most simple and effective solution is to set custom settings in each spider itself.
<pre><code>custom_settings = {'ITEM_PIPELINES': {'project_name.pipelines.SecondPipeline': 300}}
</code></pre>
After that you need to set them in the settings.py file
<pre><code>ITEM_PIPELINES = {
 'project_name.pipelines.FistPipeline': 300,
 'project_name.pipelines.SecondPipeline': 400
}
</code></pre>
in that way each spider will use the respective pipeline.

I have a scrapy project which contains multiple spiders.
Is there any way I can define which pipelines to use for which spider? Not all the pipelines i have defined are applicable for every spider.

Thanks

How can I use different pipelines for different spiders in a single Scrapy project

容器

我有一个包含多个蜘蛛的抓取项目。有没有什么方法可以定义哪个管道用于哪个爬虫？并不是我定义的所有管道都适用于每个爬行器。谢谢

问如何在一个Scrapy项目中为不同的爬行器使用不同的管道
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在一个Scrapy项目中为不同的爬行器使用不同的管道EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在一个Scrapy项目中为不同的爬行器使用不同的管道
EN