blocks|key|1525790|text|缓存可以通过两种方式禁用。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1525791|更改setting.py文件中与缓存相关的设置中的值。通过保持HTTPCACHE_ENABLED=False|ordered-list-item|1525792|也可以在运行时“抓取爬行-名称-set+HTTPCACHE_ENABLED=False”中完成。|1525793|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$]]|$1|E|3|F|5|D|7|K|8|@]|9|@]|A|$]]|$1|G|3|-4|5|6|7|L|8|@]|9|@]|A|$]]]|H|$]]

Cache can be disable in 2 ways

<ol>
<li>Changing values in cache related settings in setting.py file. By Keeping HTTPCACHE_ENABLED=False</li>
<li>Or it can be done in runtime " scrapy crawl crawl-name --set HTTPCACHE_ENABLED=False</li>
</ol>

blocks|key|1525804|text|在这里，我假设您只是想避免只缓存特定的请求。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1525805|对于本例，它意味着避免在start_requests下缓存这些请求，并缓存所有其他请求(在parseResponse下可能有)。|offset|length|style|CODE|1525806|要做到这一点，只需将productResponse.meta['dont_cache']+=+True行添加到代码中，并在settings.py下设置HTTPCACHE_ENABLED=True|1525807|现在，所有其他请求都将被缓存。|1525808|def+start_requests(self):
++++++++meta+=+{'REDIRECT_ENABLED':True}
++++++++productUrl+=+"http://xyz"
++++++++cookies+=+[{'name':+'',+'value':+'=='},{'name':+'',+'value':+'=='}]
++++++++for+cook+in+cookies:

++++++++++++header+=+{"User-Agent":"Mozilla/5.0+(Windows+NT+6.3;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/45.0.2454.85+Safari/537.36"}
++++++++++++productResponse+=+scrapy.Request(productUrl,callback=self.parseResponse,method='GET',
+++++++++++++++++++++++++++++++++++++++++++++meta=meta,body=str(),cookies=[cook],
+++++++++++++++++++++++++++++++++++++++++++++encoding='utf-8',priority=0,dont_filter=True)
++++++++++++productResponse.meta['dont_cache']+=+True
++++++++++++yield+productResponse

def+parseResponse(self,response):+
+++++selector+=+Selector(response)
+++++print+selector.xpath("xpaths+here").extract()
+++++yield+None|code-block|syntax|javascript|1525809|entityMap^0|0|C|E|19|D|0|A|15|1P|B|23|M|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|T|8|@$D|U|E|V|F|G]|$D|W|E|X|F|G]]|9|@]|A|$]]|$1|H|3|I|5|6|7|Y|8|@$D|Z|E|10|F|G]|$D|11|E|12|F|G]|$D|13|E|14|F|G]]|9|@]|A|$]]|$1|J|3|K|5|6|7|15|8|@]|9|@]|A|$]]|$1|L|3|M|5|N|7|16|8|@]|9|@]|A|$O|P]]|$1|Q|3|-4|5|6|7|17|8|@]|9|@]|A|$]]]|R|$]]

Here I assume that you just want to avoid caching only specific requests.

For this example it means avoid caching those requests under <code>start_requests</code> and cache all other requests (which you may have under <code>parseResponse</code>).

To do this just add <code>productResponse.meta['dont_cache'] = True</code> line to your code and 
set <code>HTTPCACHE_ENABLED=True</code> under <code>settings.py</code>

Now all other requests will be cached.

<pre><code>def start_requests(self):
 meta = {'REDIRECT_ENABLED':True}
 productUrl = "http://xyz"
 cookies = [{'name': '', 'value': '=='},{'name': '', 'value': '=='}]
 for cook in cookies:

 header = {"User-Agent":"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36"}
 productResponse = scrapy.Request(productUrl,callback=self.parseResponse,method='GET',
 meta=meta,body=str(),cookies=[cook],
 encoding='utf-8',priority=0,dont_filter=True)
 productResponse.meta['dont_cache'] = True
 yield productResponse

def parseResponse(self,response): 
 selector = Selector(response)
 print selector.xpath("xpaths here").extract()
 yield None
</code></pre>

blocks|key|1525826|text|只需向请求url添加一个虚拟参数即可。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1525827|import+random

productUrl+=+"http://xyz"+%2B+"?dummy="+%2B+str(random.random())|code-block|syntax|javascript|1525828|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Just add a dummy parameter to the request url
<pre class="lang-py prettyprint-override"><code>import random

productUrl = &quot;http://xyz&quot; + &quot;?dummy=&quot; + str(random.random())
</code></pre>

I am trying to crawl a webpage on a particular website.The webpage varies a little for different set of cookies that I sent through <code>scrapy.Request()</code>.

If I make the request to webpage one by one , it gives me the correct result, but when I send these cookies in for loop, it is giving me the same result. I think scrapy is creating cache for me and in the second request its taking the response from that cache.Here is my code : 

<pre><code>def start_requests(self):
 meta = {'REDIRECT_ENABLED':True}
 productUrl = "http://xyz"
 cookies = [{'name': '', 'value': '=='},{'name': '', 'value': '=='}]
 for cook in cookies:

 header = {"User-Agent":"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36"}
 productResponse = scrapy.Request(productUrl,callback=self.parseResponse,method='GET',meta=meta,body=str(),cookies=[cook],encoding='utf-8',priority=0,dont_filter=True)
 yield productResponse


def parseResponse(self,response): 
 selector = Selector(response)
 print selector.xpath("xpaths here").extract()
 yield None
</code></pre>

I expect that the print statement should give different result for the two requests.

If anything isn't clear , please mention in comments.

How to disable cache in scrapy?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我试图在一个特定的website.The网页上爬行一个网页，对于我通过scrapy.Request()发送的不同的cookies集，有一些不同的变化。如果我一个一个地请求网页，它会给我正确的结果，但是当我把这些cookie发送到for循环中时，它给了我相同的结果。我认为scrapy正在为我创建缓存，在第二个请求中，它从...

问如何在刮伤中禁用缓存？
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在刮伤中禁用缓存？EN