blocks|key|736620|text|显然，最好的方法是在CloudFlare中白名单您的IP；如果不合适，让我推荐云焰-擦伤库。您可以使用它获取cookie令牌，然后在刮伤请求中将此cookie令牌提供给服务器。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|736621|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/Anorov/cloudflare-scrape|1|http://doc.scrapy.org/en/latest/topics/request-response.html^0|13|5|0|1U|4|1|0^^$0|@$1|2|3|4|5|6|7|N|8|@]|9|@$A|O|B|P|1|Q]|$A|R|B|S|1|T]]|C|$]]|$1|D|3|-4|5|6|7|U|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]|L|$5|G|H|I|C|$J|M]]]]

Obviously the best way to do this would be to whitelist your IP in CloudFlare; if this isn't suitable let me recommend the <a href="https://github.com/Anorov/cloudflare-scrape" rel="noreferrer">cloudflare-scrape</a> library. You can use this to get the cookie token, then provide this cookie token in your <a href="http://doc.scrapy.org/en/latest/topics/request-response.html" rel="noreferrer">Scrapy request</a> back to the server.

blocks|key|1898622|text|因此，我在JavaScript的帮助下使用Python执行了云焰-擦伤。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1898623|在刮板中，需要添加以下代码：|1898624|def+start_requests(self):
++for+url+in+self.start_urls:
++++token,+agent+=+cfscrape.get_tokens(url,+'Your+prefarable+user+agent,+_optional_')
++++yield+Request(url=url,+cookies=token,+headers={'User-Agent':+agent})|code-block|syntax|javascript|1898625|以及解析函数。就这样！|1898626|当然，您需要先安装cloudflare，然后将它导入到您的蜘蛛中。您还需要安装一个JS执行引擎。我已经有Node.JS了，没有抱怨。|1898627|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/Anorov/cloudflare-scrape^0|U|5|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|W|8|@]|9|@$A|X|B|Y|1|Z]]|C|$]]|$1|D|3|E|5|6|7|10|8|@]|9|@]|C|$]]|$1|F|3|G|5|H|7|11|8|@]|9|@]|C|$I|J]]|$1|K|3|L|5|6|7|12|8|@]|9|@]|C|$]]|$1|M|3|N|5|6|7|13|8|@]|9|@]|C|$]]|$1|O|3|-4|5|6|7|14|8|@]|9|@]|C|$]]]|P|$Q|$5|R|S|T|C|$U|V]]]]

So I executed JavaScript using Python with help of <a href="https://github.com/Anorov/cloudflare-scrape" rel="noreferrer">cloudflare-scrape</a>.

To your scraper, you need to add the following code:

<pre><code>def start_requests(self):
 for url in self.start_urls:
 token, agent = cfscrape.get_tokens(url, 'Your prefarable user agent, _optional_')
 yield Request(url=url, cookies=token, headers={'User-Agent': agent})
</code></pre>

alongside parsing functions. And that's it!

Of course, you need to install cloudflare-scrape first and import it to your spider. You also need a JS execution engine installed. I had Node.JS already, no complaints.

blocks|key|1345360|text|如果您得到503个错误，您可以遵循以下指导原则：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1345361|转到settings.py|ordered-list-item|offset|length|style|CODE|1345362|搜索：USER_AGENT|1345363|在这里，您将看到默认的bot用户代理。将默认值替换为:+USER_AGENT+=+'Mozilla/5.0+(WindowsNT10.0；Win64；x64)+AppleWebKit/537.36+(KHTML，类似壁虎)+Chrome/83.0.4103.116+Safari/537.36‘|BOLD|1345364|entityMap^0|0|2|B|0|3|A|0|8|A|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|Q|8|@$E|R|F|S|G|H]]|9|@]|A|$]]|$1|I|3|J|5|D|7|T|8|@$E|U|F|V|G|H]]|9|@]|A|$]]|$1|K|3|L|5|D|7|W|8|@$E|X|F|Y|G|M]]|9|@]|A|$]]|$1|N|3|-4|5|6|7|Z|8|@]|9|@]|A|$]]]|O|$]]

If you're getting 503 Error you can follow these guidelines:
<ol>
<li>Go to <code>settings.py</code></li>
<li>Search for: <code>USER_AGENT</code></li>
<li>Here you will see the default bot user agent by scrapy.
Replace that default with this:
<pre><code>USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'
</code></pre>
</li>
</ol>

I used to scrape e-commerce webpage occasionally to get product prices information. I have not used the scraper built using Scrapy in a while and yesterday was trying to use it - I run into a problem with bot protection. 

It is using CloudFlare’s DDOS protection which is basically using JavaScript evaluation to filter out the browsers (and therefore scrapers) with JS disabled. Once the function is evaluated, the response with calculated number is generated. In return, service sends back two authentication cookies which attached to each request allow to normally crawl the site. <a href="https://khromov.wordpress.com/2013/02/05/cloudflare-client-side-ddos-detection-how-does-it-work/" rel="noreferrer">Here</a>'s the description of how it works.

I have also found a <a href="https://github.com/Anorov/cloudflare-scrape" rel="noreferrer">cloudflare-scrape</a> Python module that uses external JS evaluation engine to calculate the number and send the request back to server. I'm not sure how to integrate it into <a href="http://scrapy.org" rel="noreferrer">Scrapy</a> though. Or maybe there's a smarter way without using JS execution? In the end, it's a form...

I'd apriciate any help.

How to bypass cloudflare bot/ddos protection in Scrapy?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋 

腾讯云代码助手

CODING DevOps

Cloud Studio

SDK中心

API中心

命令行工具

我以前偶尔会刮电子商务网页来获取产品价格信息。我已经有一段时间没有使用用Scrapy建造的铲运机了，昨天我试着使用它--我遇到了机器人保护方面的问题。它使用CloudFlare的DDOS保护，这基本上是使用JavaScript评估过滤掉禁用JS的浏览器(因此也是刮板)。一旦对函数进行了计算，就会生成具有计算数量的响应。...

问如何绕过cloudflare bot/ddos保护？
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何绕过cloudflare bot/ddos保护？EN