blocks|key|4660901|text|scrapy命令是一个python脚本，这意味着您可以从PyCharm内部启动它。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|4660902|当您检查scrapy二进制文件(which+scrapy)时，您会注意到这实际上是一个python脚本：|4660903|#!/usr/bin/python

from+scrapy.cmdline+import+execute
execute()|code-block|syntax|javascript|4660904|这意味着像scrapy+crawl+IcecatCrawler这样的命令也可以像这样执行：python+/Library/Python/2.7/site-packages/scrapy/cmdline.py+crawl+IcecatCrawler|4660905|尝试查找scrapy.cmdline包。在我的例子中，位置是这里：/Library/Python/2.7/site-packages/scrapy/cmdline.py|4660906|在PyCharm中创建一个运行/调试配置，并将该脚本作为脚本。使用scrapy命令和爬行器填充脚本参数。在本例中为crawl+IcecatCrawler。|4660907|如下所示：|4660908|📷|atomic|4660909|​|4660910|把你的断点放在你的爬虫代码中的任何地方，它应该可以在™中工作。|4660911|entityMap|0|IMAGE|mutability|IMMUTABLE|imageUrl|https://ask.qcloudimg.com/http-save/yehe-900000/4067eae314c74d2fa916c0266e0954c5.png|imageAlt^0|0|6|0|G|C|0|0|5|Q|19|26|0|X|1F|0|1L|J|0|0|0|1|0|0|0|0^^$0|@$1|2|3|4|5|6|7|1A|8|@$9|1B|A|1C|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|1D|8|@$9|1E|A|1F|B|C]]|D|@]|E|$]]|$1|H|3|I|5|J|7|1G|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|1H|8|@$9|1I|A|1J|B|C]|$9|1K|A|1L|B|C]]|D|@]|E|$]]|$1|O|3|P|5|6|7|1M|8|@$9|1N|A|1O|B|C]]|D|@]|E|$]]|$1|Q|3|R|5|6|7|1P|8|@$9|1Q|A|1R|B|C]]|D|@]|E|$]]|$1|S|3|T|5|6|7|1S|8|@]|D|@]|E|$]]|$1|U|3|V|5|W|7|1T|8|@]|D|@$9|1U|A|1V|1|1W]]|E|$]]|$1|X|3|Y|5|6|7|1X|8|@]|D|@]|E|$]]|$1|Z|3|10|5|6|7|1Y|8|@]|D|@]|E|$]]|$1|11|3|-4|5|6|7|1Z|8|@]|D|@]|E|$]]]|12|$13|$5|14|15|16|E|$17|18|19|-4]]]]

The <code>scrapy</code> command is a python script which means you can start it from inside PyCharm.

When you examine the scrapy binary (<code>which scrapy</code>) you will notice that this is actually a python script:

<pre><code>#!/usr/bin/python

from scrapy.cmdline import execute
execute()
</code></pre>

This means that a command like 
<code>scrapy crawl IcecatCrawler</code> can also be executed like this: <code>python /Library/Python/2.7/site-packages/scrapy/cmdline.py crawl IcecatCrawler</code>

Try to find the scrapy.cmdline package.
In my case the location was here: <code>/Library/Python/2.7/site-packages/scrapy/cmdline.py</code>

Create a run/debug configuration inside PyCharm with that script as script. Fill the script parameters with the scrapy command and spider. In this case <code>crawl IcecatCrawler</code>. 

Like this:
<img src="https://i.stack.imgur.com/bcDQH.png" alt="PyCharm Run/Debug Configuration">

Put your breakpoints anywhere in your crawling code and it should work™.

blocks|key|154836|text|从2018.1开始，这变得容易多了。现在，您可以在项目的Run/Debug+Configuration中选择Module+name。将其设置为scrapy.cmdline，并将Working+directory设置为scrapy项目(其中包含settings.py的项目)的根目录。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|154837|如下所示：|154838|​|154839|📷|atomic|154840|154841|现在，您可以添加断点来调试代码。|154842|entityMap|0|IMAGE|mutability|IMMUTABLE|imageUrl|https://ask.qcloudimg.com/http-save/yehe-900000/87d4c8d21306222f92a1f0fac663c2b6.png|imageAlt^0|S|N|1I|B|1Z|E|2G|H|3D|B|0|0|0|0|1|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Y|8|@$9|Z|A|10|B|C]|$9|11|A|12|B|C]|$9|13|A|14|B|C]|$9|15|A|16|B|C]|$9|17|A|18|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|19|8|@]|D|@]|E|$]]|$1|H|3|I|5|6|7|1A|8|@]|D|@]|E|$]]|$1|J|3|K|5|L|7|1B|8|@]|D|@$9|1C|A|1D|1|1E]]|E|$]]|$1|M|3|I|5|6|7|1F|8|@]|D|@]|E|$]]|$1|N|3|O|5|6|7|1G|8|@]|D|@]|E|$]]|$1|P|3|-4|5|6|7|1H|8|@]|D|@]|E|$]]]|Q|$R|$5|S|T|U|E|$V|W|X|-4]]]]

As of 2018.1 this became a lot easier. You can now select <code>Module name</code> in your project's <code>Run/Debug Configuration</code>. Set this to <code>scrapy.cmdline</code> and the <code>Working directory</code> to the root dir of the scrapy project (the one with <code>settings.py</code> in it).

Like so:

<a href="https://i.stack.imgur.com/mAOiY.png" rel="noreferrer"><img src="https://i.stack.imgur.com/mAOiY.png" alt="PyCharm Scrapy debug configuration"></a>

Now you can add breakpoints to debug your code.

blocks|key|154646|text|我在使用Python3.5.0的虚拟环境中运行scrapy，将"script“参数设置为/path_to_project_env/env/bin/scrapy为我解决了这个问题。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|154647|entityMap^0|18|Z|0^^$0|@$1|2|3|4|5|6|7|H|8|@$9|I|A|J|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|K|8|@]|D|@]|E|$]]]|G|$]]

I am running scrapy in a virtualenv with Python 3.5.0 and setting the "script" parameter to <code>/path_to_project_env/env/bin/scrapy</code> solved the issue for me.

blocks|key|154357|text|我也在使用PyCharm，但我没有使用它的内置调试功能。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|154358|为了调试，我使用了ipdb。我设置了一个键盘快捷键，以便在希望断点出现的任何行上插入import+ipdb;+ipdb.set_trace()。|offset|length|style|CODE|154359|然后，我可以输入n来执行下一条语句，输入s来单步执行一个函数，输入任何对象名来查看它的值，更改执行环境，输入c来继续执行...|154360|这非常灵活，可以在PyCharm以外的环境中工作，在这些环境中，您不需要控制执行环境。|154361|只需键入您的虚拟环境pip+install+ipdb，并将import+ipdb;+ipdb.set_trace()放在您希望暂停执行的行上。|154362|更新|154363|您也可以使用pip+install+pdbpp并使用标准的import+pdb;+pdb.set_trace而不是ipdb。在我看来，PDB%2B%2B更好。|154364|entityMap|0|LINK|mutability|MUTABLE|url|https://pypi.python.org/pypi/ipdb^0|0|9|4|16|T|9|4|0|0|8|1|K|1|1I|1|0|0|A|G|T|T|0|0|6|H|T|P|1L|4|0^^$0|@$1|2|3|4|5|6|7|Z|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|10|8|@$D|11|E|12|F|G]|$D|13|E|14|F|G]]|9|@$D|15|E|16|1|17]]|A|$]]|$1|H|3|I|5|6|7|18|8|@$D|19|E|1A|F|G]|$D|1B|E|1C|F|G]|$D|1D|E|1E|F|G]]|9|@]|A|$]]|$1|J|3|K|5|6|7|1F|8|@]|9|@]|A|$]]|$1|L|3|M|5|6|7|1G|8|@$D|1H|E|1I|F|G]|$D|1J|E|1K|F|G]]|9|@]|A|$]]|$1|N|3|O|5|6|7|1L|8|@]|9|@]|A|$]]|$1|P|3|Q|5|6|7|1M|8|@$D|1N|E|1O|F|G]|$D|1P|E|1Q|F|G]|$D|1R|E|1S|F|G]]|9|@]|A|$]]|$1|R|3|-4|5|6|7|1T|8|@]|9|@]|A|$]]]|S|$T|$5|U|V|W|A|$X|Y]]]]

I am also using PyCharm, but I am not using its built-in debugging features.
For debugging I am using <a href="https://pypi.python.org/pypi/ipdb" rel="nofollow noreferrer"><code>ipdb</code></a>. I set up a keyboard shortcut to insert <code>import ipdb; ipdb.set_trace()</code> on any line I want the break point to happen.
Then I can type <code>n</code> to execute the next statement, <code>s</code> to step into a function, type any object name to see its value, alter execution environment, type <code>c</code> to continue execution...
This is very flexible, works in environments other than PyCharm, where you don't control the execution environment.
Just type in your virtual environment <code>pip install ipdb</code> and place <code>import ipdb; ipdb.set_trace()</code> on a line where you want the execution to pause.
UPDATE
You can also <code>pip install pdbpp</code> and use the standard <code>import pdb; pdb.set_trace</code> instead of <code>ipdb</code>. PDB++ is nicer in my opinion.

blocks|key|4660982|text|为了在被接受的答案中添加一点，几乎一个小时后，我发现我必须从下拉列表(靠近图标工具栏的中心)中选择正确的运行配置，然后单击Debug按钮以使其工作。希望这能有所帮助！|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4660983|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

To add a bit to the accepted answer, after almost an hour I found I had to select the correct Run Configuration from the dropdown list (near the center of the icon toolbar), then click the Debug button in order to get it to work. Hope this helps!

blocks|key|154678|text|根据文档https://doc.scrapy.org/en/latest/topics/practices.html|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|154679|import+scrapy
from+scrapy.crawler+import+CrawlerProcess

class+MySpider(scrapy.Spider):
++++#+Your+spider+definition
++++...

process+=+CrawlerProcess({
++++'USER_AGENT':+'Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1)'
})

process.crawl(MySpider)
process.start()+#+the+script+will+block+here+until+the+crawling+is+finished|code-block|syntax|javascript|154680|entityMap|0|LINK|mutability|MUTABLE|url|https://doc.scrapy.org/en/latest/topics/practices.html^0|4|1I|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

According to the documentation <a href="https://doc.scrapy.org/en/latest/topics/practices.html" rel="nofollow noreferrer">https://doc.scrapy.org/en/latest/topics/practices.html</a>

<pre><code>import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider(scrapy.Spider):
 # Your spider definition
 ...

process = CrawlerProcess({
 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished
</code></pre>

blocks|key|154858|text|我使用这个简单的脚本：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|154859|from+scrapy.crawler+import+CrawlerProcess
from+scrapy.utils.project+import+get_project_settings

process+=+CrawlerProcess(get_project_settings())

process.crawl('your_spider_name')
process.start()|code-block|syntax|javascript|154860|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

I use this simple script:

<pre><code>from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())

process.crawl('your_spider_name')
process.start()
</code></pre>

blocks|key|154920|text|扩展了@Rodrigo的答案版本，我添加了这个脚本，现在我可以从配置中设置爬行器名称，而不是在字符串中更改。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|154921|import+sys
from+scrapy+import+cmdline

cmdline.execute(f"scrapy+crawl+{sys.argv[1]}".split())|code-block|syntax|javascript|154922|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Extending @Rodrigo's version of the answer I added this script and now I can set spider name from configuration instead of changing in the string.

<pre><code>import sys
from scrapy import cmdline

cmdline.execute(f"scrapy crawl {sys.argv[1]}".split())
</code></pre>

I am working on Scrapy 0.20 with Python 2.7. I found PyCharm has a good Python debugger. I want to test my Scrapy spiders using it. Anyone knows how to do that please?
<h3> What I have tried</h3>
Actually I tried to run the spider as a script. As a result, I built that script. Then, I tried to add my Scrapy project to PyCharm as a model like this:
<pre><code>File-&gt;Setting-&gt;Project structure-&gt;Add content root.
</code></pre>
But I don't know what else I have to do

How to use PyCharm to debug Scrapy projects

我正在使用Python 2.7开发Scrapy 0.20。我发现PyCharm有一个很好的Python调试器。我想用它来测试我的抓取蜘蛛。有谁知道怎么做吗？我尝试过的东西实际上，我尝试将爬行器作为脚本运行。因此，我构建了该脚本。然后，我尝试将我的Scrapy项目作为模型添加到PyCharm中，如下所示：File->Setting->Project structure->Add content ro

问如何使用PyCharm调试Scrapy项目
EN

回答 8

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何使用PyCharm调试Scrapy项目EN

回答 8

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何使用PyCharm调试Scrapy项目
EN