给出了从代码中运行scrapy爬虫的许多方法:
import scrapy
from scrapy.crawler import CrawlerProcess
class MySpider(scrapy.Spider):
# Your spider definition
...
process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})
process.crawl(MySpider)
process.
我试图在一个vps上运行一个爬虫(使用scrapyjs,它使用python-gtk2)。在运行爬行器时,我得到了错误
/root/myporj/venv/local/lib/python2.7/dist-packages/gtk-2.0/gtk/__init__.py:57: GtkWarning: could not open display
如何在无头设置中运行此程序?