blocks|key|1003699|text|C%2B%2B+-如果你知道你在做什么。您不需要web服务器和web应用程序，因为web爬虫毕竟只是一个客户端。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1003700|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

C++ - if you know what you're doing. You will not need a web server and a web application, because a web crawler is just a client, after all.

blocks|key|2968769|text|大多数语言可能都很适合，关键的组件是|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2968770|用HTML库处理互联网中的Protcols|2968771|Libraries处理常规expressions|unordered-list-item|2968772|
|2968773|Libraries来解析|2968774|内容|ordered-list-item|2968775|2968776|2968777|如今，大多数语言都有对上述功能提供良好支持的库，当然，您需要一些方法来持久化结果，这些结果可能是某种数据库。|2968778|比起语言，更重要的是理解你需要处理的所有概念。这里有一些Python示例，可能会帮助您入门。|2968779|http://www.example-code.com/python/pythonspider.asp|offset|length|2968780|entityMap|0|LINK|mutability|MUTABLE|url^0|0|0|0|0|0|0|0|0|0|0|0|1F|0|0^^$0|@$1|2|3|4|5|6|7|14|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|15|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|16|8|@]|9|@]|A|$]]|$1|G|3|H|5|6|7|17|8|@]|9|@]|A|$]]|$1|I|3|J|5|F|7|18|8|@]|9|@]|A|$]]|$1|K|3|L|5|M|7|19|8|@]|9|@]|A|$]]|$1|N|3|-4|5|6|7|1A|8|@]|9|@]|A|$]]|$1|O|3|-4|5|6|7|1B|8|@]|9|@]|A|$]]|$1|P|3|Q|5|6|7|1C|8|@]|9|@]|A|$]]|$1|R|3|S|5|6|7|1D|8|@]|9|@]|A|$]]|$1|T|3|U|5|6|7|1E|8|@]|9|@$V|1F|W|1G|1|1H]]|A|$]]|$1|X|3|-4|5|6|7|1I|8|@]|9|@]|A|$]]]|Y|$Z|$5|10|11|12|A|$13|U]]]]

Most languages would probably be a reasonable fit, the critical components are 

<ol>
<li>Libraries to deal with the Internet Protcols</li>
<li>Libraries to deal with regular expressions</li>
<li>Libraries to parse HTML content</li>
</ol>

Today most languages have libraries with good support for the above, of course you will need some way to persist the results that might be a database of some sorts. 

The more important thing rather than the language is understanding all concepts you need to deal with. Here are some Python examples that might help get you started.

<a href="http://www.example-code.com/python/pythonspider.asp" rel="noreferrer">http://www.example-code.com/python/pythonspider.asp</a>

blocks|key|1001416|text|任何一种你可以很容易使用的语言，有一个很好的网络库，并且支持解析你想要抓取的格式。这些真的是唯一的条件。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1001417|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

Any language you can easily use with a good network library and support for parsing the formats you want to crawl. Those are really the only qualifications.

blocks|key|1746522|text|您可以考虑使用python和PyGtkMozEmbed的组合，或者PyWebKitGtk和javascript的组合来创建爬行器。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1746523|爬行可以在页面和所有其他脚本加载后在javascript中完成。|1746524|您将拥有支持javascript的为数不多的web爬虫之一，并且可能会发现一些其他人看不到的隐藏内容:)|1746525|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|H|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|I|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|J|8|@]|9|@]|A|$]]|$1|F|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|G|$]]

You could consider using a combination of python and PyGtkMozEmbed or PyWebKitGtk plus javascript to create your spider.

The spidering could be done in javascript after the page and all other scripts have loaded.

You'd have one of the few web spiders that supports javascript, and might pick up some hidden stuff the others don't see :)

blocks|key|1003815|text|既然可以复制http://code.activestate.com/recipes/576551-simple-web-crawler/，为什么还要自己写呢|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1003816|您可能需要在这里和那里修复一些东西，比如使用htmlentities而不是用&替换&|1003817|entityMap|0|LINK|mutability|MUTABLE|url|http://code.activestate.com/recipes/576551-simple-web-crawler/^0|6|1Q|0|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@]|9|@$A|O|B|P|1|Q]]|C|$]]|$1|D|3|E|5|6|7|R|8|@]|9|@]|C|$]]|$1|F|3|-4|5|6|7|S|8|@]|9|@]|C|$]]]|G|$H|$5|I|J|K|C|$L|M]]]]

why write your own when you can copy <a href="http://code.activestate.com/recipes/576551-simple-web-crawler/" rel="nofollow noreferrer">http://code.activestate.com/recipes/576551-simple-web-crawler/</a>

you might need to fix a few things here and there, like use htmlentities instead of replacing &amp; with &amp;

blocks|key|2968921|text|当涉及到编写多核/线程爬虫时，C是所有的GoD，但它也有它自己的复杂性。在C语言之后，一些人转向了JAVA+(由于广泛的探索和使用)，而另一些人则转向了Python。如果你有很好的架构，我可以向你保证这三种语言真的不会限制你的效率。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2968922|这段python代码是一个C+Curl实现，在一台优秀的服务器上，它可以在300秒内抓取大约10,000个页面|2968923|#!+/usr/bin/env+python
#+-*-+coding:+iso-8859-1+-*-
#+vi:ts=4:et
#+$Id:+retriever-multi.py,v+1.29+2005/07/28+11:04:13+mfx+Exp+$

#
#+Usage:+python+retriever-multi.py+<file+with+URLs+to+fetch>+[<#+of
#++++++++++concurrent+connections>]
#

import+sys
import+pycurl

#+We+should+ignore+SIGPIPE+when+using+pycurl.NOSIGNAL+-+see
#+the+libcurl+tutorial+for+more+info.
try:
++++import+signal
++++from+signal+import+SIGPIPE,+SIG_IGN
++++signal.signal(signal.SIGPIPE,+signal.SIG_IGN)
except+ImportError:
++++pass


#+Get+args
num_conn+=+10
try:
++++if+sys.argv[1]+==+"-":
++++++++urls+=+sys.stdin.readlines()
++++else:
++++++++urls+=+open(sys.argv[1]).readlines()
++++if+len(sys.argv)+>=+3:
++++++++num_conn+=+int(sys.argv[2])
except:
++++print+"Usage:+%25s+<file+with+URLs+to+fetch>+[<#+of+concurrent+connections>]"+%25+sys.argv[0]
++++raise+SystemExit


#+Make+a+queue+with+(url,+filename)+tuples
queue+=+[]
for+url+in+urls:
++++url+=+url.strip()
++++if+not+url+or+url[0]+==+"#":
++++++++continue
++++filename+=+"doc_%2503d.dat"+%25+(len(queue)+%2B+1)
++++queue.append((url,+filename))


#+Check+args
assert+queue,+"no+URLs+given"
num_urls+=+len(queue)
num_conn+=+min(num_conn,+num_urls)
assert+1+<=+num_conn+<=+10000,+"invalid+number+of+concurrent+connections"
print+"PycURL+%25s+(compiled+against+0x%25x)"+%25+(pycurl.version,+pycurl.COMPILE_LIBCURL_VERSION_NUM)
print+"-----+Getting",+num_urls,+"URLs+using",+num_conn,+"connections+-----"


#+Pre-allocate+a+list+of+curl+objects
m+=+pycurl.CurlMulti()
m.handles+=+[]
for+i+in+range(num_conn):
++++c+=+pycurl.Curl()
++++c.fp+=+None
++++c.setopt(pycurl.FOLLOWLOCATION,+1)
++++c.setopt(pycurl.MAXREDIRS,+5)
++++c.setopt(pycurl.CONNECTTIMEOUT,+30)
++++c.setopt(pycurl.TIMEOUT,+300)
++++c.setopt(pycurl.NOSIGNAL,+1)
++++m.handles.append(c)


#+Main+loop
freelist+=+m.handles[:]
num_processed+=+0
while+num_processed+<+num_urls:
++++#+If+there+is+an+url+to+process+and+a+free+curl+object,+add+to+multi+stack
++++while+queue+and+freelist:
++++++++url,+filename+=+queue.pop(0)
++++++++c+=+freelist.pop()
++++++++c.fp+=+open(filename,+"wb")
++++++++c.setopt(pycurl.URL,+url)
++++++++c.setopt(pycurl.WRITEDATA,+c.fp)
++++++++m.add_handle(c)
++++++++#+store+some+info
++++++++c.filename+=+filename
++++++++c.url+=+url
++++#+Run+the+internal+curl+state+machine+for+the+multi+stack
++++while+1:
++++++++ret,+num_handles+=+m.perform()
++++++++if+ret+!=+pycurl.E_CALL_MULTI_PERFORM:
++++++++++++break
++++#+Check+for+curl+objects+which+have+terminated,+and+add+them+to+the+freelist
++++while+1:
++++++++num_q,+ok_list,+err_list+=+m.info_read()
++++++++for+c+in+ok_list:
++++++++++++c.fp.close()
++++++++++++c.fp+=+None
++++++++++++m.remove_handle(c)
++++++++++++print+"Success:",+c.filename,+c.url,+c.getinfo(pycurl.EFFECTIVE_URL)
++++++++++++freelist.append(c)
++++++++for+c,+errno,+errmsg+in+err_list:
++++++++++++c.fp.close()
++++++++++++c.fp+=+None
++++++++++++m.remove_handle(c)
++++++++++++print+"Failed:+",+c.filename,+c.url,+errno,+errmsg
++++++++++++freelist.append(c)
++++++++num_processed+=+num_processed+%2B+len(ok_list)+%2B+len(err_list)
++++++++if+num_q+==+0:
++++++++++++break
++++#+Currently+no+more+I/O+is+pending,+could+do+something+in+the+meantime
++++#+(display+a+progress+bar,+etc.).
++++#+We+just+call+select()+to+sleep+until+some+more+data+is+available.
++++m.select(1.0)


#+Cleanup
for+c+in+m.handles:
++++if+c.fp+is+not+None:
++++++++c.fp.close()
++++++++c.fp+=+None
++++c.close()
m.close()|code-block|syntax|javascript|2968924|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|L|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|M|8|@]|9|@]|A|$G|H]]|$1|I|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|J|$]]

C is GoD of all when it comes to writing multicore/threaded crawlers but then it has its own complication. After C, some go for JAVA (due to wide exploration and usage) while other go to Python. If you have nice architecture, I can assure you these three language would really not limit your efficiency. 

This python code is a C Curl implementation and can crawl around 10,000 pages in 300 secs on a nice server

<pre><code>#! /usr/bin/env python
# -*- coding: iso-8859-1 -*-
# vi:ts=4:et
# $Id: retriever-multi.py,v 1.29 2005/07/28 11:04:13 mfx Exp $

#
# Usage: python retriever-multi.py &lt;file with URLs to fetch&gt; [&lt;# of
# concurrent connections&gt;]
#

import sys
import pycurl

# We should ignore SIGPIPE when using pycurl.NOSIGNAL - see
# the libcurl tutorial for more info.
try:
 import signal
 from signal import SIGPIPE, SIG_IGN
 signal.signal(signal.SIGPIPE, signal.SIG_IGN)
except ImportError:
 pass


# Get args
num_conn = 10
try:
 if sys.argv[1] == "-":
 urls = sys.stdin.readlines()
 else:
 urls = open(sys.argv[1]).readlines()
 if len(sys.argv) &gt;= 3:
 num_conn = int(sys.argv[2])
except:
 print "Usage: %s &lt;file with URLs to fetch&gt; [&lt;# of concurrent connections&gt;]" % sys.argv[0]
 raise SystemExit


# Make a queue with (url, filename) tuples
queue = []
for url in urls:
 url = url.strip()
 if not url or url[0] == "#":
 continue
 filename = "doc_%03d.dat" % (len(queue) + 1)
 queue.append((url, filename))


# Check args
assert queue, "no URLs given"
num_urls = len(queue)
num_conn = min(num_conn, num_urls)
assert 1 &lt;= num_conn &lt;= 10000, "invalid number of concurrent connections"
print "PycURL %s (compiled against 0x%x)" % (pycurl.version, pycurl.COMPILE_LIBCURL_VERSION_NUM)
print "----- Getting", num_urls, "URLs using", num_conn, "connections -----"


# Pre-allocate a list of curl objects
m = pycurl.CurlMulti()
m.handles = []
for i in range(num_conn):
 c = pycurl.Curl()
 c.fp = None
 c.setopt(pycurl.FOLLOWLOCATION, 1)
 c.setopt(pycurl.MAXREDIRS, 5)
 c.setopt(pycurl.CONNECTTIMEOUT, 30)
 c.setopt(pycurl.TIMEOUT, 300)
 c.setopt(pycurl.NOSIGNAL, 1)
 m.handles.append(c)


# Main loop
freelist = m.handles[:]
num_processed = 0
while num_processed &lt; num_urls:
 # If there is an url to process and a free curl object, add to multi stack
 while queue and freelist:
 url, filename = queue.pop(0)
 c = freelist.pop()
 c.fp = open(filename, "wb")
 c.setopt(pycurl.URL, url)
 c.setopt(pycurl.WRITEDATA, c.fp)
 m.add_handle(c)
 # store some info
 c.filename = filename
 c.url = url
 # Run the internal curl state machine for the multi stack
 while 1:
 ret, num_handles = m.perform()
 if ret != pycurl.E_CALL_MULTI_PERFORM:
 break
 # Check for curl objects which have terminated, and add them to the freelist
 while 1:
 num_q, ok_list, err_list = m.info_read()
 for c in ok_list:
 c.fp.close()
 c.fp = None
 m.remove_handle(c)
 print "Success:", c.filename, c.url, c.getinfo(pycurl.EFFECTIVE_URL)
 freelist.append(c)
 for c, errno, errmsg in err_list:
 c.fp.close()
 c.fp = None
 m.remove_handle(c)
 print "Failed: ", c.filename, c.url, errno, errmsg
 freelist.append(c)
 num_processed = num_processed + len(ok_list) + len(err_list)
 if num_q == 0:
 break
 # Currently no more I/O is pending, could do something in the meantime
 # (display a progress bar, etc.).
 # We just call select() to sleep until some more data is available.
 m.select(1.0)


# Cleanup
for c in m.handles:
 if c.fp is not None:
 c.fp.close()
 c.fp = None
 c.close()
m.close()
</code></pre>

blocks|key|1749910|text|对于这一点，C#和C%2B%2B可能是最好的两种语言，这只是你更了解哪个更快的问题(C#可能更容易)。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1749911|我不推荐Python、Javascript或PHP。与C系列语言相比，它们在文本处理方面通常会更慢。如果你想抓取任何重要的网页，你将需要所有你能得到的速度。|1749912|我以前用过C#和HtmlAgilityPack来做这件事，它工作得相当好，而且很容易上手。能够像使用XML一样使用许多相同的命令来处理HTML，这使它变得很好(我有在C#中使用XML的经验)。|1749913|您可能希望测试可用的C#+HTML解析库与C%2B%2B解析库的速度。我知道在我的应用程序中，我每秒浏览60-70个相当混乱的页面，并从每个页面中提取大量数据(但这是一个布局相当稳定的网站)。|1749914|编辑:我注意到你提到了访问数据库。C%2B%2B和C#都有适用于大多数常见数据库系统的库，从SQLite+(这对于一些站点上的快速爬虫来说非常好)到MySQL和MSSQL这样的中端引擎，再到更大的数据库引擎(我从未使用过这两种语言的Oracle或DB2，但这是可能的)。|1749915|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|M|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|N|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|O|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|P|8|@]|9|@]|A|$]]|$1|J|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|K|$]]

C# and C++ are probably the best two languages for this, it's just a matter of which you know better and which is faster (C# is probably easier).

I wouldn't recommend Python, Javascript, or PHP. They will usually be slower in text processing compared to a C-family language. If you're looking to crawl any significant chunk of the web, you'll need all the speed you can get.

I've used C# and the HtmlAgilityPack to do so before, it works relatively well and is pretty easy to pick up. The ability to use a lot of the same commands to work with HTML as you would XML makes it nice (I had experience working with XML in C#).

You might want to test the speed of available C# HTML parsing libraries vs C++ parsing libraries. I know in my app, I was running through 60-70 fairly messy pages a second and pulling a good bit of data out of each (but that was a site with a pretty constant layout).

Edit: I notice you mentioned accessing a database. Both C++ and C# have libraries to work with most common database systems, from SQLite (which would be great for a quick crawler on a few sites) to midrange engines like MySQL and MSSQL up to the bigger DB engines (I've never used Oracle or DB2 from either language, but it's possible).

I have substantial PHP experience, although I realize that PHP probably isn't the best language for a large-scale web crawler because a process can't run indefinitely. What languages do people suggest?

What languages are good for writing a web crawler?

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我有丰富的PHP经验，尽管我意识到PHP可能不是大型网络爬虫的最佳语言，因为进程不能无限运行。人们推荐的语言是什么？

问哪些语言适合编写网络爬虫？
EN

回答 7

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问哪些语言适合编写网络爬虫？EN

回答 7

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问哪些语言适合编写网络爬虫？
EN