blocks|key|288083|text|Scrapy是一个用于抓取网站的python库。它相当小(与Nutch相比)，专为有限的站点爬行而设计。它有一种Django类型的MVC风格，我发现它很容易定制。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|288084|entityMap|0|LINK|mutability|MUTABLE|url|http://scrapy.org/^0|0|6|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

<a href="http://scrapy.org/" rel="nofollow noreferrer">Scrapy</a> is a python library that crawls web sites. It is fairly small (compared to Nutch) and designed for limited site crawls. It has a Django type MVC style that I found pretty easy to customize.

blocks|key|3222763|text|对于爬行部分，我真的很喜欢anemone和crawler4j。它们都允许您为链接选择和页面处理添加自定义逻辑。对于您决定保留的每个页面，都可以轻松地添加对Solr的调用。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|3222764|entityMap|0|LINK|mutability|MUTABLE|url|http://anemone.rubyforge.org/|1|http://code.google.com/p/crawler4j/^0|D|7|0|L|9|1|0^^$0|@$1|2|3|4|5|6|7|N|8|@]|9|@$A|O|B|P|1|Q]|$A|R|B|S|1|T]]|C|$]]|$1|D|3|-4|5|6|7|U|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]|L|$5|G|H|I|C|$J|M]]]]

For the crawling part, I really like <a href="http://anemone.rubyforge.org/" rel="nofollow noreferrer">anemone</a> and <a href="http://code.google.com/p/crawler4j/" rel="nofollow noreferrer">crawler4j</a>. They both allow you to add your custom logic for links selection and page handling. For each page that you decide to keep, you can easily add the call to Solr.

blocks|key|288086|text|在C#中，但要简单得多，您可以直接与作者交流。(我)|type|unstyled|depth|inlineStyleRanges|entityRanges|data|288087|我曾经使用Nutch，你是对的；使用它是一件很难的事情。|288088|http://arachnode.net|offset|length|288089|entityMap|0|LINK|mutability|MUTABLE|url|http://arachnode.net/^0|0|0|0|K|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Q|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|R|8|@]|9|@$F|S|G|T|1|U]]|A|$]]|$1|H|3|-4|5|6|7|V|8|@]|9|@]|A|$]]]|I|$J|$5|K|L|M|A|$N|O]]]]

In, C#, but a lot simpler and you can communicate directly with the author. (me)

I used to use Nutch and you are correct; it is a bear to work with.

<a href="http://arachnode.net" rel="nofollow">http://arachnode.net</a>

blocks|key|289968|text|这取决于有多少网站和你认为爬行的URL。Apache+Nutch在Apache+(依赖于Apache+Hadoop)上存储页面文档，它很可靠，但是很难设置和管理。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|289969|由于爬虫只是一个页面抓取(如CURL)和检索链接列表，以馈送您的CURL数据库，我相信你可以自己编写一个爬虫(特别是如果你有几个网站)，使用一个简单的MySQL数据库(可能是像RabbitMQ这样的队列软件来安排爬行作业)。|289970|另一方面，爬虫可能会更复杂，你可能想从HTML文档中移除头部，只保留页面的真正“内容”，等等……|289971|此外，Nutch可以使用PageRank算法对页面进行排序，您可以使用Apache+Spark来做同样的事情(效率更高，因为Spark可以在内存中缓存数据)。|289972|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|J|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|K|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|L|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|M|8|@]|9|@]|A|$]]|$1|H|3|-4|5|6|7|N|8|@]|9|@]|A|$]]]|I|$]]

It depends on how many web sites and so URLs you think crawl. Apache Nutch stores page documents on Apache HBase (which relies on Apache Hadoop), it's solid but very hard to setup and administrate.

Since a crawler is just a page fetch (like a CURL) and retrieve list of links to feed your URLs data base, I am sure you can write a crawler on your own (especially if you have a few web sites), use a simple MySQL database (maybe a queue software like RabbitMQ to schedule the crawl jobs).

On other side, a crawler could be more sophisticated, you could want to remove from your HTML document the HEAD part, and keep only the real "content" of the page etc...

Also, Nutch can rank your pages, with a PageRank algo., you could use Apache Spark to do the same thing (more efficiently because Spark can cache data in memory).

blocks|key|289974|text|我相信nutch是你的应用程序的最佳选择，但如果你想，有一个简单的工具：Heritrix。除此之外，我还推荐使用js作为前端语言，因为solr返回的json很容易被js处理。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|289975|entityMap|0|LINK|mutability|MUTABLE|url|https://webarchive.jira.com/wiki/display/Heritrix/Heritrix^0|10|8|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

I do believe the nutch is the best choice for you application, but if you want, there is a simple tool: <a href="https://webarchive.jira.com/wiki/display/Heritrix/Heritrix" rel="nofollow">Heritrix</a>.
Besides that, I recommand js for the front-end language, because solr returns json which is easily handled by js.

I'm trying to build a specialised search engine web site that indexes a limited number of web sites. The solution I came up with is:

<ul>
<li>using Nutch as the web crawler,</li>
<li>using Solr as the search engine,</li>
<li>the front-end and the site logic is coded with Wicket.</li>
</ul>

The problem is that I find Nutch quite complex and it's a big piece of software to customise, despite the fact that a detailed documentation (books, recent tutorials.. etc) does just not exist.

Questions now:

<ol>
<li>Any constructive criticism about the hole idea of the site?</li>
<li>Is there a good yet simple alternative to Nutch (as the crawling part of the site)?</li>
</ol>

Thanks

An alternative web crawler to Nutch

我正在尝试建立一个专门的搜索引擎网站，索引有限数量的网站。我想出的解决方案是：使用Nutch作为网络爬虫，使用Solr作为搜索引擎，作为前端，站点逻辑使用Wicket编码。问题是我发现Nutch相当复杂，而且它是一个需要定制的大软件，尽管有一个详细的文档(书籍，最近的教程..等)并不存在。现在提出问题：对这个网站的漏洞...

问Nutch的替代网络爬虫
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Nutch的替代网络爬虫EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Nutch的替代网络爬虫
EN