blocks|key|688004|text|你最好的选择是使用Mechanize.It可以跟踪链接，提交表单，任何你需要的东西，web客户端。顺便说一下，不要使用正则表达式来解析HTML。使用HTML解析器。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|688005|entityMap|0|LINK|mutability|MUTABLE|url|http://docs.seattlerb.org/mechanize/GUIDE_rdoc.html^0|9|9|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

Your best bet would be to use <a href="http://docs.seattlerb.org/mechanize/GUIDE_rdoc.html" rel="nofollow">Mechanize</a>.It can follow links, submit forms, anything you will need, web client-wise. By the way, don't use regexes to parse HTML. Use an HTML parser.

blocks|key|688039|text|我知道答案已经被接受了，但是Hpricot在解析超文本标记语言方面也非常流行。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|688040|您所要做的就是查看页面的html源代码，并尝试找到与所需元素匹配的XPath或CSS表达式，然后使用如下内容：|688041|doc.search("//p[@class='posted']")|code-block|syntax|javascript|688042|entityMap|0|LINK|mutability|MUTABLE|url|http://rubygems.org/gems/hpricot^0|E|7|0|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@]|9|@$A|T|B|U|1|V]]|C|$]]|$1|D|3|E|5|6|7|W|8|@]|9|@]|C|$]]|$1|F|3|G|5|H|7|X|8|@]|9|@]|C|$I|J]]|$1|K|3|-4|5|6|7|Y|8|@]|9|@]|C|$]]]|L|$M|$5|N|O|P|C|$Q|R]]]]

I know the answer has been accepted, but <a href="http://rubygems.org/gems/hpricot" rel="noreferrer">Hpricot</a> is also very popular for parsing HTML.

All you have to do is take a look at the html source of the pages and try to find a XPath or CSS expression that matches the desired elements, then use something like:

<pre><code>doc.search("//p[@class='posted']")
</code></pre>

blocks|key|688077|text|如果你想要更高层次的东西，可以试试wombat，这是我在Mechanize和Nokogiri之上构建的宝石。它能够使用非常简单和高级的DSL来解析页面和跟踪链接。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|688078|entityMap|0|LINK|mutability|MUTABLE|url|http://github.com/felipecsl/wombat^0|H|6|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

If you want something more high level, try <a href="http://github.com/felipecsl/wombat">wombat</a>, which is this gem I built on top of Mechanize and Nokogiri. It is able to parse pages and follow links using a really simple and high level DSL.

blocks|key|688121|text|机械化是很棒的。如果你想学习一些新的东西，你可以看看Scrubyt：https://github.com/scrubber/scrubyt。看起来像是机械化%2B+Hpricot。我从来没有用过它，但它看起来很有趣。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|688122|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/scrubber/scrubyt^0|Y|Z|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

Mechanize is awesome. If you're looking to learn something new though, you could take a look at Scrubyt: <a href="https://github.com/scrubber/scrubyt" rel="nofollow">https://github.com/scrubber/scrubyt</a>. It looks like Mechanize + Hpricot. I've never used it, but it seems interesting.

blocks|key|688189|text|Nokogiri很棒，但我发现它的输出很混乱。我编写了一个ruby+gem来轻松地创建基于HTML的类：https://github.com/jassa/hyper_api|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|688190|HyperAPI+gem使用Nokogiri解析带有CSS选择器的超文本标记语言。|688191|例如。|688192|Post+=+HyperAPI.new_class+do
++string+title:+'div#title'
++string+body:+'div#body'
++string+author:+'#details+.author'
++integer+comments_count:+'#extra+.comment'+do
++++size
++end
end
#+=>+Post

post+=+Post.new(html_string)
#+=>+#<Post+title:+'Hi+there!',+body:+'This+blog+post+will+talk+about...',+author:+'Bob',+comments_count:+74>|code-block|syntax|javascript|688193|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/jassa/hyper_api^0|1G|Y|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|U|8|@]|9|@$A|V|B|W|1|X]]|C|$]]|$1|D|3|E|5|6|7|Y|8|@]|9|@]|C|$]]|$1|F|3|G|5|6|7|Z|8|@]|9|@]|C|$]]|$1|H|3|I|5|J|7|10|8|@]|9|@]|C|$K|L]]|$1|M|3|-4|5|6|7|11|8|@]|9|@]|C|$]]]|N|$O|$5|P|Q|R|C|$S|T]]]]

Nokogiri is great, but I find the output messy to work with. I wrote a ruby gem to easily create classes off HTML: <a href="https://github.com/jassa/hyper_api" rel="nofollow">https://github.com/jassa/hyper_api</a>

The HyperAPI gem uses Nokogiri to parse HTML with CSS selectors.

E.g.

<pre><code>Post = HyperAPI.new_class do
 string title: 'div#title'
 string body: 'div#body'
 string author: '#details .author'
 integer comments_count: '#extra .comment' do
 size
 end
end
# =&gt; Post

post = Post.new(html_string)
# =&gt; #&lt;Post title: 'Hi there!', body: 'This blog post will talk about...', author: 'Bob', comments_count: 74&gt;
</code></pre>

I would like to crawl a popular site (say Quora) that doesn't have an API and get some specific information and dump it into a file - say either a csv, .txt, or .html formatted nicely :)

E.g. return only a list of all the 'Bios' of the Users of Quora that have, listed in their publicly available information, the occupation 'UX designer'.

How would I do that in Ruby ?

I have a moderate enough level of understanding of how Ruby &amp; Rails work. I just completed a Rails app - mainly all written by myself. But I am no guru by any stretch of the imagination.

I understand RegExs, etc.

How do I write a web scraper in Ruby?

我想抓取一个没有API的流行站点(比如Quora)，获取一些特定的信息，并将其转储到一个文件中--比如csv、.txt或.html，这些文件的格式很好:)例如，只返回Quora用户的所有‘Bio’的列表，这些用户在他们的公开信息中列出了职业'UX designer‘。我该如何在Ruby中做到这一点？我对Ruby和Rai...

问我如何用Ruby写一个网络爬行器？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问我如何用Ruby写一个网络爬行器？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问我如何用Ruby写一个网络爬行器？
EN