blocks|key|1756188|text|我还没有用Ruby语言做过任何多线程的事情，只用Java语言做过，但是看起来很简单：http://www.tutorialspoint.com/ruby/ruby_multithreading.htm|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|1756189|根据您所描述的，您不需要任何队列和工人(嗯，我相信您也可以这样做，但我怀疑您不会获得太多好处)。只需在几个线程之间划分您的urls，让每个线程完成每个块并使用结果更新数据库。例如，创建100个线程，并为每个线程提供1000个数据库行的范围以供处理。|1756190|如果您更愿意处理进程而不是线程，您甚至可以创建100个单独的进程，并将行作为参数提供给它们。|1756191|要获取URL状态，我认为您需要执行一个HTTP+HEAD请求，我猜它在ruby中是http://apidock.com/ruby/Net/HTTP/request_head。|1756192|entityMap|0|LINK|mutability|MUTABLE|url|http://www.tutorialspoint.com/ruby/ruby_multithreading.htm|1|http://apidock.com/ruby/Net/HTTP/request_head^0|16|1M|0|0|0|0|15|19|1|0^^$0|@$1|2|3|4|5|6|7|T|8|@]|9|@$A|U|B|V|1|W]]|C|$]]|$1|D|3|E|5|6|7|X|8|@]|9|@]|C|$]]|$1|F|3|G|5|6|7|Y|8|@]|9|@]|C|$]]|$1|H|3|I|5|6|7|Z|8|@]|9|@$A|10|B|11|1|12]]|C|$]]|$1|J|3|-4|5|6|7|13|8|@]|9|@]|C|$]]]|K|$L|$5|M|N|O|C|$P|Q]]|R|$5|M|N|O|C|$P|S]]]]

I haven't done anything multithreaded in Ruby, only in Java, but it seems pretty straightforward: <a href="http://www.tutorialspoint.com/ruby/ruby_multithreading.htm" rel="nofollow">http://www.tutorialspoint.com/ruby/ruby_multithreading.htm</a>

From what you described, you don't need any queue and workers (well, I'm sure you can do it that way too, but I doubt you'll get much benefit). Just partition your urls between several threads, and let each thread do each chunk and update the database with the results. E.g., create 100 threads, and give each thread a range of 1000 database rows to process.

You could even just create 100 separate processes and give them rows as arguments, if you'd rather deal with processes than threads.

To get the URL status, I think you do an HTTP HEAD request, which I guess is <a href="http://apidock.com/ruby/Net/HTTP/request_head" rel="nofollow">http://apidock.com/ruby/Net/HTTP/request_head</a> in ruby.

blocks|key|4474786|text|看看功能非常强大的Typhoeus+and+Hydra组合。这两种方法使得并发处理多个URL变得非常容易。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|4474787|"Times“示例应该能让您快速上手并运行。在on_complete块中，将您的代码写入到DB中。您可以使用线程来构建和维护处于健康状态的队列请求，或者将队列排成一定数量，让它们都运行到完成，然后循环到另一个组。由你决定。|style|CODE|4474788|Paul+Dix，最初的作者，在他的博客上talked+about+his+design+goals。|4474789|这是我编写的一些示例代码，用于下载存档的邮件列表，以便进行本地搜索。我故意删除了URL，以防止人们开始运行代码时网站受到DOS攻击：|4474790|#!/usr/bin/env+ruby

require+'nokogiri'
require+'addressable/uri'
require+'typhoeus'

BASE_URL+=+''

url+=+Addressable::URI.parse(BASE_URL)
resp+=+Typhoeus::Request.get(url.to_s)
doc+=+Nokogiri::HTML(resp.body)

hydra+=+Typhoeus::Hydra.new(:max_concurrency+=>+10)
doc.css('a').map{+%7Cn%7C+n['href']+}.select{+%7Chref%7C+href[/\.gz$/]+}.each+do+%7Cgzip%7C
++gzip_url+=+url.join(gzip)
++request+=+Typhoeus::Request.new(gzip_url.to_s)

++request.on_complete+do+%7Cresp%7C
++++gzip_filename+=+resp.request.url.split('/').last
++++puts+"writing+#{gzip_filename}"
++++File.open("gz/#{gzip_filename}",+'w')+do+%7Cfo%7C
++++++fo.write+resp.body
++++end++
++end
++puts+"queuing+#{+gzip+}"
++hydra.queue(request)
end

hydra.run|code-block|syntax|javascript|4474791|在我用了几年的DSL上运行代码，在不到20秒的时间里，通过无线方式传输到MacBook上，获得了76个文件，总大小为11MB。如果你只做HEAD请求，你的吞吐量会更好。你会想要弄乱并发设置，因为有更多的并发会话只会减慢你的速度，并且不必要地使用资源。|4474792|我给它打了8分(满分10分)；它的节拍很棒，我可以跟着它跳舞。|4474793|4474794|编辑：|4474795|在检查删除URL时，您可以使用HEAD请求，或对If-Modified-Since使用GET。它们可以为您提供响应，您可以使用这些响应来确定URL的新鲜度。|4474796|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/dbalatero/typhoeus|1|https://github.com/dbalatero/typhoeus/blob/master/examples/times.rb|2|http://www.pauldix.net/2009/05/breath-fire-over-http-in-ruby-with-typhoeus.html|3|http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.25^0|9|I|0|0|N|B|1|5|1|0|L|T|2|0|0|0|1W|4|0|0|0|0|O|H|O|H|3|0^^$0|@$1|2|3|4|5|6|7|1D|8|@]|9|@$A|1E|B|1F|1|1G]]|C|$]]|$1|D|3|E|5|6|7|1H|8|@$A|1I|B|1J|F|G]]|9|@$A|1K|B|1L|1|1M]]|C|$]]|$1|H|3|I|5|6|7|1N|8|@]|9|@$A|1O|B|1P|1|1Q]]|C|$]]|$1|J|3|K|5|6|7|1R|8|@]|9|@]|C|$]]|$1|L|3|M|5|N|7|1S|8|@]|9|@]|C|$O|P]]|$1|Q|3|R|5|6|7|1T|8|@$A|1U|B|1V|F|G]]|9|@]|C|$]]|$1|S|3|T|5|6|7|1W|8|@]|9|@]|C|$]]|$1|U|3|-4|5|6|7|1X|8|@]|9|@]|C|$]]|$1|V|3|W|5|6|7|1Y|8|@]|9|@]|C|$]]|$1|X|3|Y|5|6|7|1Z|8|@$A|20|B|21|F|G]]|9|@$A|22|B|23|1|24]]|C|$]]|$1|Z|3|-4|5|6|7|25|8|@]|9|@]|C|$]]]|10|$11|$5|12|13|14|C|$15|16]]|17|$5|12|13|14|C|$15|18]]|19|$5|12|13|14|C|$15|1A]]|1B|$5|12|13|14|C|$15|1C]]]]

Take a look at the very capable <a href="https://github.com/dbalatero/typhoeus" rel="nofollow">Typhoeus and Hydra</a> combo. The two make it very easy to concurrently process multiple URLs. 

The "<a href="https://github.com/dbalatero/typhoeus/blob/master/examples/times.rb" rel="nofollow">Times</a>" example should get you up and running quickly. In the <code>on_complete</code> block put your code to write your statuses to the DB. You could use a thread to build and maintain the queued requests at a healthy level, or queue a set number, let them all run to completion, then loop for another group. It's up to you.

Paul Dix, the original author, <a href="http://www.pauldix.net/2009/05/breath-fire-over-http-in-ruby-with-typhoeus.html" rel="nofollow">talked about his design goals</a> on his blog.

This is some sample code I wrote to download archived mail lists so I could do local searches. I deliberately removed the URL to keep from subjecting the site to DOS attacks if people start running the code:

<pre><code>#!/usr/bin/env ruby

require 'nokogiri'
require 'addressable/uri'
require 'typhoeus'

BASE_URL = ''

url = Addressable::URI.parse(BASE_URL)
resp = Typhoeus::Request.get(url.to_s)
doc = Nokogiri::HTML(resp.body)

hydra = Typhoeus::Hydra.new(:max_concurrency =&gt; 10)
doc.css('a').map{ |n| n['href'] }.select{ |href| href[/\.gz$/] }.each do |gzip|
 gzip_url = url.join(gzip)
 request = Typhoeus::Request.new(gzip_url.to_s)

 request.on_complete do |resp|
 gzip_filename = resp.request.url.split('/').last
 puts "writing #{gzip_filename}"
 File.open("gz/#{gzip_filename}", 'w') do |fo|
 fo.write resp.body
 end 
 end
 puts "queuing #{ gzip }"
 hydra.queue(request)
end

hydra.run
</code></pre>

Running the code on my several-year-old MacBook Pro pulled in 76 files totaling 11MB in just under 20 seconds, over wireless to DSL. If you're only doing <code>HEAD</code> requests your throughput will be better. You'll want to mess with the concurrency setting because there is a point where having more concurrent sessions only slow you down and needlessly use resources.

I give it a 8 out of 10; It's got a great beat and I can dance to it.

<hr>

EDIT:

When checking the remove URLs you can use a HEAD request, or a GET with the <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.25" rel="nofollow"><code>If-Modified-Since</code></a>. They can give you responses you can use to determine the freshness of your URLs.

blocks|key|4474798|text|work_queue+gem是在应用程序中异步并发执行任务的最简单方法。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|4474799|wq+=+WorkQueue.new+10

urls.each+do+%7Curl%7C
++wq.enqueue_b+do
++++response+=+Net::HTTP.get_response(uri)
++++puts+response.code
++end
end

wq.join|code-block|syntax|javascript|4474800|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/fmmfonseca/work_queue^0|0|A|0|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@$A|R|B|S|1|T]]|C|$]]|$1|D|3|E|5|F|7|U|8|@]|9|@]|C|$G|H]]|$1|I|3|-4|5|6|7|V|8|@]|9|@]|C|$]]]|J|$K|$5|L|M|N|C|$O|P]]]]

The <a href="https://github.com/fmmfonseca/work_queue" rel="nofollow">work_queue</a> gem is the easiest way to perform tasks asynchronously and concurrently in your application.

<pre><code>wq = WorkQueue.new 10

urls.each do |url|
 wq.enqueue_b do
 response = Net::HTTP.get_response(uri)
 puts response.code
 end
end

wq.join
</code></pre>

Here's what I'm trying to accomplish. Let's say I have 100,000 urls stored in a database and I want to check each of these for http status and store that status. I want to be able to do this concurrently in a fairly small amount of time.

I was wondering what the best way(s) to do this would be. I thought about using some sort of queue with workers/consumers or some sort of evented model, but I don't really have enough experience to know what would work best in this scenario.

Ideas?

Best way to concurrently check urls (for status i.e. 200,301,404) for multiple urls in database

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

这是我想要实现的目标。假设我有100,000个urls存储在数据库中，我想检查每个urls的http状态并存储该状态。我希望能够在相当短的时间内同时做到这一点。我想知道做这件事最好的方法是什么。我考虑过对工作人员/消费者使用某种类型的队列或某种事件模型，但我真的没有足够的经验来了解在这种情况下最好的工作方式。想法？

问并发检查数据库中多个urls的urls (状态为200,301,404)的最佳方式
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问并发检查数据库中多个urls的urls (状态为200,301,404)的最佳方式EN