blocks|key|3230384|text|有很多不同的技术..。“最糟糕的”就是你所描述的那个。(基于时间的轮询)|type|unstyled|depth|inlineStyleRanges|entityRanges|data|3230385|您需要考虑的第一件事是，它们可能并不都在服务器端进行解析。例如，我知道Netvibes在客户端进行解析(但在服务器上缓存内容)，因此为他们节省了大量资源。这样，它们只会在用户请求时才轮询提要，因此不需要运行某种时间循环。|3230386|不幸的是，基于时间的轮询仍然是最常见的解决方案。有很多技术可以确定何时是进行民意调查的最佳时机。根据过去的更新频率，基于暂停的用户数量...这些人也可以使用旧的XML-RPC+ping服务器。|3230387|最有效的技术是使用PubSubHubbub，这是一个开放的协议，谷歌阅读器，Netvibes和其他几千个应用程序(如Digg.com，Twitterfeed，Friendfeed...)。它是一个开放的协议，允许提要发布者直接将提要的内容推送到订阅应用程序。它非常高效，但需要发布者来实现它。一个偶然的机会，所有的大博客平台+(Tumblr，Posterous，Wordpress，Blogger，SixApart...等)已经实现了它。其他feed发布应用程序(如feedburner、Gowalla等)也实现了它。如果你确实发布了提要，我会鼓励你加入这个群体，如果你打算消费一些提要，请也实现susbcriber这一端。|offset|length|style|BOLD|3230388|最后一个解决方案是使用第三方应用程序进行数据收集(使用上面的所有技术)，并在这些提要实际有新内容时对您执行ping操作。我创建了一个：Superfeedr，我相信我们在这方面做得很好。我们还对内容进行了标准化，并做了一些其他事情来帮助您以最简单、最便宜的方式使用提要数据(轮询可能非常昂贵)。此外，我们使用完全相同的PubSubHubbub协议来推送来自任何提要的内容，这使得我们的用户除了订阅可用的集线器之外，还可以非常简单地使用我们的服务。|3230389|此外，我应该补充说，我能够快速回复您的问题，因为我使用了一个应用程序，该应用程序会向我推送标记为RSS的问题的提要内容:)|3230390|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/pubsubhubbub|1|http://superfeedr.com/^0|0|0|0|4A|8|9|C|0|0|1V|A|1|0|0^^$0|@$1|2|3|4|5|6|7|Z|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|10|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|11|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|12|8|@$H|13|I|14|J|K]]|9|@$H|15|I|16|1|17]]|A|$]]|$1|L|3|M|5|6|7|18|8|@]|9|@$H|19|I|1A|1|1B]]|A|$]]|$1|N|3|O|5|6|7|1C|8|@]|9|@]|A|$]]|$1|P|3|-4|5|6|7|1D|8|@]|9|@]|A|$]]]|Q|$R|$5|S|T|U|A|$V|W]]|X|$5|S|T|U|A|$V|Y]]]]

There is a lot of different techniques... the "worst" one being the one that you describe. (time based polling).

The first thing you need to consider is that they may not all do the parsing on the server side. For example, I know that Netvibes was doing the parsing on the client side (but cached the content on the server), so it saved them a lot of resources. This way they would poll feeds only when users asked from them, so there is no need for them to run some kind of time loop.

Time based polling is still, unfortunately the most frequent solution. There are a lot of techniques to determine when is the best time to do a poll. Based on the frequency of past updates, based on the number of users who susbcribed... etc. The old XML-RPC ping servers can also be used by these guys.

The most efficient technique is to use <a href="https://github.com/pubsubhubbub" rel="noreferrer">PubSubHubbub</a>, which is a open protocol used by Google Reader, Netvibes and a few thousand other apps (like Digg.com, Twitterfeed, Friendfeed...). It's an open protocol that allows the feed publisher to directly push the content of the feed to subscribing applications. It's very efficient, but requires the publisher to implement it. By chance, all the big blogging platforms (Tumblr, Posterous, Wordpress, Blogger, SixApart... etc) have implemented it. Other feed publishing apps (like feedburner, Gowalla, ...) also implemented it. If you do publish feeds, I would encourage joining this crowd, and if you plan on consuming some, please, implement the susbcriber side as well.

The last solution is to use a 3rd party application do this data gathering (using all the techniques above) and ping you when these feeds actually have new content. I created one : <a href="http://superfeedr.com" rel="noreferrer">Superfeedr</a> and I believe we do a good job with this. We also normalize the content and do a few other things to help you consume feed data in the simplest and cheap way (polling can be crazy expensive). Also, we use the exact same PubSubHubbub protocol to push content from any feed, which makes it very simple for our users to use our service in addition to subscribing to available hubs.

Also, I should add that I was able to reply quickly to your question, because I use an app that pushes me the content of the feed for questions tagged RSS :)

I wonder how web applications like Google Reader, Blogline, techronati works, and what technics they follow to parse millions of RSS feeds using cron job at one time?

How large RSS reader works (netvibes, Google reader...)

服务器

我想知道像Google Reader，Blogline，techronati这样的web应用程序是如何工作的，以及它们遵循什么技术来使用cron job一次解析数百万个RSS提要？

问大型RSS阅读器的工作原理(netvibes、Google reader...)
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问大型RSS阅读器的工作原理(netvibes、Google reader...)EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问大型RSS阅读器的工作原理(netvibes、Google reader...)
EN