blocks|key|4660571|text|您可以使用Apache+Kafka作为分布式的健壮队列，它可以处理大量数据，并使您能够将消息从一个端点传递到另一个端点。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4660572|风暴不是一个队列。它是一个具有分布式实时处理能力的系统，这意味着你可以并行地对实时数据执行所有类型的操作。|4660573|这些工具的常见流程(据我所知)如下：|4660574|实时系统-->+Kafka+-->暴风-->+NoSql+-->+BI(可选)|4660575|因此，您的实时应用程序可以处理大量数据，并将其发送到Kafka队列。Storm从kafka中提取数据，并应用一些所需的操作。此时，您通常希望从这些数据中获得一些好处，因此您可以将其发送到某个Nosql+db以进行额外的BI计算，或者您可以简单地从任何其他系统查询此NoSql。|4660576|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|M|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|N|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|O|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|P|8|@]|9|@]|A|$]]|$1|J|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|K|$]]

You use Apache Kafka as a distributed and robust queue that can handle high volume data and enables you to pass messages from one end-point to another.

Storm is not a queue. It is a system that has distributed real time processing abilities, meaning you can execute all kind of manipulations on real time data in parallel.

The common flow of these tools (as I know it) goes as follows:

real-time-system --> Kafka --> Storm --> NoSql --> BI(optional)

So you have your real time app handling high volume data, sends it to Kafka queue. Storm pulls the data from kafka and applies some required manipulation. At this point you usually like to get some benefits from this data, so you either send it to some Nosql db for additional BI calculations, or you could simply query this NoSql from any other system.

blocks|key|4660622|text|正如每个人向您解释的那样，Apache+Kafka:是连续消息队列|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4660623|Apache+Storm:持续处理工具|4660624|在这方面，Kafka将使用API从FB、Twitter等任何网站获取数据，并使用Apache+Storm处理这些数据，您可以将处理后的数据存储在任何您喜欢的数据库中。|4660625|https://github.com/miguno/kafka-storm-starter|offset|length|4660626|只要跟随它，你就会有一些想法|4660627|entityMap|0|LINK|mutability|MUTABLE|url^0|0|0|0|0|19|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|T|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|U|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|V|8|@]|9|@$H|W|I|X|1|Y]]|A|$]]|$1|J|3|K|5|6|7|Z|8|@]|9|@]|A|$]]|$1|L|3|-4|5|6|7|10|8|@]|9|@]|A|$]]]|M|$N|$5|O|P|Q|A|$R|G]]]]

As every one explain you that 
Apache Kafka: is continuous messaging queue

Apache Storm: is continuous processing tool 

here in this aspect Kafka will get the data from any website like FB,Twitter by using API's and that data is processed by using Apache Storm and you can store the processed data in either in any databases you like.

<a href="https://github.com/miguno/kafka-storm-starter" rel="nofollow noreferrer">https://github.com/miguno/kafka-storm-starter</a>

Just follow it you will get some idea

blocks|key|154176|text|我知道这是一个较老的线程，Apache+Kafka和Storm的比较在编写时是有效和正确的，但值得注意的是，Apache+Kafka在过去几年中发生了很大变化，自0.10版本(2016年4月)以来，Kafka包含了一个Kafka+Streams+API，它提供流处理功能，而不需要任何额外的软件，如Storm。Kafka还包括用于连接到各种数据源和接收器(目的地)的Connect+API。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|154177|公告博客-+https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/|offset|length|154178|最新的Apache文档-+https://kafka.apache.org/documentation/streams/|154179|在0.11Kafka中，流处理功能得到了进一步扩展，提供了只需一次的语义和事务。|154180|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/|154181|entityMap|0|LINK|mutability|MUTABLE|url|https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/|1|https://kafka.apache.org/documentation/streams/|2^0|0|6|2E|0|0|D|1B|1|0|0|0|2P|2|0^^$0|@$1|2|3|4|5|6|7|W|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|X|8|@]|9|@$D|Y|E|Z|1|10]]|A|$]]|$1|F|3|G|5|6|7|11|8|@]|9|@$D|12|E|13|1|14]]|A|$]]|$1|H|3|I|5|6|7|15|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|16|8|@]|9|@$D|17|E|18|1|19]]|A|$]]|$1|L|3|-4|5|6|7|1A|8|@]|9|@]|A|$]]]|M|$N|$5|O|P|Q|A|$R|S]]|T|$5|O|P|Q|A|$R|U]]|V|$5|O|P|Q|A|$R|K]]]]

I know that this is an older thread and the comparisons of Apache Kafka and Storm were valid and correct when they were written but it is worth noting that Apache Kafka has evolved a lot over the years and since version 0.10 (April 2016) Kafka has included a Kafka Streams API which provides stream processing capabilities without the need for any additional software such as Storm. Kafka also includes the Connect API for connecting into various sources and sinks (destinations) of data.

Announcement blog - <a href="https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/" rel="noreferrer">https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/</a>

Current Apache documentation - <a href="https://kafka.apache.org/documentation/streams/" rel="noreferrer">https://kafka.apache.org/documentation/streams/</a>

In 0.11 Kafka the stream processing functionality was further expanded to provide Exactly Once Semantics and Transactions.

<a href="https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/" rel="noreferrer">https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/</a>

blocks|key|4660677|text|当我有一个需要我可视化或提醒模式(考虑twitter趋势)的用例，同时继续处理事件时，我有几个模式。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4660678|NiFi将允许我处理一个事件，并使用非常少的自定义代码更新具有低(Er)批处理聚合的持久化数据存储。|4660679|Storm+(大量的自定义编码)让我可以几乎实时地访问热门事件。|4660680|如果我可以等待几秒钟，那么我就可以批量退出kafka，进入hdfs+(拼图)并进行处理。|4660681|如果我需要在几秒钟内知道，我需要NiFi，甚至可能是Storm。(考虑一下监控数千个地面站，在那里我需要查看龙卷风警告的小区域天气状况)。|4660682|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|M|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|N|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|O|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|P|8|@]|9|@]|A|$]]|$1|J|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|K|$]]

When I have a use case that requires me to visualize or alert on patterns (think of twitter trends), while continuing to process the events, I have a several patterns. 
NiFi would allow me to process an event and update a persistent data store with low(er) batch aggregation with very, very little custom coding. 
Storm (lots of custom coding) allows me nearly real time access to the trending events. 
If I can wait for many seconds, then I can batch out of kafka, into hdfs (Parquet) and process. 
If I need to know in seconds, I need NiFi, and probably even Storm. (Think of monitoring thousands of earth stations, where I need to see small region weather conditions for tornado warnings).

Apache Kafka: Distributed messaging system 
Apache Storm: Real Time Message Processing 

How we can use both technologies in a real-time data pipeline for processing event data?

In terms of real time data pipeline both seems to me do the job identical. How can we use both the technologies on a data pipeline?

Apache Kafka vs Apache Storm

消息队列

数据存储

数据管道

数据库

Kafka

Storm

Apache Kafka:分布式消息传递系统Apache Storm:实时消息处理我们如何在实时数据管道中使用这两种技术来处理事件数据？在实时数据管道方面，在我看来两者做的工作是一样的。我们如何在数据管道上使用这两种技术？

问Apache Kafka vs Apache Storm
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Apache Kafka vs Apache StormEN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Apache Kafka vs Apache Storm
EN