blocks|key|187892|text|在为Kafka构建数据结构时，这真的取决于数据的使用方式。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|187893|在我看来，主题是一组相似类型的消息，将被相同类型的消费者消费，所以在上面的例子中，我只有一个主题，如果你决定通过Kafka推送一些其他类型的数据，你可以稍后添加一个新的主题。|187894|主题在ZooKeeper中注册，这意味着如果尝试添加太多主题，您可能会遇到问题，例如，如果您有一百万用户，并决定为每个用户创建一个主题。|187895|另一方面，分区是并行化消息消耗的一种方式。+broker集群中的分区总数需要至少与消费者组中的消费者数量相同，才能理解分区功能。消费者组中的消费者会根据分区将处理主题的负担分摊给自己，从而使一个消费者只关注被分配到的分区本身中的消息。|offset|length|style|BOLD|187896|分区可以使用生产者端的分区键显式设置，如果没有提供，将为每条消息选择一个随机分区。|187897|entityMap^0|0|0|0|L|1D|0|0^^$0|@$1|2|3|4|5|6|7|P|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Q|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|R|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|S|8|@$H|T|I|U|J|K]]|9|@]|A|$]]|$1|L|3|M|5|6|7|V|8|@]|9|@]|A|$]]|$1|N|3|-4|5|6|7|W|8|@]|9|@]|A|$]]]|O|$]]

When structuring your data for Kafka it really depends on how it´s meant to be consumed.
In my mind, a topic is a grouping of messages of a similar type that will be consumed by the same type of consumer so in the example above, I would just have a single topic and if you´ll decide to push some other kind of data through Kafka, you can add a new topic for that later.
Topics are registered in ZooKeeper which means that you might run into issues if trying to add too many of them, e.g. the case where you have a million users and have decided to create a topic per user.
Partitions on the other hand is a way to parallelize the consumption of the messages. The total number of partitions in a broker cluster need to be at least the same as the number of consumers in a consumer group to make sense of the partitioning feature. Consumers in a consumer group will split the burden of processing the topic between themselves according to the partitioning so that one consumer will only be concerned with messages in the partition itself is &quot;assigned to&quot;.
Partitioning can either be explicitly set using a partition key on the producer side or if not provided, a random partition will be selected for every message.

blocks|key|2802228|text|一旦您知道如何对事件流进行分区，主题名称就很容易了，所以让我们先回答这个问题。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2802229|@Ludd是正确的-您选择的分区结构在很大程度上取决于您希望如何处理事件流。理想情况下，您需要一个分区键，这意味着您的事件处理是partition-local.|offset|length|style|BOLD|2802230|例如：|2802231|2802232|如果你关心用户在站点上的平均时间，那么你应该按:user-id分区。这样，与单个用户的站点活动相关的所有事件都将在同一分区中可用。这意味着像Apache+Samza这样的流处理引擎可以通过查看单个分区中的事件来计算给定用户的平均现场时间。这避免了必须执行任何类型的昂贵的partition-global处理|ordered-list-item|CODE|2802233|如果你关心你的网站上最受欢迎的页面，你应该按:viewed页面分区。同样，通过查看单个分区|2802234|中的事件，Samza将能够对给定页面的视图进行计数|2802235|通常，我们试图避免依赖全局状态(例如将计数保存在DynamoDB或Cassandra等远程数据库中)，而是能够使用分区本地状态工作。这是因为local+state+is+a+fundamental+primitive+in+stream+processing。|2802236|如果您需要上述两种用例，那么Kafka的常见模式是首先使用:user-id进行分区，然后使用:viewed进行re-partition，为下一阶段的处理做好准备。|2802237|在主题名称上-这里一个明显的名称是events或user-events。更具体地说，你可以使用events-by-user-id和/或events-by-viewed。|2802238|entityMap|0|LINK|mutability|MUTABLE|url|http://samza.apache.org/|1|http://radar.oreilly.com/2014/07/why-local-state-is-a-fundamental-primitive-in-stream-processing.html^0|0|1S|F|0|0|0|N|8|3R|G|1Y|C|0|0|M|7|0|0|1Y|1N|1|0|T|8|1A|7|1J|C|0|H|6|O|B|1B|H|1V|G|0^^$0|@$1|2|3|4|5|6|7|18|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|19|8|@$D|1A|E|1B|F|G]]|9|@]|A|$]]|$1|H|3|I|5|6|7|1C|8|@]|9|@]|A|$]]|$1|J|3|-4|5|6|7|1D|8|@]|9|@]|A|$]]|$1|K|3|L|5|M|7|1E|8|@$D|1F|E|1G|F|N]|$D|1H|E|1I|F|G]]|9|@$D|1J|E|1K|1|1L]]|A|$]]|$1|O|3|P|5|M|7|1M|8|@$D|1N|E|1O|F|N]]|9|@]|A|$]]|$1|Q|3|R|5|6|7|1P|8|@]|9|@]|A|$]]|$1|S|3|T|5|6|7|1Q|8|@]|9|@$D|1R|E|1S|1|1T]]|A|$]]|$1|U|3|V|5|6|7|1U|8|@$D|1V|E|1W|F|N]|$D|1X|E|1Y|F|N]|$D|1Z|E|20|F|G]]|9|@]|A|$]]|$1|W|3|X|5|6|7|21|8|@$D|22|E|23|F|N]|$D|24|E|25|F|N]|$D|26|E|27|F|N]|$D|28|E|29|F|N]]|9|@]|A|$]]|$1|Y|3|-4|5|6|7|2A|8|@]|9|@]|A|$]]]|Z|$10|$5|11|12|13|A|$14|15]]|16|$5|11|12|13|A|$14|17]]]]

Once you know how to partition your event stream, the topic name will be easy, so let's answer that question first.

@Ludd is correct - the partition structure you choose will depend largely on how you want to process the event stream. Ideally you want a partition key which means that your event processing is partition-local.

For example:

<ol>
<li>If you care about users' average time-on-site, then you should partition by <code>:user-id</code>. That way, all the events related to a single user's site activity will be available within the same partition. This means that a stream processing engine such as <a href="http://samza.apache.org/">Apache Samza</a> can calculate average time-on-site for a given user just by looking at the events in a single partition. This avoids having to perform any kind of costly partition-global processing</li>
<li>If you care about the most popular pages on your website, you should partition by the <code>:viewed</code> page. Again, Samza will be able to keep a count of a given page's views just by looking at the events in a single partition</li>
</ol>

Generally, we are trying to avoid having to rely on global state (such as keeping counts in a remote database like DynamoDB or Cassandra), and instead be able to work using partition-local state. This is because <a href="http://radar.oreilly.com/2014/07/why-local-state-is-a-fundamental-primitive-in-stream-processing.html">local state is a fundamental primitive in stream processing</a>.

If you need both of the above use-cases, then a common pattern with Kafka is to first partition by say <code>:user-id</code>, and then to re-partition by <code>:viewed</code> ready for the next phase of processing.

On topic names - an obvious one here would be <code>events</code> or <code>user-events</code>. To be more specific you could go with with <code>events-by-user-id</code> and/or <code>events-by-viewed</code>.

blocks|key|185809|text|我认为主题名称是一种消息的结论，生产者通过订阅主题发布消息到主题，消费者订阅消息。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|185810|一个主题可以有多个分区。分区有利于并行性。分区也是复制的单位，所以在卡夫卡中，领导者和追随者也是在分区的层面上说的。实际上，分区是一个有序队列，其顺序是消息到达的顺序。并且该主题由一个或多个队列在一个简单的单词中组成。这对我们建模我们的结构很有用。|185811|Kafka是由LinkedIn开发的用于日志聚合和分发的工具。这个场景就是一个很好的例子。|185812|您的web服务器可以记录用户在Web或应用程序上的事件，然后通过生产者将其发送到Kafka+broker。在producer中，您可以指定分区方式，例如:事件类型(不同的事件保存在不同的分区中)或事件时间(根据您的应用逻辑将一天划分为不同的周期)或用户类型或仅无逻辑，并将所有日志均衡到多个分区中。|185813|对于您的案例，您可以创建一个名为"page-view-event“的主题，并通过哈希键创建N个分区，将日志均匀分布到所有分区。或者你可以选择一种分区逻辑，让日志按你的精神分布。|185814|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|M|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|N|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|O|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|P|8|@]|9|@]|A|$]]|$1|J|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|K|$]]

I think topic name is a conclusion of a kind of messages, and producer publish message to the topic and consumer subscribe message through subscribe topic. 

A topic could have many partitions. partition is good for parallelism. partition is also the unit of replication,so in Kafka, leader and follower is also said at the level of partition. Actually a partition is an ordered queue which the order is the message arrived order. And the topic is composed by one or more queue in a simple word. This is useful for us to model our structure.

Kafka is developed by LinkedIn for log aggregation and delivery. this scene is very good as a example.

The user's events on your web or app can be logged by your Web sever and then sent to Kafka broker through the producer. In producer, you could specific the partition method, for example : event type (different event is saved in different partition) or event time (partition a day into different period according your app logic) or user type or just no logic and balance all logs into many partitions.

About your case in question, you can create one topic called "page-view-event", and create N partitions through hash keys to distribute the logs into all partitions evenly. Or you could choose a partition logic to make log distributing by your spirit.

blocks|key|187956|text|这与问题并不完全相关，但是如果你已经决定了基于主题的记录的逻辑分离，并且想要优化Kafka中的主题/分区计数，this博客帖子可能会派上用场。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|187957|主要内容概括如下：|187958|187959|一般来说，Kafka集群中的分区越多，吞吐量就越高。让单个生产分区上可实现的最大吞吐量为p，消耗为c。假设您的目标吞吐量是t。然后，您至少需要有max(t/p，t/c)分区。|unordered-list-item|style|BOLD|187960|187961|187962|目前，在Kafka中，每个代理打开每个日志段的索引和数据文件的文件句柄。因此，分区越多，在底层操作系统中配置打开文件句柄限制的要求就越高。例如，在我们的生产系统中，我们曾经看到一个错误too+many+files+are+open，而我们有大约3600个主题分区。|CODE|187963|187964|187965|当代理不干净地关闭时(例如kill+-9)，观察到的不可用可能与分区数成正比。|187966|187967|187968|+Kafka中的端到端延迟由生产者发布消息到消费者读取消息的时间来定义。根据经验，如果您关心延迟，最好将每个代理的分区数量限制为100+x+b+x+r，其中b是Kafka集群中的代理数量，r是复制因子。|187969|187970|entityMap|0|LINK|mutability|MUTABLE|url|https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/^0|1J|4|0|0|0|0|18|1|1D|1|1P|1|24|1|26|1|28|1|2A|1|0|0|0|2K|N|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|17|8|@]|9|@$A|18|B|19|1|1A]]|C|$]]|$1|D|3|E|5|6|7|1B|8|@]|9|@]|C|$]]|$1|F|3|-4|5|6|7|1C|8|@]|9|@]|C|$]]|$1|G|3|H|5|I|7|1D|8|@$A|1E|B|1F|J|K]|$A|1G|B|1H|J|K]|$A|1I|B|1J|J|K]|$A|1K|B|1L|J|K]|$A|1M|B|1N|J|K]|$A|1O|B|1P|J|K]|$A|1Q|B|1R|J|K]]|9|@]|C|$]]|$1|L|3|-4|5|6|7|1S|8|@]|9|@]|C|$]]|$1|M|3|-4|5|6|7|1T|8|@]|9|@]|C|$]]|$1|N|3|O|5|I|7|1U|8|@$A|1V|B|1W|J|P]]|9|@]|C|$]]|$1|Q|3|-4|5|6|7|1X|8|@]|9|@]|C|$]]|$1|R|3|-4|5|6|7|1Y|8|@]|9|@]|C|$]]|$1|S|3|T|5|I|7|1Z|8|@]|9|@]|C|$]]|$1|U|3|-4|5|6|7|20|8|@]|9|@]|C|$]]|$1|V|3|-4|5|6|7|21|8|@]|9|@]|C|$]]|$1|W|3|X|5|I|7|22|8|@]|9|@]|C|$]]|$1|Y|3|-4|5|6|7|23|8|@]|9|@]|C|$]]|$1|Z|3|-4|5|6|7|24|8|@]|9|@]|C|$]]]|10|$11|$5|12|13|14|C|$15|16]]]]

This is not exactly related to the question, but in case you already have decided upon the logical segregation of records based on topics, and want to optimize the topic/partition count in Kafka, <a href="https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/" rel="nofollow noreferrer">this</a> blog post might come handy.
Key takeaways in a nutshell:
<ul>
<li>In general, the more partitions there are in a Kafka cluster, the higher the throughput one can achieve. Let the max throughout achievable on a single partition for production be p and consumption be c. Let’s say your target throughput is t. Then you need to have at least max(t/p, t/c) partitions.
</li>
<li>Currently, in Kafka, each broker opens a file handle of both the index and the data file of every log segment. So, the more partitions, the higher that one needs to configure the open file handle limit in the underlying operating system. E.g. in our production system, we once saw an error saying <code>too many files are open</code>, while we had around 3600 topic partitions.
</li>
<li>When a broker is shut down uncleanly (e.g., kill -9), the observed unavailability could be proportional to the number of partitions.
</li>
<li>The end-to-end latency in Kafka is defined by the time from when a message is published by the producer to when the message is read by the consumer. As a rule of thumb, if you care about latency, it’s probably a good idea to limit the number of partitions per broker to 100 x b x r, where b is the number of brokers in a Kafka cluster and r is the replication factor.
</li>
</ul>

One of the first things I think about when using a new service (such as a non-RDBMS data store or a message queue) is: "How should I structure my data?".

I've read and watched some introductory materials. In particular, take, for example, <a href="http://www.systemswemake.com/papers/kafka">Kafka: a Distributed Messaging System for Log Processing</a>, which writes:

<ul>
<li>"a Topic is the container with which messages are associated"</li>
<li>"the smallest unit of parallelism is the partition of a topic. This implies that all messages that ... belong to a particular partition of a topic will be consumed by a consumer in a consumer group."</li>
</ul>

Knowing this, what would be a good example that illustrates how to use topics and partitions? When should something be a topic? When should something be a partition?

As an example, let's say my (Clojure) data looks like:

<pre><code>{:user-id 101 :viewed "/page1.html" :at #inst "2013-04-12T23:20:50.22Z"}
{:user-id 102 :viewed "/page2.html" :at #inst "2013-04-12T23:20:55.50Z"}
</code></pre>

Should the topic be based on <code>user-id</code>? <code>viewed</code>? <code>at</code>? What about the partition?

How do I decide?

Data Modeling with Kafka? Topics and Partitions

在使用新服务(例如非RDBMS数据存储或消息队列)时，我首先考虑的事情之一是：“我应该如何构造我的数据？”我已经阅读并观看了一些介绍性材料。特别是，以为例，它写道：“主题是用于存储消息的容器，最小并行单位是主题的分区。这意味着...属于主题的特定分区的所有消息都将由使用者组中的使用者使用。”了解了这一点，什么是一个很好...

问使用Kafka进行数据建模？主题和分区
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问使用Kafka进行数据建模？主题和分区EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问使用Kafka进行数据建模？主题和分区
EN