blocks|key|4607615|text|根据Hadoop+:+Definitive+Guide的说法|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|4607616|4607617|+Hadoop的默认策略是将第一个副本放置在与客户端相同的节点上(对于在群集外部运行的客户端，将随机选择一个节点，尽管系统会尝试不选择太满或太忙的节点)。第二个副本放置在与第一个副本不同的机架上(机架外)，随机选择。第三个副本与第二个副本放置在相同的机架上，但位于随机选择的不同节点上。虽然系统会尽量避免在同一机架上放置太多副本，但会将更多副本放置在群集上的随机节点上。|blockquote|4607618|4607619|4607620|这个逻辑是有意义的，因为它减少了不同节点之间的网络聊天。但是，这本书是在2009年出版的，Hadoop框架有很多变化。|4607621|我认为这取决于客户端是否与Hadoop节点相同。如果客户端是Hadoop节点，则所有拆分都将位于同一节点上。尽管集群中有多个节点，但这并不能提供更好的读/写吞吐量。如果客户端与Hadoop节点不同，则会为每个拆分随机选择节点，因此拆分将分布在集群中的节点上。现在，这提供了更好的读/写吞吐量。|4607622|写入多个节点的一个优点是，即使其中一个节点宕机，也可能会有几个拆分中断，但至少可以通过某种方式从剩余拆分中恢复一些数据。|4607623|entityMap|0|LINK|mutability|MUTABLE|url|http://shop.oreilly.com/product/0636920010388.do^0|2|P|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|X|8|@]|9|@$A|Y|B|Z|1|10]]|C|$]]|$1|D|3|-4|5|6|7|11|8|@]|9|@]|C|$]]|$1|E|3|F|5|G|7|12|8|@]|9|@]|C|$]]|$1|H|3|-4|5|6|7|13|8|@]|9|@]|C|$]]|$1|I|3|-4|5|6|7|14|8|@]|9|@]|C|$]]|$1|J|3|K|5|6|7|15|8|@]|9|@]|C|$]]|$1|L|3|M|5|6|7|16|8|@]|9|@]|C|$]]|$1|N|3|O|5|6|7|17|8|@]|9|@]|C|$]]|$1|P|3|-4|5|6|7|18|8|@]|9|@]|C|$]]]|Q|$R|$5|S|T|U|C|$V|W]]]]

According to the <a href="http://shop.oreilly.com/product/0636920010388.do" rel="noreferrer">Hadoop : Definitive Guide</a>

<blockquote>
 Hadoop’s default strategy is to place the first replica on the same node as the client (for
 clients running outside the cluster, a node is chosen at random, although the system
 tries not to pick nodes that are too full or too busy). The second replica is placed on a
 different rack from the first (off-rack), chosen at random. The third replica is placed on
 the same rack as the second, but on a different node chosen at random. Further replicas
 are placed on random nodes on the cluster, although the system tries to avoid placing
 too many replicas on the same rack.
</blockquote>

This logic makes sense as it decreases the network chatter between the different nodes. But, the book was published in 2009 and there had been a lot of changes in the Hadoop framework.

I think it depends on, whether the client is same as a Hadoop node or not. If the client is a Hadoop node then all the splits will be on the same node. This doesn't provide any better read/write throughput in-spite of having multiple nodes in the cluster. If the client is not same as the Hadoop node, then the node is chosen at random for each split, so the splits are spread across the nodes in a cluster. Now, this provides a better read/write throughput.

One advantage of writing to multiple nodes is that even if one of the node goes down, a couple of splits might be down, but at least some data can be recovered somehow from the remaining splits.

blocks|key|584614|text|如果将replication设置为1，则文件将仅出现在客户端节点上，即您要从其上载文件的节点上。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|584615|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

If you set replication to be 1, then the file will be present only on the client node, that is the node from where you are uploading the file.

blocks|key|4324393|text|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4324394|如果您的群集是单节点，则当您上载文件时，该文件将根据数据块大小溢出，并保留在单机中。|unordered-list-item|4324395|如果您的群集是多节点，则当您上载文件时，该文件将根据数据块大小溢出，并将通过管道分发到群集中的不同数据节点，NameNode将决定应将数据移动到群集中的哪个位置。|4324396|4324397|HDFS复制因子用于制作数据的副本(即)如果您的复制者因子为2，则您上传到HDFS的所有数据都将具有副本。|4324398|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|-4|4|5|6|K|7|@]|8|@]|9|$]]|$1|A|3|B|4|C|6|L|7|@]|8|@]|9|$]]|$1|D|3|E|4|C|6|M|7|@]|8|@]|9|$]]|$1|F|3|-4|4|5|6|N|7|@]|8|@]|9|$]]|$1|G|3|H|4|5|6|O|7|@]|8|@]|9|$]]|$1|I|3|-4|4|5|6|P|7|@]|8|@]|9|$]]]|J|$]]

<ul>
<li>If your cluster is single node then when you upload a file it will be spilled according to the blocksize and it remains in single machine.</li>
<li>If your cluster is Multi node then when you upload a file it will be spilled according to the blocksize and it will be distributed to different datanode in your cluster via pipeline and NameNode will decide where the data should be moved in the cluster.</li>
</ul>

HDFS replication factor is used to make a copy of the data (i.e) if your replicator factor is 2 then all the data which you upload to HDFS will have a copy.

blocks|key|587201|text|如果您将复制因子设置为1，则表示单节点群集。它只有一个客户端节点http://commandstech.com/replication-factor-in-hadoop/。您可以在其中上传文件，然后在单个节点或客户端节点中使用。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|587202|entityMap|0|LINK|mutability|MUTABLE|url|http://commandstech.com/replication-factor-in-hadoop/^0|W|1H|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

If you set replication factor is 1 it means that the single node cluster. It has only one client node <a href="http://commandstech.com/replication-factor-in-hadoop/" rel="nofollow noreferrer">http://commandstech.com/replication-factor-in-hadoop/</a>. Where you can upload files then use in a single node or client node.

When I'm uploading a file to HDFS, if I set the replication factor to 1 then the file splits gonna reside on one single machine or the splits would be distributed to multiple machines across the network ? 

<pre><code>hadoop fs -D dfs.replication=1 -copyFromLocal file.txt /user/ablimit
</code></pre>

HDFS replication factor

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

当我将文件上传到HDFS时，如果我将复制因子设置为1，那么文件拆分将驻留在一台计算机上，或者拆分将分布到网络上的多台计算机上？hadoop fs -D dfs.replication=1 -copyFromLocal file.txt /user/ablimit

问HDFS复制系数
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问HDFS复制系数EN