blocks|key|123452|text|RDD是dataset的逻辑参考，它跨集群中的许多服务器机器进行分区。RDD是不可变的，并且在发生故障时可以自我恢复。|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|CODE|entityRanges|data|123453|dataset可以是用户从外部加载的数据。它可以是json文件、csv文件或没有特定数据结构的文本文件。|123454|​|123455|📷|atomic|123456|123457|UPDATE：Here是一篇描述内部机制的文章：|123458|希望这能有所帮助。|123459|entityMap|0|IMAGE|mutability|IMMUTABLE|imageUrl|https://ask.qcloudimg.com/http-save/yehe-900000/53240e146748c552a59b3735e5294fb7.png|imageAlt|1|LINK|MUTABLE|url|https://github.com/linbojin/spark-notes/blob/master/research-papers/Resilient%2520Distributed%2520Datasets%2520A%2520Fault%2520Tolerant%2520Abstraction%2520for%2520In-Memory%2520Cluster%2520Computing.pdf^0|0|3|Z|3|4|7|0|0|7|0|0|0|1|0|0|0|0|6|G|8|7|4|1|0|0|9|0^^$0|@$1|2|3|4|5|6|7|16|8|@$9|17|A|18|B|C]|$9|19|A|1A|B|C]|$9|1B|A|1C|B|D]]|E|@]|F|$]]|$1|G|3|H|5|6|7|1D|8|@$9|1E|A|1F|B|D]]|E|@]|F|$]]|$1|I|3|J|5|6|7|1G|8|@]|E|@]|F|$]]|$1|K|3|L|5|M|7|1H|8|@]|E|@$9|1I|A|1J|1|1K]]|F|$]]|$1|N|3|J|5|6|7|1L|8|@]|E|@]|F|$]]|$1|O|3|P|5|6|7|1M|8|@$9|1N|A|1O|B|C]|$9|1P|A|1Q|B|C]]|E|@$9|1R|A|1S|1|1T]]|F|$]]|$1|Q|3|R|5|6|7|1U|8|@$9|1V|A|1W|B|C]]|E|@]|F|$]]|$1|S|3|-4|5|6|7|1X|8|@]|E|@]|F|$]]]|T|$U|$5|V|W|X|F|$Y|Z|10|-4]]|11|$5|12|W|13|F|$14|15]]]]

RDD is a logical reference of a <code>dataset</code> which is partitioned across many server machines in the cluster. RDDs are Immutable and are self recovered in case of failure.

<code>dataset</code> could be the data loaded externally by the user. It could be a json file, csv file or a text file with no specific data structure.

<a href="https://i.stack.imgur.com/IEcsA.png" rel="noreferrer"><img src="https://i.stack.imgur.com/IEcsA.png" alt="enter image description here"></a>

UPDATE: <a href="https://github.com/linbojin/spark-notes/blob/master/research-papers/Resilient%20Distributed%20Datasets%20A%20Fault%20Tolerant%20Abstraction%20for%20In-Memory%20Cluster%20Computing.pdf" rel="noreferrer">Here</a> is the paper what describe RDD internals: 

Hope this helps.

blocks|key|121629|text|从本质上讲，RDD是一组数据的Spark表示，这些数据分布在多台机器上，使用API让您可以对其执行操作。RDD可以来自任何数据源，例如文本文件、通过JDBC的数据库等。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|121630|正式的定义是：|121631|121632|+RDDs是一种容错的并行数据结构，允许用户显式地将中间结果持久化到内存中，控制其分区以优化数据放置，并使用一组丰富的操作符对其进行操作。|blockquote|121633|121634|121635|如果你想了解RDD的完整细节，请阅读Spark的核心学术论文之一，Resilient+Distributed+Datasets:+A+Fault-Tolerant+Abstraction+for+In-Memory+Cluster+Computing|offset|length|121636|entityMap|0|LINK|mutability|MUTABLE|url|https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf^0|0|0|0|0|0|0|X|2K|0|0^^$0|@$1|2|3|4|5|6|7|V|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|W|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|X|8|@]|9|@]|A|$]]|$1|E|3|F|5|G|7|Y|8|@]|9|@]|A|$]]|$1|H|3|-4|5|6|7|Z|8|@]|9|@]|A|$]]|$1|I|3|-4|5|6|7|10|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|11|8|@]|9|@$L|12|M|13|1|14]]|A|$]]|$1|N|3|-4|5|6|7|15|8|@]|9|@]|A|$]]]|O|$P|$5|Q|R|S|A|$T|U]]]]

An RDD is, essentially, the Spark representation of a set of data, spread across multiple machines, with APIs to let you act on it. An RDD could come from any datasource, e.g. text files, a database via JDBC, etc.

The formal definition is:

<blockquote>
 RDDs are fault-tolerant, parallel data structures that let users
 explicitly persist intermediate results in memory, control their
 partitioning to optimize data placement, and manipulate them using a
 rich set of operators.
</blockquote>

If you want the full details on what an RDD is, read one of the core Spark academic papers, <a href="https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf" rel="noreferrer">Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing</a>

blocks|key|121642|text|为了比较RDD和scala集合，下面是一些不同之处|type|unstyled|depth|inlineStyleRanges|entityRanges|data|121643|121644|相同，但在群集上运行|ordered-list-item|121645|Lazy本质上的scala集合是严格的|121646|RDD始终是不可变的，即，您不能更改集合中数据的状态|121647|RDD是自恢复的，即|121648|121649|entityMap^0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|P|8|@]|9|@]|A|$]]|$1|C|3|D|5|E|7|Q|8|@]|9|@]|A|$]]|$1|F|3|G|5|E|7|R|8|@]|9|@]|A|$]]|$1|H|3|I|5|E|7|S|8|@]|9|@]|A|$]]|$1|J|3|K|5|E|7|T|8|@]|9|@]|A|$]]|$1|L|3|-4|5|6|7|U|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|V|8|@]|9|@]|A|$]]]|N|$]]

To compare RDD with scala collection, below are few differences 

<ol>
<li>Same but runs on a cluster</li>
<li>Lazy in nature where scala collections are strict </li>
<li>RDD is always Immutable i.e., you can not change the state of the data in the collection </li>
<li>RDD are self recovered i.e., fault-tolerant </li>
</ol>

blocks|key|2888457|text|Resilient+Distributed+Dataset是Spark表示数据的方式。数据可以来自不同的来源：|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|2888458|2888459|文本文件|unordered-list-item|2888460|CSV文件|2888461|JSON文件|2888462|数据库(通过JBDC驱动程序)|2888463|2888464|与相关的Spark+RDD|2888465|Spark仅仅是RDD的一个实现。|2888466|Hadoop与的关系|2888467|Hadoop的强大之处在于，它允许用户编写并行计算，而不必担心工作分配和容错。然而，对于重用中间结果的应用程序来说，Hadoop的效率很低。例如，迭代机器学习算法，如PageRank，K均值聚类和逻辑回归，重用中间结果。|2888468|RDD允许在RAM中存储中间结果。Hadoop必须将其写入外部稳定的存储系统，从而生成磁盘I/O和序列化。对于迭代应用程序，使用RDD时，Spark的速度比Hadoop快20倍。|2888469|关于Spark的进一步实现细节|2888470|粗粒度转换|2888471|应用于RDD的转换是粗粒度的。这意味着对RDD的操作将应用于整个数据集，而不是其单个元素。因此，像map、filter、group、reduce这样的操作是允许的，但是像set(i)和get(i)这样的操作是不允许的。|2888472|粗粒度的反面是细粒度的。一个细粒度的存储系统将是一个数据库。|2888473|容错|2888474|RDD是容错的，这是一种属性，使系统能够在其一个组件发生故障时继续正常工作。|2888475|Spark的容错能力与其粗粒度特性紧密相关。在细粒度存储系统中实现容错的唯一方法是跨机器复制数据或记录更新。然而，在像Spark这样的粗粒度系统中，只记录转换。如果RDD的一个分区丢失，RDD有足够的信息快速重新计算它。|2888476|数据存储|2888477|RDD在分区中是“分布式的”(分离的)。每个分区可以存在于内存中或机器的磁盘上。当Spark想要在一个分区上启动一个任务时，他会将它发送到包含该分区的机器。这就是所谓的“本地感知调度”。|2888478|来源:关于火花的优秀研究论文：http://spark.apache.org/research.html|2888479|包括Ewan+Leith建议的论文。|2888480|entityMap|0|LINK|mutability|MUTABLE|url|http://spark.apache.org/research.html^0|0|T|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|F|11|0|0|0^^$0|@$1|2|3|4|5|6|7|1U|8|@$9|1V|A|1W|B|C]]|D|@]|E|$]]|$1|F|3|-4|5|6|7|1X|8|@]|D|@]|E|$]]|$1|G|3|H|5|I|7|1Y|8|@]|D|@]|E|$]]|$1|J|3|K|5|I|7|1Z|8|@]|D|@]|E|$]]|$1|L|3|M|5|I|7|20|8|@]|D|@]|E|$]]|$1|N|3|O|5|I|7|21|8|@]|D|@]|E|$]]|$1|P|3|-4|5|6|7|22|8|@]|D|@]|E|$]]|$1|Q|3|R|5|6|7|23|8|@]|D|@]|E|$]]|$1|S|3|T|5|6|7|24|8|@]|D|@]|E|$]]|$1|U|3|V|5|6|7|25|8|@]|D|@]|E|$]]|$1|W|3|X|5|6|7|26|8|@]|D|@]|E|$]]|$1|Y|3|Z|5|6|7|27|8|@]|D|@]|E|$]]|$1|10|3|11|5|6|7|28|8|@]|D|@]|E|$]]|$1|12|3|13|5|6|7|29|8|@]|D|@]|E|$]]|$1|14|3|15|5|6|7|2A|8|@]|D|@]|E|$]]|$1|16|3|17|5|6|7|2B|8|@]|D|@]|E|$]]|$1|18|3|19|5|6|7|2C|8|@]|D|@]|E|$]]|$1|1A|3|1B|5|6|7|2D|8|@]|D|@]|E|$]]|$1|1C|3|1D|5|6|7|2E|8|@]|D|@]|E|$]]|$1|1E|3|1F|5|6|7|2F|8|@]|D|@]|E|$]]|$1|1G|3|1H|5|6|7|2G|8|@]|D|@]|E|$]]|$1|1I|3|1J|5|6|7|2H|8|@]|D|@$9|2I|A|2J|1|2K]]|E|$]]|$1|1K|3|1L|5|6|7|2L|8|@]|D|@]|E|$]]|$1|1M|3|-4|5|6|7|2M|8|@]|D|@]|E|$]]]|1N|$1O|$5|1P|1Q|1R|E|$1S|1T]]]]

Resilient Distributed Dataset (RDD) is the way Spark represents data. The data can come from various sources :

<ul>
<li>Text File</li>
<li>CSV File</li>
<li>JSON File </li>
<li>Database (via JBDC driver)</li>
</ul>

RDD in relation to Spark

Spark is simply an implementation of RDD. 

RDD in relation to Hadoop

The power of Hadoop reside in the fact that it let users write parallel computations without having to worry about work distribution and fault tolerance. However, Hadoop is inefficient for the applications that reuse intermediate results. For example, iterative machine learning algorithms, such as PageRank, K-means clustering and logistic regression, reuse intermediate results. 

RDD allows to store intermediate results inside the RAM. Hadoop would have to write it to an external stable storage system, which generate disk I/O and serialization. With RDD, Spark is up to 20X faster than Hadoop for iterative applications. 

<h1>Futher implementations details about Spark</h1>

<h2>Coarse-Grained transformations</h2>

The transformations applied to an RDD are Coarse-Grained. This means that the operations on a RDD are applied to the whole dataset, not on its individual elements. Therefore, operations like map, filter, group, reduce are allowed, but operations like set(i) and get(i) are not. 

The inverse of coarse-grained is fine-grained. A fine-grained storage system would be a database. 

<h2>Fault Tolerant</h2>

RDD are fault tolerant, which is a property that enable the system to continue working properly in the event of the failure of one of its components. 

The fault tolerance of Spark is strongly linked to its coarse-grained nature. The only-way to implement fault tolerance in a fine-grained storage system is to replicate its data or log updates across machines. However, in a coarse-grained system like Spark, only the transformations are logged. If a partition of an RDD is lost, the RDD has enough information the recompute it quickly.

<h2>Data storage</h2>

The RDD is "distributed" (separated) in partitions. Each partitions can be present in the memory or on the disk of a machine. When Spark wants to launch a task on a partition, he sends it to the machine containing the partition. This is know as "locally aware scheduling".

Sources : 
Great research papers about Spark : 
<a href="http://spark.apache.org/research.html" rel="noreferrer">http://spark.apache.org/research.html</a>

Include the paper suggested by Ewan Leith.

blocks|key|2676964|text|RDD+=弹性分布式数据集|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2676965|(指物质或物体)(在弯曲、拉伸或压缩后能反弹或弹回原形)|2676966|RDD被定义为(来自LearningSpark+-+OREILLY)：总是重新计算RDD的能力实际上就是RDD被称为“弹性”的原因。当保存RDD数据的机器出现故障时，Spark使用此功能重新计算丢失的分区，这对用户是透明的。|2676967|这意味着“数据”肯定是随时可用的。此外，Spark可以在没有Hadoop的情况下运行，因此不会复制数据。Hadoop2.0最好的特性之一是在被动备用Namenode的帮助下的“高可用性”。Spark中的RDD也实现了同样的效果。|2676968|给定的RDD+(数据)可以跨越Spark集群中的各种节点(就像基于Hadoop的集群一样)。|2676969|如果任何节点崩溃，Spark可以重新计算RDD并将数据加载到其他节点，并且数据始终可用。Spark围绕着弹性分布式数据集(RDD)的概念，它是可以并行操作(http://spark.apache.org/docs/latest/programming-guide.html#resilient-distributed-datasets-rdds)的元素的容错集合|offset|length|style|BOLD|2676970|entityMap|0|LINK|mutability|MUTABLE|url|http://spark.apache.org/docs/latest/programming-guide.html#resilient-distributed-datasets-rdds^0|0|0|0|0|0|1G|3G|26|2M|0|0^^$0|@$1|2|3|4|5|6|7|X|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Y|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|Z|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|10|8|@]|9|@]|A|$]]|$1|H|3|I|5|6|7|11|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|12|8|@$L|13|M|14|N|O]]|9|@$L|15|M|16|1|17]]|A|$]]|$1|P|3|-4|5|6|7|18|8|@]|9|@]|A|$]]]|Q|$R|$5|S|T|U|A|$V|W]]]]

RDD = Resilient Distributed Dataset

Resilient (Dictionary meaning) = (of a substance or object) able to recoil or spring back into shape after bending, stretching, or being compressed

RDD is defined as (from LearningSpark - OREILLY): The ability to always recompute an RDD is actually why RDDs are called “resilient.” When a machine holding RDD data fails, Spark uses this ability to recompute the missing partitions, transparent to the user.

This means 'data' is surely available at all times. Also, Spark can run without Hadoop and hence data is NOT replicated. One of the best characterstics of Hadoop2.0 is 'High Availbility' with the help of Passive Standby Namenode. The same is achieved by RDD in Spark.

A given RDD (Data) can span across various nodes in Spark cluster (like in Hadoop based cluster).

If any node crashes, Spark can re-compute the RDD and loads the data in some other node, and data is always available.
Spark revolves around the concept of a resilient distributed dataset (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel (<a href="http://spark.apache.org/docs/latest/programming-guide.html#resilient-distributed-datasets-rdds" rel="noreferrer">http://spark.apache.org/docs/latest/programming-guide.html#resilient-distributed-datasets-rdds</a>)

blocks|key|2676987|text|从形式上讲，RDD是一个只读的、分区的记录集合。RDDs只能通过(1)稳定存储中的数据或(2)其他RDDs上的确定性操作来创建。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2676988|RDDs具有以下属性-|2676989|2676990|Immutability和分区:由分区的记录集合组成的+RDDs。分区是关系数据挖掘中并行的基本单位，每个分区是数据的一个逻辑分区，它是不变的，通过对已有的partitions.Immutability进行一些变换而产生的，有助于实现计算的一致性。|ordered-list-item|offset|length|style|BOLD|2676991|2676992|如果粗粒度操作：粗粒度操作是应用于数据集中所有元素的操作，则用户可以基于他们想要连接多个数据集的键来定义他们自己的分区标准。例如-将对RDD.|2676993|Fault容差分区中的所有元素执行的映射、过滤器或groupBy操作：由于RDD是在一组转换上创建的，因此它会记录这些转换，而不是这些转换的实际data.Graph，以生成一个RDD称为谱系图。|unordered-list-item|2676994|2676995|例如：|2676996|firstRDD=sc.textFile("hdfs://...")

secondRDD=firstRDD.filter(someFunction);

thirdRDD+=+secondRDD.map(someFunction);

result+=+thirdRDD.count()|code-block|syntax|javascript|2676997|如果我们丢失了RDD的某个分区，我们可以在该分区上重放转换以实现相同的计算，而不是跨多个nodes.This特性进行数据复制。RDD的最大好处是它节省了大量的数据管理和复制工作，从而实现了更快的计算速度。|2676998|2676999|Lazy评估：+Spark在动作中第一次使用RDD时会懒惰地计算RDD，以便它可以流水线转换。因此，在上述示例中，仅当invoked.|2677000|Persistence:+()动作为|2677001|时，才会评估RDD。用户可以指示他们将重用哪些RDD，并为它们选择存储策略(例如，内存中存储或磁盘上等)|2677002|RDDs的这些属性使它们对于快速计算非常有用。|2677003|entityMap^0|0|0|0|0|R|0|0|0|0|Z|0|0|0|0|0|0|0|7|0|0|C|0|0|0^^$0|@$1|2|3|4|5|6|7|1C|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|1D|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|1E|8|@]|9|@]|A|$]]|$1|E|3|F|5|G|7|1F|8|@$H|1G|I|1H|J|K]]|9|@]|A|$]]|$1|L|3|-4|5|6|7|1I|8|@]|9|@]|A|$]]|$1|M|3|N|5|6|7|1J|8|@]|9|@]|A|$]]|$1|O|3|P|5|Q|7|1K|8|@$H|1L|I|1M|J|K]]|9|@]|A|$]]|$1|R|3|-4|5|6|7|1N|8|@]|9|@]|A|$]]|$1|S|3|T|5|6|7|1O|8|@]|9|@]|A|$]]|$1|U|3|V|5|W|7|1P|8|@]|9|@]|A|$X|Y]]|$1|Z|3|10|5|6|7|1Q|8|@]|9|@]|A|$]]|$1|11|3|-4|5|6|7|1R|8|@]|9|@]|A|$]]|$1|12|3|13|5|G|7|1S|8|@$H|1T|I|1U|J|K]]|9|@]|A|$]]|$1|14|3|15|5|G|7|1V|8|@$H|1W|I|1X|J|K]]|9|@]|A|$]]|$1|16|3|17|5|6|7|1Y|8|@]|9|@]|A|$]]|$1|18|3|19|5|6|7|1Z|8|@]|9|@]|A|$]]|$1|1A|3|-4|5|6|7|20|8|@]|9|@]|A|$]]]|1B|$]]

Formally, an RDD is a read-only, partitioned collection of records. RDDs can only be created through deterministic operations on either (1) data in stable storage or (2) other RDDs.

RDDs have the following properties –

<ol>
<li>Immutability and partitioning:
 RDDs composed of collection of records which are partitioned. Partition is basic unit of parallelism in a RDD, and each partition is one logical division of data which is immutable and created through some transformations on existing partitions.Immutability helps to achieve consistency in computations.

Users can define their own criteria for partitioning based on keys on which they want to join multiple datasets if needed.</li>
<li>Coarse grained operations:
Coarse grained operations are operations which are applied to all elements in datasets. For example – a map, or filter or groupBy operation which will be performed on all elements in a partition of RDD.</li>
<li>Fault Tolerance:
 Since RDDs are created over a set of transformations , it logs those transformations, rather than actual data.Graph of these transformations to produce one RDD is called as Lineage Graph.</li>
</ol>

For example –

<pre><code>firstRDD=sc.textFile("hdfs://...")

secondRDD=firstRDD.filter(someFunction);

thirdRDD = secondRDD.map(someFunction);

result = thirdRDD.count()
</code></pre>

In case of we lose some partition of RDD , we can replay the transformation on that partition in lineage to achieve the same computation, rather than doing data replication across multiple nodes.This characteristic is biggest benefit of RDD , because it saves a lot of efforts in data management and replication and thus achieves faster computations.

<ol start="4">
<li>Lazy evaluations:
Spark computes RDDs lazily the first time they are used in an action, so that it can pipeline transformations. So , in above example RDD will be evaluated only when count() action is invoked.</li>
<li>Persistence:
Users can indicate which RDDs they will reuse and choose a storage strategy for them (e.g., in-memory storage or on Disk etc.)</li>
</ol>

These properties of RDDs make them useful for fast computations.

blocks|key|123677|text|RDD+(Resilient+Distributed+Datasets)是一种表示数据的抽象。从形式上讲，它们是一个只读的、分区的记录集合，提供了一个方便的API。|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|123678|RDD通过解决一些关键问题，为在MapReduce等集群计算框架上处理大型数据集提供了一种高效的解决方案：|123679|弹性数据保存在内存中以减少磁盘I/O；这与迭代计算特别相关--不必将中间数据持久化到磁盘|123680|fault-tolerance+(弹性)不是通过复制数据获得的，而是通过跟踪应用于初始数据集(|unordered-list-item|123681|)的所有转换来获得的。这样，在发生故障时，丢失的数据始终可以根据其沿袭关系重新计算，避免再次复制数据可降低存储开销|123682|惰性评估，即在需要时先执行计算|123683|123684|123685|RDD有两个主要限制：|123686|仅允许粗粒度转换(即，应用于整个数据集的操作)+(read-only)|123687|they+(仅允许粗粒度转换(即，应用于整个dataset的操作))|123688|123689|RDD的一个很好的概念优势是，它们将数据和代码打包在一起，从而更容易重用数据管道。|123690|资料来源：Resilient+Distributed+Datasets:+A+Fault-Tolerant+Abstraction+for+In-Memory+Cluster+Computing，An+Architecture+for+Fast+and+General+Data+Processing+on+Large+Clusters|123691|entityMap|0|LINK|mutability|MUTABLE|url|https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf|1|https://www2.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-12.pdf^0|0|3|5|1|F|1|R|1|0|0|0|0|0|0|0|0|0|0|0|0|0|5|2K|0|2Q|1Y|1|0^^$0|@$1|2|3|4|5|6|7|1D|8|@$9|1E|A|1F|B|C]|$9|1G|A|1H|B|C]|$9|1I|A|1J|B|C]|$9|1K|A|1L|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|1M|8|@]|D|@]|E|$]]|$1|H|3|I|5|6|7|1N|8|@]|D|@]|E|$]]|$1|J|3|K|5|L|7|1O|8|@]|D|@]|E|$]]|$1|M|3|N|5|L|7|1P|8|@]|D|@]|E|$]]|$1|O|3|P|5|L|7|1Q|8|@]|D|@]|E|$]]|$1|Q|3|-4|5|6|7|1R|8|@]|D|@]|E|$]]|$1|R|3|-4|5|6|7|1S|8|@]|D|@]|E|$]]|$1|S|3|T|5|6|7|1T|8|@]|D|@]|E|$]]|$1|U|3|V|5|6|7|1U|8|@]|D|@]|E|$]]|$1|W|3|X|5|L|7|1V|8|@]|D|@]|E|$]]|$1|Y|3|-4|5|6|7|1W|8|@]|D|@]|E|$]]|$1|Z|3|10|5|6|7|1X|8|@]|D|@]|E|$]]|$1|11|3|12|5|6|7|1Y|8|@]|D|@$9|1Z|A|20|1|21]|$9|22|A|23|1|24]]|E|$]]|$1|13|3|-4|5|6|7|25|8|@]|D|@]|E|$]]]|14|$15|$5|16|17|18|E|$19|1A]]|1B|$5|16|17|18|E|$19|1C]]]]

RDD (Resilient Distributed Datasets) are an abstraction for representing data. Formally they are a read-only, partitioned collection of records that provides a convenient API.

RDD provide a performant solution for processing large datasets on cluster computing frameworks such as MapReduce by addressing some key issues:

<ul>
<li>data is kept in memory to reduce disk I/O; this is particularly relevant for iterative computations -- not having to persist intermediate data to disk </li>
<li>fault-tolerance (resilience) is obtained not by replicating data but by keeping track of all transformations applied to the initial dataset (the lineage). This way, in case of failure lost data can always be recomputed from its lineage and avoiding data replication again reduces storage overhead </li>
<li>lazy evaluation, i.e. computations are carried out first when they're needed</li>
</ul>

RDD's have two main limitations:

<ul>
<li>they're immutable (read-only)</li>
<li>they only allow coarse-grained transformations (i.e. operations that apply to the entire dataset)</li>
</ul>

One nice conceptual advantage of RDD's is that they pack together data and code making it easier to reuse data pipelines.

Sources: <a href="https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf" rel="nofollow noreferrer">Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing</a>, <a href="https://www2.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-12.pdf" rel="nofollow noreferrer">An Architecture for Fast and General Data Processing on Large Clusters</a>

blocks|key|123693|text|RDD是一种在spark.The中表示数据的方式。数据源可以是JSON、CSV文本文件或其他数据源。RDD是容错的，这意味着它将数据存储在多个位置(即数据以分布式形式存储)，因此如果一个节点发生故障，数据可以恢复。在RDD中，数据始终可用。然而，RDD很慢，很难编码，因此过时了。它已经被DataFrame和Dataset的概念所取代。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|123694|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

RDD is a way of representing data in spark.The source of data can be JSON,CSV textfile or some other source. 
RDD is fault tolerant which means that it stores data on multiple locations(i.e the data is stored in distributed form ) so if a node fails the data can be recovered.
In RDD data is available at all times. 
However RDD are slow and hard to code hence outdated.
It has been replaced by concept of DataFrame and Dataset.

blocks|key|123555|text|RDD是一个弹性的分布式数据集。它是spark的核心部分。它是spark的一个低级API。DataFrame和DataSets都是建立在关系数据库之上的。RDD只是行级数据，即位于n个执行器上。RDD是不可变的.means，你不能改变RDD。但是您可以使用转换和操作创建新的RDD|type|unstyled|depth|inlineStyleRanges|entityRanges|data|123556|entityMap^0|0^^$0|@$1|2|3|4|5|6|7|D|8|@]|9|@]|A|$]]|$1|B|3|-4|5|6|7|E|8|@]|9|@]|A|$]]]|C|$]]

RDD
is an Resilient Distributed Data Set.
It is an core part of spark.
It is an Low Level API of spark.
DataFrame and DataSets are built on top of RDD.
RDD are nothing but row level data i.e. sits on n number of executors.
RDD's are immutable .means you cannot change the RDD. But you can create new RDD using Transformation and Actions

blocks|key|121789|text|弹性分布式数据集(RDDs)|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|121790|Resilient：如果在spark中的节点上执行操作时丢失，可以从历史中重建数据集。|121791|+distributed+：RDDs中的数据被划分为一个或多个分区，并作为对象在内存中的集合分布在集群中的工作节点上。|121792|+dataset+：RDDs是由记录组成的数据集，记录是数据集中唯一可识别的数据集合。|121793|entityMap^0|0|E|0|0|9|0|0|D|0|0|9|0^^$0|@$1|2|3|4|5|6|7|N|8|@$9|O|A|P|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|Q|8|@$9|R|A|S|B|C]]|D|@]|E|$]]|$1|H|3|I|5|6|7|T|8|@$9|U|A|V|B|C]]|D|@]|E|$]]|$1|J|3|K|5|6|7|W|8|@$9|X|A|Y|B|C]]|D|@]|E|$]]|$1|L|3|-4|5|6|7|Z|8|@]|D|@]|E|$]]]|M|$]]

Resilient Distributed Datasets (RDDs)
Resilient: If an operation is lost while performing on a node in spark, the dataset can be reconstituted from history.
Distributed: Data in RDDs is divided into one or many partitions and distributed as in-memory collections of objects across worker nodes in the cluster.
Dataset: RDDs are datasets that consist of records, records are uniquely identifiable data collections within a dataset.

Definition says: 

<blockquote>
 RDD is immutable distributed collection of objects 
</blockquote>

I don't quite understand what does it mean. Is it like data (partitioned objects) stored on hard disk If so then how come RDD's can have user-defined classes (Such as java, scala or python)

From this link: <a href="https://www.safaribooksonline.com/library/view/learning-spark/9781449359034/ch03.html" rel="noreferrer">https://www.safaribooksonline.com/library/view/learning-spark/9781449359034/ch03.html</a> It mentions:

<blockquote>
 Users create RDDs in two ways: by loading an external dataset, or by
 distributing a collection of objects (e.g., a list or set) in their
 driver program
</blockquote>

I am really confused understanding RDD in general and in relation to spark and hadoop.

Can some one please help.

What is RDD in spark

机器学习算法

数据挖掘

数据存储

机器学习

数据结构

数据管理

数据管道

MapReduce

服务器

数据库

Hadoop 

Spark 

JSON

定义是： RDD是不可变的分布式对象集合。我不太明白这是什么意思。它与存储在硬盘上的数据(分区对象)一样吗?如果是这样，那么为什么RDD可以有用户定义的类(如java、scala或python)？从这个链接：它提到：用户以两种方式创建RDDs :通过加载外部数据集，或者通过在其驱动程序中分发对象集合(例如，列表或集合我...

问什么是spark中的RDD
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问什么是spark中的RDDEN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问什么是spark中的RDD
EN