blocks|key|142599|text|以下是作为spark-shell会话的不同之处的示例：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|142600|首先，一些数据-两行文本：|142601|val+rdd+=+sc.parallelize(Seq("Roses+are+red",+"Violets+are+blue"))++//+lines

rdd.collect

++++res0:+Array[String]+=+Array("Roses+are+red",+"Violets+are+blue")|code-block|syntax|javascript|142602|现在，map将一个长度为N的RDD转换为另一个长度为N的RDD。|142603|例如，它从两行映射到两行长度：|142604|rdd.map(_.length).collect

++++res1:+Array[Int]+=+Array(13,+16)|142605|但是flatMap+(不严格地说)将长度为N的RDD转换为N个集合的集合，然后将这些集合展平为结果的单个RDD。|142606|rdd.flatMap(_.split("+")).collect

++++res2:+Array[String]+=+Array("Roses",+"are",+"red",+"Violets",+"are",+"blue")|142607|我们每行有多个单词，并且有多行，但是我们最终得到了一个单词的输出数组|142608|为了说明这一点，从行集合到单词集合的flatMapping如下所示：|142609|["aa+bb+cc",+"",+"dd"]+=>+[["aa","bb","cc"],[],["dd"]]+=>+["aa","bb","cc","dd"]|142610|因此，对于flatMap，输入和输出的RDDs通常具有不同的大小。|142611|如果我们试图将map与我们的split函数一起使用，我们最终会得到嵌套结构(单词数组的RDD，类型为RDD[Array[String]])，因为我们必须在每个输入中只有一个结果：|142612|rdd.map(_.split("+")).collect

++++res3:+Array[Array[String]]+=+Array(
+++++++++++++++++++++++++++++++++++++Array(Roses,+are,+red),+
+++++++++++++++++++++++++++++++++++++Array(Violets,+are,+blue)
+++++++++++++++++++++++++++++++++)|142613|最后，一个有用的特例是映射到一个可能不返回答案的函数，因此返回一个Option。我们可以使用flatMap过滤掉返回None的元素，并从返回Some的元素中提取值|142614|val+rdd+=+sc.parallelize(Seq(1,2,3,4))

def+myfn(x:+Int):+Option[Int]+=+if+(x+<=+2)+Some(x+*+10)+else+None

rdd.flatMap(myfn).collect

++++res3:+Array[Int]+=+Array(10,20)|142615|(请注意，此处选项的行为非常类似于包含一个元素或零个元素的列表)|142616|entityMap^0|5|B|0|0|0|3|3|0|0|0|2|7|0|0|0|0|0|5|7|0|7|3|E|5|1E|I|0|0|X|6|1A|7|1M|4|1Y|4|0|0|0^^$0|@$1|2|3|4|5|6|7|1G|8|@$9|1H|A|1I|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|1J|8|@]|D|@]|E|$]]|$1|H|3|I|5|J|7|1K|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|1L|8|@$9|1M|A|1N|B|C]]|D|@]|E|$]]|$1|O|3|P|5|6|7|1O|8|@]|D|@]|E|$]]|$1|Q|3|R|5|J|7|1P|8|@]|D|@]|E|$K|L]]|$1|S|3|T|5|6|7|1Q|8|@$9|1R|A|1S|B|C]]|D|@]|E|$]]|$1|U|3|V|5|J|7|1T|8|@]|D|@]|E|$K|L]]|$1|W|3|X|5|6|7|1U|8|@]|D|@]|E|$]]|$1|Y|3|Z|5|6|7|1V|8|@]|D|@]|E|$]]|$1|10|3|11|5|J|7|1W|8|@]|D|@]|E|$K|L]]|$1|12|3|13|5|6|7|1X|8|@$9|1Y|A|1Z|B|C]]|D|@]|E|$]]|$1|14|3|15|5|6|7|20|8|@$9|21|A|22|B|C]|$9|23|A|24|B|C]|$9|25|A|26|B|C]]|D|@]|E|$]]|$1|16|3|17|5|J|7|27|8|@]|D|@]|E|$K|L]]|$1|18|3|19|5|6|7|28|8|@$9|29|A|2A|B|C]|$9|2B|A|2C|B|C]|$9|2D|A|2E|B|C]|$9|2F|A|2G|B|C]]|D|@]|E|$]]|$1|1A|3|1B|5|J|7|2H|8|@]|D|@]|E|$K|L]]|$1|1C|3|1D|5|6|7|2I|8|@]|D|@]|E|$]]|$1|1E|3|-4|5|6|7|2J|8|@]|D|@]|E|$]]]|1F|$]]

Here is an example of the difference, as a <code>spark-shell</code> session:

First, some data - two lines of text:

<pre class="lang-scala prettyprint-override"><code>val rdd = sc.parallelize(Seq("Roses are red", "Violets are blue")) // lines

rdd.collect

 res0: Array[String] = Array("Roses are red", "Violets are blue")
</code></pre>

Now, <code>map</code> transforms an RDD of length N into another RDD of length N.

For example, it maps from two lines into two line-lengths:

<pre class="lang-scala prettyprint-override"><code>rdd.map(_.length).collect

 res1: Array[Int] = Array(13, 16)
</code></pre>

But <code>flatMap</code> (loosely speaking) transforms an RDD of length N into a collection of N collections, then flattens these into a single RDD of results. 

<pre class="lang-scala prettyprint-override"><code>rdd.flatMap(_.split(" ")).collect

 res2: Array[String] = Array("Roses", "are", "red", "Violets", "are", "blue")
</code></pre>

We have multiple words per line, and multiple lines, but we end up with a single output array of words

Just to illustrate that, flatMapping from a collection of lines to a collection of words looks like:

<pre><code>["aa bb cc", "", "dd"] =&gt; [["aa","bb","cc"],[],["dd"]] =&gt; ["aa","bb","cc","dd"]
</code></pre>

The input and output RDDs will therefore typically be of different sizes for <code>flatMap</code>.

If we had tried to use <code>map</code> with our <code>split</code> function, we'd have ended up with nested structures (an RDD of arrays of words, with type <code>RDD[Array[String]]</code>) because we have to have exactly one result per input:

<pre class="lang-scala prettyprint-override"><code>rdd.map(_.split(" ")).collect

 res3: Array[Array[String]] = Array(
 Array(Roses, are, red), 
 Array(Violets, are, blue)
 )
</code></pre>

Finally, one useful special case is mapping with a function which might not return an answer, and so returns an <code>Option</code>. We can use <code>flatMap</code> to filter out the elements that return <code>None</code> and extract the values from those that return a <code>Some</code>:

<pre class="lang-scala prettyprint-override"><code>val rdd = sc.parallelize(Seq(1,2,3,4))

def myfn(x: Int): Option[Int] = if (x &lt;= 2) Some(x * 10) else None

rdd.flatMap(myfn).collect

 res3: Array[Int] = Array(10,20)
</code></pre>

(noting here that an Option behaves rather like a list that has either one element, or zero elements)

blocks|key|4649171|text|如果您正在询问Spark中RDD.map和RDD.flatMap之间的区别，map会将一个大小为N的RDD.map转换为另一个大小为N的RDD。例如：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4649172|myRDD.map(x+=>+x*2)|code-block|syntax|javascript|4649173|例如，如果myRDD由Double组成。|4649174|而flatMap可以将RDD转换为另一个不同大小的RDD：|4649175|myRDD.flatMap(x+=>new+Seq(2*x,3*x))|4649176|将返回大小为2*N的RDD或|4649177|myRDD.flatMap(x+=>if+x<10+new+Seq(2*x,3*x)+else+new+Seq(x)+)|4649178|entityMap^0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|T|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|U|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|V|8|@]|9|@]|A|$]]|$1|K|3|L|5|D|7|W|8|@]|9|@]|A|$E|F]]|$1|M|3|N|5|6|7|X|8|@]|9|@]|A|$]]|$1|O|3|P|5|D|7|Y|8|@]|9|@]|A|$E|F]]|$1|Q|3|-4|5|6|7|Z|8|@]|9|@]|A|$]]]|R|$]]

If you are asking the difference between RDD.map and RDD.flatMap in Spark, map transforms an RDD of size N to another one of size N . eg.

<pre><code>myRDD.map(x =&gt; x*2)
</code></pre>

for example, if myRDD is composed of Doubles .

While flatMap can transform the RDD into anther one of a different size: 
eg.:

<pre><code>myRDD.flatMap(x =&gt;new Seq(2*x,3*x))
</code></pre>

which will return an RDD of size 2*N
 or 

<pre><code>myRDD.flatMap(x =&gt;if x&lt;10 new Seq(2*x,3*x) else new Seq(x) )
</code></pre>

blocks|key|4649197|text|map和flatMap很相似，因为它们从输入RDD中提取一行并对其应用函数。它们的不同之处在于，map中的函数只返回一个元素，而flatMap中的函数可以返回一个元素列表(0个或更多个)作为迭代器。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4649198|此外，flatMap的输出也是展平的。尽管flatMap中的函数返回一个元素列表，但flatMap返回一个RDD，其中以平面方式(而不是列表)包含列表中的所有元素。|4649199|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|G|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|H|8|@]|9|@]|A|$]]]|E|$]]

map and flatMap are similar, in the sense they take a line from the input RDD and apply a function on it. The way they differ is that the function in map returns only one element, while function in flatMap can return a list of elements (0 or more) as an iterator.

Also, the output of the flatMap is flattened. Although the function in flatMap returns a list of elements, the flatMap returns an RDD which has all the elements from the list in a flat way (not a list).

blocks|key|142685|text|以test.md为例：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|142686|➜++spark-1.6.1+cat+test.md
This+is+the+first+line;
This+is+the+second+line;
This+is+the+last+line.

scala>+val+textFile+=+sc.textFile("test.md")
scala>+textFile.map(line+=>+line.split("+")).count()
res2:+Long+=+3

scala>+textFile.flatMap(line+=>+line.split("+")).count()
res3:+Long+=+15

scala>+textFile.map(line+=>+line.split("+")).collect()
res0:+Array[Array[String]]+=+Array(Array(This,+is,+the,+first,+line;),+Array(This,+is,+the,+second,+line;),+Array(This,+is,+the,+last,+line.))

scala>+textFile.flatMap(line+=>+line.split("+")).collect()
res1:+Array[String]+=+Array(This,+is,+the,+first,+line;,+This,+is,+the,+second,+line;,+This,+is,+the,+last,+line.)|code-block|syntax|javascript|142687|如果你使用map方法，你会得到test.md的行数，对于flatMap方法，你会得到单词的数量。|142688|map方法类似于flatMap，它们都返回一个新的RDD。map方法经常使用return一个新的RDD，flatMap方法经常使用分词。|142689|entityMap^0|1|7|0|0|5|3|F|7|S|7|0|0|3|8|7|T|3|1G|7|0^^$0|@$1|2|3|4|5|6|7|Q|8|@$9|R|A|S|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|T|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|U|8|@$9|V|A|W|B|C]|$9|X|A|Y|B|C]|$9|Z|A|10|B|C]]|D|@]|E|$]]|$1|M|3|N|5|6|7|11|8|@$9|12|A|13|B|C]|$9|14|A|15|B|C]|$9|16|A|17|B|C]|$9|18|A|19|B|C]]|D|@]|E|$]]|$1|O|3|-4|5|6|7|1A|8|@]|D|@]|E|$]]]|P|$]]

Use <code>test.md</code> as a example:

<pre><code>➜ spark-1.6.1 cat test.md
This is the first line;
This is the second line;
This is the last line.

scala&gt; val textFile = sc.textFile("test.md")
scala&gt; textFile.map(line =&gt; line.split(" ")).count()
res2: Long = 3

scala&gt; textFile.flatMap(line =&gt; line.split(" ")).count()
res3: Long = 15

scala&gt; textFile.map(line =&gt; line.split(" ")).collect()
res0: Array[Array[String]] = Array(Array(This, is, the, first, line;), Array(This, is, the, second, line;), Array(This, is, the, last, line.))

scala&gt; textFile.flatMap(line =&gt; line.split(" ")).collect()
res1: Array[String] = Array(This, is, the, first, line;, This, is, the, second, line;, This, is, the, last, line.)
</code></pre>

If you use <code>map</code> method, you will get the lines of <code>test.md</code>, for <code>flatMap</code> method, you will get the number of words.

The <code>map</code> method is similar to <code>flatMap</code>, they are all return a new RDD. <code>map</code> method often to use return a new RDD, <code>flatMap</code> method often to use split words.

blocks|key|4649278|text|map和flatMap的输出差异：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4649279|1.flatMap|offset|length|style|CODE|4649280|val+a+=+sc.parallelize(1+to+10,+5)

a.flatMap(1+to+_).collect()|code-block|syntax|javascript|4649281|输出：|4649282|+1,+1,+2,+1,+2,+3,+1,+2,+3,+4,+1,+2,+3,+4,+5,+1,+2,+3,+4,+5,+6,+1,+2,+3,+4,+5,+6,+7,+1,+2,+3,+4,+5,+6,+7,+8,+1,+2,+3,+4,+5,+6,+7,+8,+9,+1,+2,+3,+4,+5,+6,+7,+8,+9,+10|4649283|2.map|4649284|val+a+=+sc.parallelize(List("dog",+"salmon",+"salmon",+"rat",+"elephant"),+3)

val+b+=+a.map(_.length).collect()|4649285|4649286|3+6+6+3+8|4649287|entityMap^0|0|2|7|0|0|0|0|2|3|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Z|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|10|8|@$D|11|E|12|F|G]]|9|@]|A|$]]|$1|H|3|I|5|J|7|13|8|@]|9|@]|A|$K|L]]|$1|M|3|N|5|6|7|14|8|@]|9|@]|A|$]]|$1|O|3|P|5|J|7|15|8|@]|9|@]|A|$K|L]]|$1|Q|3|R|5|6|7|16|8|@$D|17|E|18|F|G]]|9|@]|A|$]]|$1|S|3|T|5|J|7|19|8|@]|9|@]|A|$K|L]]|$1|U|3|N|5|6|7|1A|8|@]|9|@]|A|$]]|$1|V|3|W|5|J|7|1B|8|@]|9|@]|A|$K|L]]|$1|X|3|-4|5|6|7|1C|8|@]|9|@]|A|$]]]|Y|$]]

Difference in output of map and flatMap:

1.<code>flatMap</code>

<pre><code>val a = sc.parallelize(1 to 10, 5)

a.flatMap(1 to _).collect()
</code></pre>

Output:

<pre><code> 1, 1, 2, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6, 7, 1, 2, 3, 4, 5, 6, 7, 8, 1, 2, 3, 4, 5, 6, 7, 8, 9, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
</code></pre>

2.<code>map</code>:

<pre><code>val a = sc.parallelize(List("dog", "salmon", "salmon", "rat", "elephant"), 3)

val b = a.map(_.length).collect()
</code></pre>

Output:

<pre><code>3 6 6 3 8
</code></pre>

blocks|key|142761|text|差异可以从以下示例pyspark代码中看出：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|142762|rdd+=+sc.parallelize([2,+3,+4])
rdd.flatMap(lambda+x:+range(1,+x)).collect()
Output:
[1,+1,+2,+1,+2,+3]


rdd.map(lambda+x:+range(1,+x)).collect()
Output:
[[1],+[1,+2],+[1,+2,+3]]|code-block|syntax|javascript|142763|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

The difference can be seen from below sample pyspark code:

<pre><code>rdd = sc.parallelize([2, 3, 4])
rdd.flatMap(lambda x: range(1, x)).collect()
Output:
[1, 1, 2, 1, 2, 3]


rdd.map(lambda x: range(1, x)).collect()
Output:
[[1], [1, 2], [1, 2, 3]]
</code></pre>

blocks|key|4649351|text|RDD.map返回单个数组中的所有元素|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|4649352|RDD.flatMap返回数组的数组中的元素|4649353|让我们假设text.txt文件中的文本为|4649354|Spark+is+an+expressive+framework
This+text+is+to+understand+map+and+faltMap+functions+of+Spark+RDD|code-block|syntax|javascript|4649355|使用map|4649356|val+text=sc.textFile("text.txt").map(_.split("+")).collect|4649357|输出：|4649358|text:+**Array[Array[String]]**+=+Array(Array(Spark,+is,+an,+expressive,+framework),+Array(This,+text,+is,+to,+understand,+map,+and,+faltMap,+functions,+of,+Spark,+RDD))|4649359|使用flatMap|4649360|val+text=sc.textFile("text.txt").flatMap(_.split("+")).collect|4649361|4649362|+text:+**Array[String]**+=+Array(Spark,+is,+an,+expressive,+framework,+This,+text,+is,+to,+understand,+map,+and,+faltMap,+functions,+of,+Spark,+RDD)|4649363|entityMap^0|0|7|0|0|B|0|0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|15|8|@$9|16|A|17|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|18|8|@$9|19|A|1A|B|C]]|D|@]|E|$]]|$1|H|3|I|5|6|7|1B|8|@]|D|@]|E|$]]|$1|J|3|K|5|L|7|1C|8|@]|D|@]|E|$M|N]]|$1|O|3|P|5|6|7|1D|8|@]|D|@]|E|$]]|$1|Q|3|R|5|L|7|1E|8|@]|D|@]|E|$M|N]]|$1|S|3|T|5|6|7|1F|8|@]|D|@]|E|$]]|$1|U|3|V|5|L|7|1G|8|@]|D|@]|E|$M|N]]|$1|W|3|X|5|6|7|1H|8|@]|D|@]|E|$]]|$1|Y|3|Z|5|L|7|1I|8|@]|D|@]|E|$M|N]]|$1|10|3|T|5|6|7|1J|8|@]|D|@]|E|$]]|$1|11|3|12|5|L|7|1K|8|@]|D|@]|E|$M|N]]|$1|13|3|-4|5|6|7|1L|8|@]|D|@]|E|$]]]|14|$]]

<code>RDD.map</code> returns all elements in single array

<code>RDD.flatMap</code> returns elements in Arrays of array

let's assume we have text in text.txt file as

<pre><code>Spark is an expressive framework
This text is to understand map and faltMap functions of Spark RDD
</code></pre>

Using map

<pre><code>val text=sc.textFile("text.txt").map(_.split(" ")).collect
</code></pre>

output:

<pre><code>text: **Array[Array[String]]** = Array(Array(Spark, is, an, expressive, framework), Array(This, text, is, to, understand, map, and, faltMap, functions, of, Spark, RDD))
</code></pre>

Using flatMap 

<pre><code>val text=sc.textFile("text.txt").flatMap(_.split(" ")).collect
</code></pre>

output:

<pre><code> text: **Array[String]** = Array(Spark, is, an, expressive, framework, This, text, is, to, understand, map, and, faltMap, functions, of, Spark, RDD)
</code></pre>

blocks|key|142868|text|映射：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|142869|是一个更高阶的方法，它接受一个函数作为输入，并将其应用于源RDD中的每个元素。|142870|http://commandstech.com/difference-between-map-and-flatmap-in-spark-what-is-map-and-flatmap-with-examples/|offset|length|142871|flatMap：|142872|接受输入函数的高阶方法和转换操作。|142873|entityMap|0|LINK|mutability|MUTABLE|url^0|0|0|0|2Y|0|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|T|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|U|8|@]|9|@$F|V|G|W|1|X]]|A|$]]|$1|H|3|I|5|6|7|Y|8|@]|9|@]|A|$]]|$1|J|3|K|5|6|7|Z|8|@]|9|@]|A|$]]|$1|L|3|-4|5|6|7|10|8|@]|9|@]|A|$]]]|M|$N|$5|O|P|Q|A|$R|E]]]]

map :

is a higher-order method that takes a function as input and applies it to each element in the source RDD.

<a href="http://commandstech.com/difference-between-map-and-flatmap-in-spark-what-is-map-and-flatmap-with-examples/" rel="nofollow noreferrer">http://commandstech.com/difference-between-map-and-flatmap-in-spark-what-is-map-and-flatmap-with-examples/</a>

flatMap:

a higher-order method and transformation operation that takes an input function.

Can someone explain to me the difference between map and flatMap and what is a good use case for each?

What does "flatten the results" mean?
What is it good for?

What is the difference between map and flatMap and a good use case for each?

Spark 

谁能给我解释一下map和flatMap之间的区别，以及它们最好的用例是什么？“扁平化结果”是什么意思？它有什么好处？

问map和flatMap之间的区别是什么，它们的一个很好的用例是什么？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问map和flatMap之间的区别是什么，它们的一个很好的用例是什么？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问map和flatMap之间的区别是什么，它们的一个很好的用例是什么？
EN