blocks|key|21991|text|将RDD转换为双精度的RDD，然后使用.histogram(10)操作。请参阅DoubleRDD+ScalaDoc|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|21992|entityMap|0|LINK|mutability|MUTABLE|url|https://spark.apache.org/docs/1.1.0/api/scala/index.html#org.apache.spark.rdd.DoubleRDDFunctions^0|J|E|13|I|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@$9|O|A|P|B|C]]|D|@$9|Q|A|R|1|S]]|E|$]]|$1|F|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|G|$H|$5|I|J|K|E|$L|M]]]]

Convert you RDD into a RDD of Double, and then use the <code>.histogram(10)</code> action. See <a href="https://spark.apache.org/docs/1.1.0/api/scala/index.html#org.apache.spark.rdd.DoubleRDDFunctions" rel="nofollow">DoubleRDD ScalaDoc</a>

blocks|key|4529624|text|另一种替代方法是在double的RDD上使用top和last。例如，val+percentile_99th_value=scores.top((count/100).toInt).last|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4529625|这种方法更适合于单个百分位数。|4529626|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|G|8|@]|9|@]|A|$]]|$1|D|3|-4|5|6|7|H|8|@]|9|@]|A|$]]]|E|$]]

Another alternative way can be to use top and last on RDD of double. For example, val percentile_99th_value=scores.top((count/100).toInt).last

This method is more suited for individual percentiles.

blocks|key|4529653|text|我发现了这个要点|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4529654|https://gist.github.com/felixcheung/92ae74bc349ea83a9e29|offset|length|4529655|它包含以下函数：|4529656|++/**
+++*+compute+percentile+from+an+unsorted+Spark+RDD
+++*+@param+data:+input+data+set+of+Long+integers
+++*+@param+tile:+percentile+to+compute+(eg.+85+percentile)
+++*+@return+value+of+input+data+at+the+specified+percentile
+++*/
++def+computePercentile(data:+RDD[Long],+tile:+Double):+Double+=+{
++++//+NIST+method;+data+to+be+sorted+in+ascending+order
++++val+r+=+data.sortBy(x+=>+x)
++++val+c+=+r.count()
++++if+(c+==+1)+r.first()
++++else+{
++++++val+n+=+(tile+/+100d)+*+(c+%2B+1d)
++++++val+k+=+math.floor(n).toLong
++++++val+d+=+n+-+k
++++++if+(k+<=+0)+r.first()
++++++else+{
++++++++val+index+=+r.zipWithIndex().map(_.swap)
++++++++val+last+=+c
++++++++if+(k+>=+c)+{
++++++++++index.lookup(last+-+1).head
++++++++}+else+{
++++++++++index.lookup(k+-+1).head+%2B+d+*+(index.lookup(k).head+-+index.lookup(k+-+1).head)
++++++++}
++++++}
++++}
++}|code-block|syntax|javascript|4529657|entityMap|0|LINK|mutability|MUTABLE|url^0|0|0|1K|0|0|0|0^^$0|@$1|2|3|4|5|6|7|T|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|U|8|@]|9|@$D|V|E|W|1|X]]|A|$]]|$1|F|3|G|5|6|7|Y|8|@]|9|@]|A|$]]|$1|H|3|I|5|J|7|Z|8|@]|9|@]|A|$K|L]]|$1|M|3|-4|5|6|7|10|8|@]|9|@]|A|$]]]|N|$O|$5|P|Q|R|A|$S|C]]]]

I discovered this gist 

<a href="https://gist.github.com/felixcheung/92ae74bc349ea83a9e29" rel="nofollow">https://gist.github.com/felixcheung/92ae74bc349ea83a9e29</a>

that contains the following function:

<pre><code> /**
 * compute percentile from an unsorted Spark RDD
 * @param data: input data set of Long integers
 * @param tile: percentile to compute (eg. 85 percentile)
 * @return value of input data at the specified percentile
 */
 def computePercentile(data: RDD[Long], tile: Double): Double = {
 // NIST method; data to be sorted in ascending order
 val r = data.sortBy(x =&gt; x)
 val c = r.count()
 if (c == 1) r.first()
 else {
 val n = (tile / 100d) * (c + 1d)
 val k = math.floor(n).toLong
 val d = n - k
 if (k &lt;= 0) r.first()
 else {
 val index = r.zipWithIndex().map(_.swap)
 val last = c
 if (k &gt;= c) {
 index.lookup(last - 1).head
 } else {
 index.lookup(k - 1).head + d * (index.lookup(k).head - index.lookup(k - 1).head)
 }
 }
 }
 }
</code></pre>

blocks|key|22089|text|下面是我在Spark上的Python实现，用于计算包含感兴趣的值的RDD的百分位数。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|22090|def+percentile_threshold(ardd,+percentile):
++++assert+percentile+>+0+and+percentile+<=+100,+"percentile+should+be+larger+then+0+and+smaller+or+equal+to+100"

++++return+ardd.sortBy(lambda+x:+x).zipWithIndex().map(lambda+x:+(x[1],+x[0]))+\
++++++++++++.lookup(np.ceil(ardd.count()+/+100+*+percentile+-+1))[0]

#+Now+test+it+out
import+numpy+as+np
randlist+=+range(1,10001)
np.random.shuffle(randlist)
ardd+=+sc.parallelize(randlist)

print+percentile_threshold(ardd,0.001)
print+percentile_threshold(ardd,1)
print+percentile_threshold(ardd,60.11)
print+percentile_threshold(ardd,99)
print+percentile_threshold(ardd,99.999)
print+percentile_threshold(ardd,100)

#+output:
#+1
#+100
#+6011
#+9900
#+10000
#+10000|code-block|syntax|javascript|22091|另外，我定义了下面的函数来获得第10到100个百分位数。|22092|def+get_percentiles(rdd,+stepsize=10):
++++percentiles+=+[]
++++rddcount100+=+rdd.count()+/+100+
++++sortedrdd+=+ardd.sortBy(lambda+x:+x).zipWithIndex().map(lambda+x:+(x[1],+x[0]))


++++for+p+in+range(0,+101,+stepsize):
++++++++if+p+==+0:
++++++++++++pass
++++++++++++#+I+am+not+aware+of+a+formal+definition+of+0+percentile,+
++++++++++++#+you+can+put+a+place+holder+like+this+if+you+want
++++++++++++#+percentiles.append(sortedrdd.lookup(0)[0]+-+1)+
++++++++elif+p+==+100:
++++++++++++percentiles.append(sortedrdd.lookup(np.ceil(rddcount100+*+100+-+1))[0])
++++++++else:
++++++++++++pv+=+sortedrdd.lookup(np.ceil(rddcount100+*+p)+-+1)[0]
++++++++++++percentiles.append(pv)

++++return+percentiles

randlist+=+range(1,10001)
np.random.shuffle(randlist)
ardd+=+sc.parallelize(randlist)
get_percentiles(ardd,+10)

#+[1000,+2000,+3000,+4000,+5000,+6000,+7000,+8000,+9000,+10000]|22093|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

Here is my Python implementation on Spark for calculating the percentile for a RDD containing values of interest. 

<pre><code>def percentile_threshold(ardd, percentile):
 assert percentile &gt; 0 and percentile &lt;= 100, "percentile should be larger then 0 and smaller or equal to 100"

 return ardd.sortBy(lambda x: x).zipWithIndex().map(lambda x: (x[1], x[0])) \
 .lookup(np.ceil(ardd.count() / 100 * percentile - 1))[0]

# Now test it out
import numpy as np
randlist = range(1,10001)
np.random.shuffle(randlist)
ardd = sc.parallelize(randlist)

print percentile_threshold(ardd,0.001)
print percentile_threshold(ardd,1)
print percentile_threshold(ardd,60.11)
print percentile_threshold(ardd,99)
print percentile_threshold(ardd,99.999)
print percentile_threshold(ardd,100)

# output:
# 1
# 100
# 6011
# 9900
# 10000
# 10000
</code></pre>

Separately, I defined the following function to get the 10th to 100th percentile. 

<pre><code>def get_percentiles(rdd, stepsize=10):
 percentiles = []
 rddcount100 = rdd.count() / 100 
 sortedrdd = ardd.sortBy(lambda x: x).zipWithIndex().map(lambda x: (x[1], x[0]))


 for p in range(0, 101, stepsize):
 if p == 0:
 pass
 # I am not aware of a formal definition of 0 percentile, 
 # you can put a place holder like this if you want
 # percentiles.append(sortedrdd.lookup(0)[0] - 1) 
 elif p == 100:
 percentiles.append(sortedrdd.lookup(np.ceil(rddcount100 * 100 - 1))[0])
 else:
 pv = sortedrdd.lookup(np.ceil(rddcount100 * p) - 1)[0]
 percentiles.append(pv)

 return percentiles

randlist = range(1,10001)
np.random.shuffle(randlist)
ardd = sc.parallelize(randlist)
get_percentiles(ardd, 10)

# [1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000]
</code></pre>

blocks|key|22111|text|基于这里给出的答案，我使用了Median+UDAF+in+Spark/Scala来计算spark窗口的百分位数(spark+2.1)：|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|22112|首先是用于其他聚合的抽象通用UDAF|22113|import+org.apache.spark.sql.Row
import+org.apache.spark.sql.expressions.{MutableAggregationBuffer,+UserDefinedAggregateFunction}
import+org.apache.spark.sql.types._

import+scala.collection.mutable
import+scala.collection.mutable.ArrayBuffer


abstract+class+GenericUDAF+extends+UserDefinedAggregateFunction+{

++def+inputSchema:+StructType+=
++++StructType(StructField("value",+DoubleType)+::+Nil)

++def+bufferSchema:+StructType+=+StructType(
++++StructField("window_list",+ArrayType(DoubleType,+false))+::+Nil
++)

++def+deterministic:+Boolean+=+true

++def+initialize(buffer:+MutableAggregationBuffer):+Unit+=+{
++++buffer(0)+=+new+ArrayBuffer[Double]()
++}

++def+update(buffer:+MutableAggregationBuffer,input:+org.apache.spark.sql.Row):+Unit+=+{
++++var+bufferVal+=+buffer.getAs[mutable.WrappedArray[Double]](0).toBuffer
++++bufferVal%2B=input.getAs[Double](0)
++++buffer(0)+=+bufferVal
++}

++def+merge(buffer1:+MutableAggregationBuffer,+buffer2:+org.apache.spark.sql.Row):+Unit+=+{
++++buffer1(0)+=+buffer1.getAs[ArrayBuffer[Double]](0)+%2B%2B+buffer2.getAs[ArrayBuffer[Double]](0)
++}

++def+dataType:+DataType
++def+evaluate(buffer:+Row):+Any

}|code-block|syntax|javascript|22114|然后是为十进制定制的百分位数UDAF：|22115|import+org.apache.spark.sql.Row
import+org.apache.spark.sql.expressions.{MutableAggregationBuffer,+UserDefinedAggregateFunction}
import+org.apache.spark.sql.types._

import+scala.collection.mutable
import+scala.collection.mutable.ArrayBuffer


class+DecilesUDAF+extends+GenericUDAF+{

++override+def+dataType:+DataType+=+ArrayType(DoubleType,+false)

++override+def+evaluate(buffer:+Row):+Any+=+{
++++val+sortedWindow+=+buffer.getAs[mutable.WrappedArray[Double]](0).sorted.toBuffer
++++val+windowSize+=+sortedWindow.size
++++if+(windowSize+==+0)+return+null
++++if+(windowSize+==+1)+return+(0+to+10).map(_+=>+sortedWindow.head).toArray

++++(0+to+10).map(i+=>+sortedWindow(Math.min(windowSize-1,+i*windowSize/10))).toArray

++}
}|22116|然后，UDAF被实例化，并在分区和有序的窗口上调用：|22117|val+deciles+=+new+DecilesUDAF()
df.withColumn("mt_deciles",+deciles(col("mt")).over(myWindow))|22118|然后，您可以使用getItem将结果数组拆分为多列：|22119|def+splitToColumns(size:+Int,+splitCol:String)(df:+DataFrame)+=+{
++(0+to+size).foldLeft(df)+{
++++case+(df_arg,+i)+=>+df_arg.withColumn("mt_decile_"%2Bi,+col(splitCol).getItem(i))
++}
}

df.transform(splitToColumns(10,+"mt_deciles"+))|22120|UDAF比原生spark函数慢，但只要每个分组的包或每个窗口相对较小，并且适合单个执行器，就应该没问题。它的主要优点是使用了spark并行。只需很少的工作，就可以将此代码扩展到n分位数。|22121|我使用这个函数测试了代码：|22122|def+testDecilesUDAF+=+{
++++val+window+=+W.partitionBy("user")
++++val+deciles+=+new+DecilesUDAF()

++++val+schema+=+StructType(StructField("mt",+DoubleType)+::+StructField("user",+StringType)+::+Nil)

++++val+rows1+=+(1+to+20).map(i+=>+Row(i.toDouble,+"a"))
++++val+rows2+=+(21+to+40).map(i+=>+Row(i.toDouble,+"b"))

++++val+df+=+spark.createDataFrame(spark.sparkContext.makeRDD[Row](rows1%2B%2Brows2),+schema)

++++df.withColumn("deciles",+deciles(col("mt")).over(window))
++++++.transform(splitToColumns(10,+"deciles"+))
++++++.drop("deciles")
++++++.show(100,+truncate=false)
++}|22123|输出的前3行：|22124|%2B----%2B----%2B-----------%2B-----------%2B-----------%2B-----------%2B-----------%2B-----------%2B-----------%2B-----------%2B-----------%2B-----------%2B------------%2B
%7Cmt++%7Cuser%7Cmt_decile_0%7Cmt_decile_1%7Cmt_decile_2%7Cmt_decile_3%7Cmt_decile_4%7Cmt_decile_5%7Cmt_decile_6%7Cmt_decile_7%7Cmt_decile_8%7Cmt_decile_9%7Cmt_decile_10%7C
%2B----%2B----%2B-----------%2B-----------%2B-----------%2B-----------%2B-----------%2B-----------%2B-----------%2B-----------%2B-----------%2B-----------%2B------------%2B
%7C21.0%7Cb+++%7C21.0+++++++%7C23.0+++++++%7C25.0+++++++%7C27.0+++++++%7C29.0+++++++%7C31.0+++++++%7C33.0+++++++%7C35.0+++++++%7C37.0+++++++%7C39.0+++++++%7C40.0++++++++%7C
%7C22.0%7Cb+++%7C21.0+++++++%7C23.0+++++++%7C25.0+++++++%7C27.0+++++++%7C29.0+++++++%7C31.0+++++++%7C33.0+++++++%7C35.0+++++++%7C37.0+++++++%7C39.0+++++++%7C40.0++++++++%7C
%7C23.0%7Cb+++%7C21.0+++++++%7C23.0+++++++%7C25.0+++++++%7C27.0+++++++%7C29.0+++++++%7C31.0+++++++%7C33.0+++++++%7C35.0+++++++%7C37.0+++++++%7C39.0+++++++%7C40.0++++++++%7C|22125|entityMap|0|LINK|mutability|MUTABLE|url|https://stackoverflow.com/questions/44061553/how-to-calculate-moving-median-in-dataframe^0|E|Q|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|1E|8|@]|9|@$A|1F|B|1G|1|1H]]|C|$]]|$1|D|3|E|5|6|7|1I|8|@]|9|@]|C|$]]|$1|F|3|G|5|H|7|1J|8|@]|9|@]|C|$I|J]]|$1|K|3|L|5|6|7|1K|8|@]|9|@]|C|$]]|$1|M|3|N|5|H|7|1L|8|@]|9|@]|C|$I|J]]|$1|O|3|P|5|6|7|1M|8|@]|9|@]|C|$]]|$1|Q|3|R|5|H|7|1N|8|@]|9|@]|C|$I|J]]|$1|S|3|T|5|6|7|1O|8|@]|9|@]|C|$]]|$1|U|3|V|5|H|7|1P|8|@]|9|@]|C|$I|J]]|$1|W|3|X|5|6|7|1Q|8|@]|9|@]|C|$]]|$1|Y|3|Z|5|6|7|1R|8|@]|9|@]|C|$]]|$1|10|3|11|5|H|7|1S|8|@]|9|@]|C|$I|J]]|$1|12|3|13|5|6|7|1T|8|@]|9|@]|C|$]]|$1|14|3|15|5|H|7|1U|8|@]|9|@]|C|$I|J]]|$1|16|3|-4|5|6|7|1V|8|@]|9|@]|C|$]]]|17|$18|$5|19|1A|1B|C|$1C|1D]]]]

Based on the answer given here <a href="https://stackoverflow.com/questions/44061553/how-to-calculate-moving-median-in-dataframe">Median UDAF in Spark/Scala</a>, I used an UDAF to compute percentiles over spark windows (spark 2.1) :

First an abstract generic UDAF used for other aggregations

<pre><code>import org.apache.spark.sql.Row
import org.apache.spark.sql.expressions.{MutableAggregationBuffer, UserDefinedAggregateFunction}
import org.apache.spark.sql.types._

import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer


abstract class GenericUDAF extends UserDefinedAggregateFunction {

 def inputSchema: StructType =
 StructType(StructField("value", DoubleType) :: Nil)

 def bufferSchema: StructType = StructType(
 StructField("window_list", ArrayType(DoubleType, false)) :: Nil
 )

 def deterministic: Boolean = true

 def initialize(buffer: MutableAggregationBuffer): Unit = {
 buffer(0) = new ArrayBuffer[Double]()
 }

 def update(buffer: MutableAggregationBuffer,input: org.apache.spark.sql.Row): Unit = {
 var bufferVal = buffer.getAs[mutable.WrappedArray[Double]](0).toBuffer
 bufferVal+=input.getAs[Double](0)
 buffer(0) = bufferVal
 }

 def merge(buffer1: MutableAggregationBuffer, buffer2: org.apache.spark.sql.Row): Unit = {
 buffer1(0) = buffer1.getAs[ArrayBuffer[Double]](0) ++ buffer2.getAs[ArrayBuffer[Double]](0)
 }

 def dataType: DataType
 def evaluate(buffer: Row): Any

}
</code></pre>

Then the Percentile UDAF customized for deciles :

<pre><code>import org.apache.spark.sql.Row
import org.apache.spark.sql.expressions.{MutableAggregationBuffer, UserDefinedAggregateFunction}
import org.apache.spark.sql.types._

import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer


class DecilesUDAF extends GenericUDAF {

 override def dataType: DataType = ArrayType(DoubleType, false)

 override def evaluate(buffer: Row): Any = {
 val sortedWindow = buffer.getAs[mutable.WrappedArray[Double]](0).sorted.toBuffer
 val windowSize = sortedWindow.size
 if (windowSize == 0) return null
 if (windowSize == 1) return (0 to 10).map(_ =&gt; sortedWindow.head).toArray

 (0 to 10).map(i =&gt; sortedWindow(Math.min(windowSize-1, i*windowSize/10))).toArray

 }
}
</code></pre>

The UDAF is then instanciated and called over a partitionned and ordered window :

<pre><code>val deciles = new DecilesUDAF()
df.withColumn("mt_deciles", deciles(col("mt")).over(myWindow))
</code></pre>

You can then split the resulting array into multiple columns with getItem :

<pre><code>def splitToColumns(size: Int, splitCol:String)(df: DataFrame) = {
 (0 to size).foldLeft(df) {
 case (df_arg, i) =&gt; df_arg.withColumn("mt_decile_"+i, col(splitCol).getItem(i))
 }
}

df.transform(splitToColumns(10, "mt_deciles" ))
</code></pre>

The UDAF is slower than native spark functions but as long as each grouped bag or each window is relatively small and fits into a single executor, it should be fine. The main advantage is using spark parallelism.
With little effort, this code could be extend to n-quantiles.

I tested the code using this function :

<pre><code>def testDecilesUDAF = {
 val window = W.partitionBy("user")
 val deciles = new DecilesUDAF()

 val schema = StructType(StructField("mt", DoubleType) :: StructField("user", StringType) :: Nil)

 val rows1 = (1 to 20).map(i =&gt; Row(i.toDouble, "a"))
 val rows2 = (21 to 40).map(i =&gt; Row(i.toDouble, "b"))

 val df = spark.createDataFrame(spark.sparkContext.makeRDD[Row](rows1++rows2), schema)

 df.withColumn("deciles", deciles(col("mt")).over(window))
 .transform(splitToColumns(10, "deciles" ))
 .drop("deciles")
 .show(100, truncate=false)
 }
</code></pre>

First 3 lines of output : 

<pre><code>+----+----+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+------------+
|mt |user|mt_decile_0|mt_decile_1|mt_decile_2|mt_decile_3|mt_decile_4|mt_decile_5|mt_decile_6|mt_decile_7|mt_decile_8|mt_decile_9|mt_decile_10|
+----+----+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+-----------+------------+
|21.0|b |21.0 |23.0 |25.0 |27.0 |29.0 |31.0 |33.0 |35.0 |37.0 |39.0 |40.0 |
|22.0|b |21.0 |23.0 |25.0 |27.0 |29.0 |31.0 |33.0 |35.0 |37.0 |39.0 |40.0 |
|23.0|b |21.0 |23.0 |25.0 |27.0 |29.0 |31.0 |33.0 |35.0 |37.0 |39.0 |40.0 |
</code></pre>

blocks|key|4529800|text|下面是我的简单方法：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4529801|val+percentiles+=+Array(0.1,+0.2,+0.3,+0.4,+0.5,+0.6,+0.7,+0.8,+0.9,+1)
val+accuracy+=+1000000
df.stat.approxQuantile("score",+percentiles,+1.0/accuracy)|code-block|syntax|javascript|4529802|输出：|4529803|scala>+df.stat.approxQuantile("score",+percentiles,+1.0/accuracy)
res88:+Array[Double]+=+Array(0.011044141836464405,+0.02022990956902504,+0.0317261666059494,+0.04638145491480827,+0.06498630344867706,+0.0892181545495987,+0.12161539494991302,+0.16825592517852783,+0.24740923941135406,+0.9188197255134583)|4529804|精度:精度参数(默认值:+10000)是一个正数字文字，它控制近似精度，但要以内存为代价。精度值越高，精度越高，1.0/精度是近似的相对误差。|4529805|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|R|8|@]|9|@]|A|$E|F]]|$1|K|3|L|5|6|7|S|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|N|$]]

Here is my easy approach: 

<pre><code>val percentiles = Array(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1)
val accuracy = 1000000
df.stat.approxQuantile("score", percentiles, 1.0/accuracy)
</code></pre>

output:

<pre><code>scala&gt; df.stat.approxQuantile("score", percentiles, 1.0/accuracy)
res88: Array[Double] = Array(0.011044141836464405, 0.02022990956902504, 0.0317261666059494, 0.04638145491480827, 0.06498630344867706, 0.0892181545495987, 0.12161539494991302, 0.16825592517852783, 0.24740923941135406, 0.9188197255134583)
</code></pre>

accuracy: The accuracy parameter (default: 10000) is a positive numeric literal which controls approximation accuracy at the cost of memory. Higher value of accuracy yields better accuracy, 1.0/accuracy is the relative error of the approximation.

I have an rdd of integers (i.e. <code>RDD[Int]</code>) and what I would like to do is to compute the following ten percentiles: <code>[0th, 10th, 20th, ..., 90th, 100th]</code>. What is the most efficient way to do that?

How to compute percentiles in Apache Spark

Python

Spark 

我有一个整数的rdd (即RDD[Int])，我想要做的是计算以下10个百分位数：[0th, 10th, 20th, ..., 90th, 100th]。做到这一点最有效的方法是什么？

问如何在Apache Spark中计算百分位数
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在Apache Spark中计算百分位数EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在Apache Spark中计算百分位数
EN