blocks|key|32837|text|这个link展示了如何以编程方式创建一个带模式的数据框架。您可以将数据保存在单独的特征中，并将其与您的测试混合。例如,|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|32838|//+This+example+assumes+CSV+data.+But+same+approach+should+work+for+other+formats+as+well.

trait+TestData+{
++val+data1+=+List(
++++"this,is,valid,data",
++++"this,is,in-valid,data",
++)
++val+data2+=+...++
}|code-block|syntax|javascript|32839|然后使用ScalaTest，我们可以做这样的事情。|32840|class+MyDFTest+extends+FlatSpec+with+Matchers+{

++"method"+should+"perform+this"+in+new+TestData+{
+++++//+You+can+access+your+test+data+here.+Use+it+to+create+the+DataFrame.
+++++//+Your+test+here.
++}
}|32841|要创建DataFrame，您可以使用几个如下所示的工具方法。|32842|++def+schema(types:+Array[String],+cols:+Array[String])+=+{
++++val+datatypes+=+types.map+{
++++++case+"String"+=>+StringType
++++++case+"Long"+=>+LongType
++++++case+"Double"+=>+DoubleType
++++++//+Add+more+types+here+based+on+your+data.
++++++case+_+=>+StringType
++++}
++++StructType(cols.indices.map(x+=>+StructField(cols(x),+datatypes(x))).toArray)
++}

++def+df(data:+List[String],+types:+Array[String],+cols:+Array[String])+=+{
++++val+rdd+=+sc.parallelize(data)
++++val+parser+=+new+CSVParser(',')
++++val+split+=+rdd.map(line+=>+parser.parseLine(line))
++++val+rdd+=+split.map(arr+=>+Row(arr(0),+arr(1),+arr(2),+arr(3)))
++++sqlContext.createDataFrame(rdd,+schema(types,+cols))
++}|32843|我不知道有什么实用程序类用于检查DataFrame中的特定值。但我认为使用DataFrame+API编写一个应用程序应该很简单。|32844|entityMap|0|LINK|mutability|MUTABLE|url|http://spark.apache.org/docs/latest/sql-getting-started.html#programmatically-specifying-the-schema^0|2|4|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|10|8|@]|9|@$A|11|B|12|1|13]]|C|$]]|$1|D|3|E|5|F|7|14|8|@]|9|@]|C|$G|H]]|$1|I|3|J|5|6|7|15|8|@]|9|@]|C|$]]|$1|K|3|L|5|F|7|16|8|@]|9|@]|C|$G|H]]|$1|M|3|N|5|6|7|17|8|@]|9|@]|C|$]]|$1|O|3|P|5|F|7|18|8|@]|9|@]|C|$G|H]]|$1|Q|3|R|5|6|7|19|8|@]|9|@]|C|$]]|$1|S|3|-4|5|6|7|1A|8|@]|9|@]|C|$]]]|T|$U|$5|V|W|X|C|$Y|Z]]]]

This <a href="http://spark.apache.org/docs/latest/sql-getting-started.html#programmatically-specifying-the-schema" rel="nofollow noreferrer">link</a> shows how we can programmatically create a data frame with schema. You can keep the data in separate traits and mix it in with your tests. For instance,

<pre><code>// This example assumes CSV data. But same approach should work for other formats as well.

trait TestData {
 val data1 = List(
 "this,is,valid,data",
 "this,is,in-valid,data",
 )
 val data2 = ... 
}
</code></pre>

Then with ScalaTest, we can do something like this.

<pre><code>class MyDFTest extends FlatSpec with Matchers {

 "method" should "perform this" in new TestData {
 // You can access your test data here. Use it to create the DataFrame.
 // Your test here.
 }
}
</code></pre>

To create the DataFrame, you can have few util methods like below.

<pre><code> def schema(types: Array[String], cols: Array[String]) = {
 val datatypes = types.map {
 case "String" =&gt; StringType
 case "Long" =&gt; LongType
 case "Double" =&gt; DoubleType
 // Add more types here based on your data.
 case _ =&gt; StringType
 }
 StructType(cols.indices.map(x =&gt; StructField(cols(x), datatypes(x))).toArray)
 }

 def df(data: List[String], types: Array[String], cols: Array[String]) = {
 val rdd = sc.parallelize(data)
 val parser = new CSVParser(',')
 val split = rdd.map(line =&gt; parser.parseLine(line))
 val rdd = split.map(arr =&gt; Row(arr(0), arr(1), arr(2), arr(3)))
 sqlContext.createDataFrame(rdd, schema(types, cols))
 }
</code></pre>

I am not aware of any utility classes for checking specific values in a DataFrame. But I think it should be simple to write one using the DataFrame APIs.

blocks|key|32846|text|对于那些希望在Java语言中实现类似功能的人，可以通过使用以下项目在单元测试中初始化SparkContext来开始使用：https://github.com/holdenk/spark-testing-base|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|32847|我个人不得不模仿一些AVRO文件的文件结构。因此，我使用Avro-tools+(https://avro.apache.org/docs/1.8.2/gettingstartedjava.html#download_install)通过以下命令从我的二进制记录中提取模式：|32848|java+-jar+$AVRO_HOME/avro+tojson+largeAvroFile.avro+%7C+head+-3|code-block|syntax|javascript|32849|然后，使用这个小帮助器方法，您可以将输出JSON转换为在单元测试中使用的DataFrame。|32850|private+DataFrame+getDataFrameFromList()+{
++++SQLContext+sqlContext+=+new+SQLContext(jsc());
++++ImmutableList<String>+elements+=+ImmutableList.of(
++++++++{"header":{"appId":"myAppId1","clientIp":"10.22.63.3","createdDate":"2017-05-10T02:09:59.984Z"}}
++++++++{"header":{"appId":"myAppId1","clientIp":"11.22.63.3","createdDate":"2017-05-11T02:09:59.984Z"}}
++++++++{"header":{"appId":"myAppId1","clientIp":"12.22.63.3","createdDate":"2017-05-11T02:09:59.984Z"}}
++++);
++++JavaRDD<String>+parallelize+=+jsc().parallelize(elements);
++++return+sqlContext.read().json(parallelize);
}|32851|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/holdenk/spark-testing-base|1|https://avro.apache.org/docs/1.8.2/gettingstartedjava.html#download_install^0|1O|19|0|0|14|23|1|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Y|8|@]|9|@$A|Z|B|10|1|11]]|C|$]]|$1|D|3|E|5|6|7|12|8|@]|9|@$A|13|B|14|1|15]]|C|$]]|$1|F|3|G|5|H|7|16|8|@]|9|@]|C|$I|J]]|$1|K|3|L|5|6|7|17|8|@]|9|@]|C|$]]|$1|M|3|N|5|H|7|18|8|@]|9|@]|C|$I|J]]|$1|O|3|-4|5|6|7|19|8|@]|9|@]|C|$]]]|P|$Q|$5|R|S|T|C|$U|V]]|W|$5|R|S|T|C|$U|X]]]]

For those looking to achieve something similar in Java, you can use start by using this project to initialize a SparkContext within your unit tests: <a href="https://github.com/holdenk/spark-testing-base" rel="nofollow noreferrer">https://github.com/holdenk/spark-testing-base</a> 

I personally had to mimick the file structure of some AVRO files. So I used Avro-tools (<a href="https://avro.apache.org/docs/1.8.2/gettingstartedjava.html#download_install" rel="nofollow noreferrer">https://avro.apache.org/docs/1.8.2/gettingstartedjava.html#download_install</a>) to extract the schema from my binary records using the following command:

<pre><code>java -jar $AVRO_HOME/avro tojson largeAvroFile.avro | head -3
</code></pre>

Then, using this small helper method, you can convert the output JSON into a DataFrame to use in your unit tests.

<pre><code>private DataFrame getDataFrameFromList() {
 SQLContext sqlContext = new SQLContext(jsc());
 ImmutableList&lt;String&gt; elements = ImmutableList.of(
 {"header":{"appId":"myAppId1","clientIp":"10.22.63.3","createdDate":"2017-05-10T02:09:59.984Z"}}
 {"header":{"appId":"myAppId1","clientIp":"11.22.63.3","createdDate":"2017-05-11T02:09:59.984Z"}}
 {"header":{"appId":"myAppId1","clientIp":"12.22.63.3","createdDate":"2017-05-11T02:09:59.984Z"}}
 );
 JavaRDD&lt;String&gt; parallelize = jsc().parallelize(elements);
 return sqlContext.read().json(parallelize);
}
</code></pre>

blocks|key|2706850|text|您可以使用Spark用于自己的单元测试的SharedSQLContext和SharedSparkSession。有关示例，请查看我的answer。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2706851|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/apache/spark/blob/v1.6.3/sql/core/src/test/scala/org/apache/spark/sql/test/SharedSQLContext.scala|1|https://stackoverflow.com/a/50581218/8442366|2^0|K|G|11|I|K|G|0|11|I|1|1U|6|2|0^^$0|@$1|2|3|4|5|6|7|Q|8|@$9|R|A|S|B|C]|$9|T|A|U|B|C]]|D|@$9|V|A|W|1|X]|$9|Y|A|Z|1|10]|$9|11|A|12|1|13]]|E|$]]|$1|F|3|-4|5|6|7|14|8|@]|D|@]|E|$]]]|G|$H|$5|I|J|K|E|$L|M]]|N|$5|I|J|K|E|$L|O]]|P|$5|I|J|K|E|$L|O]]]]

You could use <a href="https://github.com/apache/spark/blob/v1.6.3/sql/core/src/test/scala/org/apache/spark/sql/test/SharedSQLContext.scala" rel="nofollow noreferrer"><code>SharedSQLContext</code></a> and <a href="https://stackoverflow.com/a/50581218/8442366"><code>SharedSparkSession</code></a> that Spark uses for its own unit tests. Check my <a href="https://stackoverflow.com/a/50581218/8442366">answer</a> for examples.

I'm trying to test a part of my program which performs transformations on dataframes
I want to test several different variations of these dataframe which rules out the option of reading a specific DF from a file

And so my questions are:

<ol>
<li>Is there any good tutorial on how to perform unit testing with Spark and dataframes, especially regarding the dataframes creation?</li>
<li>How can I create these different several lines dataframes without a lot of boilerplate and without reading these from a file?</li>
<li>Are there any utility classes for checking for specific values inside a dataframe?</li>
</ol>

I obviously googled that before but could not find anything which was very useful. Among the more useful links I found were:

<ul>
<li><a href="https://stackoverflow.com/questions/32004775/spark-unit-testing-with-dataframe-collect-return-empty-array">Running a basic unit test with a dataframe</a></li>
<li><a href="https://codecov.io/github/holdenk/spark-testing-base/src/main/scala/com/holdenkarau/spark/testing/DataFrameSuiteBase.scala?ref=56a1cb7f267a36e6b522075d6a84824ed41109a0" rel="noreferrer">Custom made assertions with DF</a></li>
</ul>

It would be great if examples/tutorials are in Scala but I'll take whatever language you've got

Thanks in advance

Unit testing with Spark dataframes

单元测试

Spark 

JSON

Java

我正在尝试测试我的程序中对数据帧执行转换的部分，我想测试这些数据帧的几种不同变体，这些变体排除了从文件中读取特定DF的选项所以我的问题是：是否有关于如何使用Spark和dataframe执行单元测试的好教程，特别是关于创建dataframe的教程？如何创建这些不同的几行数据帧，而不使用很多样板，并且不从文件中读取这些数...

问使用Spark数据帧进行单元测试
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问使用Spark数据帧进行单元测试EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问使用Spark数据帧进行单元测试
EN