blocks|key|402338|text|Spark+2.0.0%2B：|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|402339|在Spark2.0.0中，UserDefinedType已经成为私有的，到目前为止，它还没有Dataset友好的替代品。|CODE|402340|请参阅：SPARK-14155+(Hide+UserDefinedType+in+Spark+2.0)|402341|大多数时候，静态类型的Dataset可以作为替代，有一个悬而未决的Jira+SPARK-7768可以使UDT+API在目标版本2.4中再次公开。|402342|另请参阅How+to+store+custom+objects+in+Dataset?|402343|火花<+2.0.0|402344|402345|是否有可能为某些类型(这里是某些类型)添加或定义模式？|blockquote|402346|402347|402348|我想答案取决于你有多需要它。看起来可以创建一个UserDefinedType，但它需要访问DeveloperApi，而且并不是很简单，也没有很好的文档。|402349|import+org.apache.spark.sql.types._

@SQLUserDefinedType(udt+=+classOf[SomeUDT])
sealed+trait+Some
case+object+AType+extends+Some
case+object+BType+extends+Some

class+SomeUDT+extends+UserDefinedType[Some]+{
++override+def+sqlType:+DataType+=+IntegerType

++override+def+serialize(obj:+Any)+=+{
++++obj+match+{
++++++case+AType+=>+0
++++++case+BType+=>+1
++++}
++}

++override+def+deserialize(datum:+Any):+Some+=+{
++++datum+match+{
++++++case+0+=>+AType
++++++case+1+=>+BType
++++}
++}

++override+def+userClass:+Class[Some]+=+classOf[Some]
}|code-block|syntax|javascript|402350|您可能还应该覆盖hashCode和equals。|402351|它的PySpark对等项可能如下所示：|402352|from+enum+import+Enum,+unique
from+pyspark.sql.types+import+UserDefinedType,+IntegerType

class+SomeUDT(UserDefinedType):
++++@classmethod
++++def+sqlType(self):
++++++++return+IntegerType()

++++@classmethod
++++def+module(cls):
++++++++return+cls.__module__

++++@classmethod+
++++def+scalaUDT(cls):+#+Required+in+Spark+<+1.5
++++++++return+'net.zero323.enum.SomeUDT'

++++def+serialize(self,+obj):
++++++++return+obj.value

++++def+deserialize(self,+datum):
++++++++return+{x.value:+x+for+x+in+Some}[datum]

@unique
class+Some(Enum):
++++__UDT__+=+SomeUDT()
++++AType+=+0
++++BType+=+1|402353|在Spark+<+1.5中，Python+UDT需要成对的Scala+UDT，但在1.5中看起来不再是这样了。|402354|对于像这样的简单UDT，您可以使用简单类型(例如，IntegerType而不是整个Struct)。|402355|entityMap|0|LINK|mutability|MUTABLE|url|https://issues.apache.org/jira/browse/SPARK-14155|1|https://issues.apache.org/jira/browse/SPARK-7768|2|https://stackoverflow.com/q/36648128/1560062^0|0|C|0|D|F|1A|7|0|4|1B|0|0|B|7|12|A|1|0|4|13|2|0|0|9|0|0|0|0|0|N|F|19|C|0|0|8|8|H|6|0|0|0|0|P|B|15|6|0^^$0|@$1|2|3|4|5|6|7|1P|8|@$9|1Q|A|1R|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|1S|8|@$9|1T|A|1U|B|H]|$9|1V|A|1W|B|H]]|D|@]|E|$]]|$1|I|3|J|5|6|7|1X|8|@]|D|@$9|1Y|A|1Z|1|20]]|E|$]]|$1|K|3|L|5|6|7|21|8|@$9|22|A|23|B|H]]|D|@$9|24|A|25|1|26]]|E|$]]|$1|M|3|N|5|6|7|27|8|@]|D|@$9|28|A|29|1|2A]]|E|$]]|$1|O|3|P|5|6|7|2B|8|@$9|2C|A|2D|B|C]]|D|@]|E|$]]|$1|Q|3|-4|5|6|7|2E|8|@]|D|@]|E|$]]|$1|R|3|S|5|T|7|2F|8|@]|D|@]|E|$]]|$1|U|3|-4|5|6|7|2G|8|@]|D|@]|E|$]]|$1|V|3|-4|5|6|7|2H|8|@]|D|@]|E|$]]|$1|W|3|X|5|6|7|2I|8|@$9|2J|A|2K|B|H]|$9|2L|A|2M|B|H]]|D|@]|E|$]]|$1|Y|3|Z|5|10|7|2N|8|@]|D|@]|E|$11|12]]|$1|13|3|14|5|6|7|2O|8|@$9|2P|A|2Q|B|H]|$9|2R|A|2S|B|H]]|D|@]|E|$]]|$1|15|3|16|5|6|7|2T|8|@]|D|@]|E|$]]|$1|17|3|18|5|10|7|2U|8|@]|D|@]|E|$11|12]]|$1|19|3|1A|5|6|7|2V|8|@]|D|@]|E|$]]|$1|1B|3|1C|5|6|7|2W|8|@$9|2X|A|2Y|B|H]|$9|2Z|A|30|B|H]]|D|@]|E|$]]|$1|1D|3|-4|5|6|7|31|8|@]|D|@]|E|$]]]|1E|$1F|$5|1G|1H|1I|E|$1J|1K]]|1L|$5|1G|1H|1I|E|$1J|1M]]|1N|$5|1G|1H|1I|E|$1J|1O]]]]

Spark 2.0.0+: 

<code>UserDefinedType</code> has been made private in Spark 2.0.0 and as for now it has no <code>Dataset</code> friendly replacement.

See: <a href="https://issues.apache.org/jira/browse/SPARK-14155" rel="noreferrer">SPARK-14155 (Hide UserDefinedType in Spark 2.0)</a>

Most of the time statically typed <code>Dataset</code> can serve as replacement 
There is a pending Jira <a href="https://issues.apache.org/jira/browse/SPARK-7768" rel="noreferrer">SPARK-7768</a> to make UDT API public again with target version 2.4.

See also <a href="https://stackoverflow.com/q/36648128/1560062">How to store custom objects in Dataset?</a>

Spark &lt; 2.0.0

<blockquote>
 Is there a possibility to add or define a schema for certain types (here type Some)?
</blockquote>

I guess the answer depends on how badly you need this. It looks like it is possible to create an <code>UserDefinedType</code> but it requires access to <code>DeveloperApi</code> and is not exactly straightforward or well documented.

<pre class="lang-scala prettyprint-override"><code>import org.apache.spark.sql.types._

@SQLUserDefinedType(udt = classOf[SomeUDT])
sealed trait Some
case object AType extends Some
case object BType extends Some

class SomeUDT extends UserDefinedType[Some] {
 override def sqlType: DataType = IntegerType

 override def serialize(obj: Any) = {
 obj match {
 case AType =&gt; 0
 case BType =&gt; 1
 }
 }

 override def deserialize(datum: Any): Some = {
 datum match {
 case 0 =&gt; AType
 case 1 =&gt; BType
 }
 }

 override def userClass: Class[Some] = classOf[Some]
}
</code></pre>

You should probably override <code>hashCode</code> and <code>equals</code> as well.

Its PySpark counterpart can look like this:

<pre class="lang-py prettyprint-override"><code>from enum import Enum, unique
from pyspark.sql.types import UserDefinedType, IntegerType

class SomeUDT(UserDefinedType):
 @classmethod
 def sqlType(self):
 return IntegerType()

 @classmethod
 def module(cls):
 return cls.__module__

 @classmethod 
 def scalaUDT(cls): # Required in Spark &lt; 1.5
 return 'net.zero323.enum.SomeUDT'

 def serialize(self, obj):
 return obj.value

 def deserialize(self, datum):
 return {x.value: x for x in Some}[datum]

@unique
class Some(Enum):
 __UDT__ = SomeUDT()
 AType = 0
 BType = 1
</code></pre>

In Spark &lt; 1.5 Python UDT requires a paired Scala UDT, but it look like it is no longer the case in 1.5.

For a simple UDT like you can use simple types (for example <code>IntegerType</code> instead of whole <code>Struct</code>).

The following example code tries to put some case objects into a dataframe. The code includes the definition of a case object hierarchy and a case class using this trait:

<pre><code>import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.sql.SQLContext

sealed trait Some
case object AType extends Some
case object BType extends Some

case class Data( name : String, t: Some)

object Example {
 def main(args: Array[String]) : Unit = {
 val conf = new SparkConf()
 .setAppName( "Example" )
 .setMaster( "local[*]")

 val sc = new SparkContext(conf)
 val sqlContext = new SQLContext(sc)

 import sqlContext.implicits._

 val df = sc.parallelize( Seq( Data( "a", AType), Data( "b", BType) ), 4).toDF()
 df.show()
 }
} 
</code></pre>

When executing the code, I unfortunately encounter the following exception:

<pre><code>java.lang.UnsupportedOperationException: Schema for type Some is not supported
</code></pre>

<h2>Questions</h2>

<ul>
<li>Is there a possibility to add or define a schema for certain types (here type <code>Some</code>)?</li>
<li>Does another approach exist to represent this kind of enumerations? 

<ul>
<li>I tried to use <code>Enumeration</code> directly, but also without success. (see below)</li>
</ul></li>
</ul>

Code for <code>Enumeration</code>:

<pre><code>object Some extends Enumeration {
 type Some = Value
 val AType, BType = Value
}
</code></pre>

Thanks in advance. I hope, that the best approach is not to use strings instead.

How to define schema for custom type in Spark SQL?

Python

Spark 

下面的示例代码尝试将一些case对象放入dataframe中。代码包括使用以下特征的case对象层次结构和case类的定义：import org.apache.spark.{SparkContext, SparkConf}import org.apache.spark.sql.SQLContextsealed trai...

问如何在Spark SQL中定义自定义类型的模式？
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在Spark SQL中定义自定义类型的模式？EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何在Spark SQL中定义自定义类型的模式？
EN