将pyspark字符串转换为日期格式
将pyspark字符串转换为日期格式
提问于 2016-06-28 23:46:00
我有一个日期pyspark dataframe，其中有一个MM-dd-yyyy格式的字符串列，我正在尝试将其转换为日期列。
我试过了：
df.select(to_date(df.STRING_COLUMN).alias('new_date')).show()
然后我得到一串空值。有人能帮上忙吗？
回答 5
Stack Overflow用户
发布于 2016-06-29 00:38:41
from datetime import datetime
from pyspark.sql.functions import col, udf
from pyspark.sql.types import DateType
# Creation of a dummy dataframe:
df1 = sqlContext.createDataFrame([("11/25/1991","11/24/1991","11/30/1991"),
("11/25/1391","11/24/1992","11/30/1992")], schema=['first', 'second', 'third'])
# Setting an user define function:
# This function converts the string cell into a date:
func = udf (lambda x: datetime.strptime(x, '%m/%d/%Y'), DateType())
df = df1.withColumn('test', func(col('first')))
df.show()
df.printSchema()下面是输出：
+----------+----------+----------+----------+
| first| second| third| test|
+----------+----------+----------+----------+
|11/25/1991|11/24/1991|11/30/1991|1991-01-25|
|11/25/1391|11/24/1992|11/30/1992|1391-01-17|
+----------+----------+----------+----------+
root
|-- first: string (nullable = true)
|-- second: string (nullable = true)
|-- third: string (nullable = true)
|-- test: date (nullable = true)Stack Overflow用户
发布于 2017-10-20 05:22:10
strptime()方法对我不起作用。我得到了另一个更干净的解决方案，使用cast：
from pyspark.sql.types import DateType
spark_df1 = spark_df.withColumn("record_date",spark_df['order_submitted_date'].cast(DateType()))
#below is the result
spark_df1.select('order_submitted_date','record_date').show(10,False)
+---------------------+-----------+
|order_submitted_date |record_date|
+---------------------+-----------+
|2015-08-19 12:54:16.0|2015-08-19 |
|2016-04-14 13:55:50.0|2016-04-14 |
|2013-10-11 18:23:36.0|2013-10-11 |
|2015-08-19 20:18:55.0|2015-08-19 |
|2015-08-20 12:07:40.0|2015-08-20 |
|2013-10-11 21:24:12.0|2013-10-11 |
|2013-10-11 23:29:28.0|2013-10-11 |
|2015-08-20 16:59:35.0|2015-08-20 |
|2015-08-20 17:32:03.0|2015-08-20 |
|2016-04-13 16:56:21.0|2016-04-13 |Stack Overflow用户
发布于 2019-03-22 19:42:01
在接受答案的更新中，您看不到to_date函数的示例，因此使用它的另一个解决方案是：
from pyspark.sql import functions as F
df = df.withColumn(
'new_date',
F.to_date(
F.unix_timestamp('STRINGCOLUMN', 'MM-dd-yyyy').cast('timestamp')))页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/38080748
复制相关文章
相似问题