blocks|key|4407399|text|from+datetime+import+datetime
from+pyspark.sql.functions+import+col,+udf
from+pyspark.sql.types+import+DateType



#+Creation+of+a+dummy+dataframe:
df1+=+sqlContext.createDataFrame([("11/25/1991","11/24/1991","11/30/1991"),+
++++++++++++++++++++++++++++("11/25/1391","11/24/1992","11/30/1992")],+schema=['first',+'second',+'third'])

#+Setting+an+user+define+function:
#+This+function+converts+the+string+cell+into+a+date:
func+=++udf+(lambda+x:+datetime.strptime(x,+'%25m/%25d/%25Y'),+DateType())

df+=+df1.withColumn('test',+func(col('first')))

df.show()

df.printSchema()|type|code-block|depth|inlineStyleRanges|entityRanges|data|syntax|javascript|4407400|下面是输出：|unstyled|4407401|%2B----------%2B----------%2B----------%2B----------%2B
%7C+++++first%7C++++second%7C+++++third%7C++++++test%7C
%2B----------%2B----------%2B----------%2B----------%2B
%7C11/25/1991%7C11/24/1991%7C11/30/1991%7C1991-01-25%7C
%7C11/25/1391%7C11/24/1992%7C11/30/1992%7C1391-01-17%7C
%2B----------%2B----------%2B----------%2B----------%2B

root
+%7C--+first:+string+(nullable+=+true)
+%7C--+second:+string+(nullable+=+true)
+%7C--+third:+string+(nullable+=+true)
+%7C--+test:+date+(nullable+=+true)|4407402|entityMap^0|0|0|0^^$0|@$1|2|3|4|5|6|7|K|8|@]|9|@]|A|$B|C]]|$1|D|3|E|5|F|7|L|8|@]|9|@]|A|$]]|$1|G|3|H|5|6|7|M|8|@]|9|@]|A|$B|C]]|$1|I|3|-4|5|F|7|N|8|@]|9|@]|A|$]]]|J|$]]

<pre class="lang-py prettyprint-override"><code>from datetime import datetime
from pyspark.sql.functions import col, udf
from pyspark.sql.types import DateType



# Creation of a dummy dataframe:
df1 = sqlContext.createDataFrame([("11/25/1991","11/24/1991","11/30/1991"), 
 ("11/25/1391","11/24/1992","11/30/1992")], schema=['first', 'second', 'third'])

# Setting an user define function:
# This function converts the string cell into a date:
func = udf (lambda x: datetime.strptime(x, '%m/%d/%Y'), DateType())

df = df1.withColumn('test', func(col('first')))

df.show()

df.printSchema()
</code></pre>

Here is the output:

<pre><code>+----------+----------+----------+----------+
| first| second| third| test|
+----------+----------+----------+----------+
|11/25/1991|11/24/1991|11/30/1991|1991-01-25|
|11/25/1391|11/24/1992|11/30/1992|1391-01-17|
+----------+----------+----------+----------+

root
 |-- first: string (nullable = true)
 |-- second: string (nullable = true)
 |-- third: string (nullable = true)
 |-- test: date (nullable = true)
</code></pre>

blocks|key|4407571|text|strptime()方法对我不起作用。我得到了另一个更干净的解决方案，使用cast：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4407572|from+pyspark.sql.types+import+DateType
spark_df1+=+spark_df.withColumn("record_date",spark_df['order_submitted_date'].cast(DateType()))
#below+is+the+result
spark_df1.select('order_submitted_date','record_date').show(10,False)

%2B---------------------%2B-----------%2B
%7Corder_submitted_date+%7Crecord_date%7C
%2B---------------------%2B-----------%2B
%7C2015-08-19+12:54:16.0%7C2015-08-19+%7C
%7C2016-04-14+13:55:50.0%7C2016-04-14+%7C
%7C2013-10-11+18:23:36.0%7C2013-10-11+%7C
%7C2015-08-19+20:18:55.0%7C2015-08-19+%7C
%7C2015-08-20+12:07:40.0%7C2015-08-20+%7C
%7C2013-10-11+21:24:12.0%7C2013-10-11+%7C
%7C2013-10-11+23:29:28.0%7C2013-10-11+%7C
%7C2015-08-20+16:59:35.0%7C2015-08-20+%7C
%7C2015-08-20+17:32:03.0%7C2015-08-20+%7C
%7C2016-04-13+16:56:21.0%7C2016-04-13+%7C|code-block|syntax|javascript|4407573|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

The strptime() approach does not work for me. I get another cleaner solution, using cast:

<pre><code>from pyspark.sql.types import DateType
spark_df1 = spark_df.withColumn("record_date",spark_df['order_submitted_date'].cast(DateType()))
#below is the result
spark_df1.select('order_submitted_date','record_date').show(10,False)

+---------------------+-----------+
|order_submitted_date |record_date|
+---------------------+-----------+
|2015-08-19 12:54:16.0|2015-08-19 |
|2016-04-14 13:55:50.0|2016-04-14 |
|2013-10-11 18:23:36.0|2013-10-11 |
|2015-08-19 20:18:55.0|2015-08-19 |
|2015-08-20 12:07:40.0|2015-08-20 |
|2013-10-11 21:24:12.0|2013-10-11 |
|2013-10-11 23:29:28.0|2013-10-11 |
|2015-08-20 16:59:35.0|2015-08-20 |
|2015-08-20 17:32:03.0|2015-08-20 |
|2016-04-13 16:56:21.0|2016-04-13 |
</code></pre>

blocks|key|4407579|text|试试这个：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4407580|df+=+spark.createDataFrame([('2018-07-27+10:30:00',)],+['Date_col'])
df.select(from_unixtime(unix_timestamp(df.Date_col,+'yyyy-MM-dd+HH:mm:ss')).alias('dt_col'))
df.show()
%2B-------------------%2B++
%7C+++++++++++Date_col%7C++
%2B-------------------%2B++
%7C2018-07-27+10:30:00%7C++
%2B-------------------%2B++|code-block|syntax|javascript|4407581|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Try this:

<pre><code>df = spark.createDataFrame([('2018-07-27 10:30:00',)], ['Date_col'])
df.select(from_unixtime(unix_timestamp(df.Date_col, 'yyyy-MM-dd HH:mm:ss')).alias('dt_col'))
df.show()
+-------------------+ 
| Date_col| 
+-------------------+ 
|2018-07-27 10:30:00| 
+-------------------+ 
</code></pre>

blocks|key|4407458|text|在接受答案的更新中，您看不到to_date函数的示例，因此使用它的另一个解决方案是：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|4407459|from+pyspark.sql+import+functions+as+F

df+=+df.withColumn(
++++++++++++'new_date',
++++++++++++++++F.to_date(
++++++++++++++++++++F.unix_timestamp('STRINGCOLUMN',+'MM-dd-yyyy').cast('timestamp')))|code-block|syntax|javascript|4407460|entityMap^0|E|7|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@$9|N|A|O|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|P|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|Q|8|@]|D|@]|E|$]]]|L|$]]

In the accepted answer's update you don't see the example for the <code>to_date</code> function, so another solution using it would be:

<pre><code>from pyspark.sql import functions as F

df = df.withColumn(
 'new_date',
 F.to_date(
 F.unix_timestamp('STRINGCOLUMN', 'MM-dd-yyyy').cast('timestamp')))
</code></pre>

blocks|key|4407654|text|可能没有太多的答案，所以我想分享我的代码，这可以帮助别人|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4407655|from+pyspark.sql+import+SparkSession
from+pyspark.sql.functions+import+to_date

spark+=+SparkSession.builder.appName("Python+Spark+SQL+basic+example")\
++++.config("spark.some.config.option",+"some-value").getOrCreate()


df+=+spark.createDataFrame([('2019-06-22',)],+['t'])
df1+=+df.select(to_date(df.t,+'yyyy-MM-dd').alias('dt'))
print+df1
print+df1.show()|code-block|syntax|javascript|4407656|输出|4407657|DataFrame[dt:+date]
%2B----------%2B
%7C++++++++dt%7C
%2B----------%2B
%7C2019-06-22%7C
%2B----------%2B|4407658|上面的代码转换为日期，如果你想转换日期时间，那么使用to_timestamp。如果你有任何疑问，请告诉我。|4407659|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|R|8|@]|9|@]|A|$E|F]]|$1|K|3|L|5|6|7|S|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|N|$]]

possibly not so many answers so thinking to share my code which can help someone

<pre><code>from pyspark.sql import SparkSession
from pyspark.sql.functions import to_date

spark = SparkSession.builder.appName("Python Spark SQL basic example")\
 .config("spark.some.config.option", "some-value").getOrCreate()


df = spark.createDataFrame([('2019-06-22',)], ['t'])
df1 = df.select(to_date(df.t, 'yyyy-MM-dd').alias('dt'))
print df1
print df1.show()
</code></pre>

output

<pre><code>DataFrame[dt: date]
+----------+
| dt|
+----------+
|2019-06-22|
+----------+
</code></pre>

the above code to convert to date if you want to convert datetime then use to_timestamp.
let me know if you have any doubt.

I have a date pyspark dataframe with a string column in the format of <code>MM-dd-yyyy</code> and I am attempting to convert this into a date column. 

I tried: 

<code>df.select(to_date(df.STRING_COLUMN).alias('new_date')).show()</code>

and I get a string of nulls. Can anyone help?

Convert pyspark string to date format

Python

Spark 

我有一个日期pyspark dataframe，其中有一个MM-dd-yyyy格式的字符串列，我正在尝试将其转换为日期列。我试过了：df.select(to_date(df.STRING_COLUMN).alias('new_date')).show()然后我得到一串空值。有人能帮上忙吗？

问将pyspark字符串转换为日期格式
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问将pyspark字符串转换为日期格式EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问将pyspark字符串转换为日期格式
EN