blocks|key|26116|text|尝试如下所示：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|26117|df.select([c+for+c+in+df.columns+if+c+in+['_2','_4','_5']]).show()|code-block|syntax|javascript|26118|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Try something like this:

<pre><code>df.select([c for c in df.columns if c in ['_2','_4','_5']]).show()
</code></pre>

blocks|key|24715|text|前两列和前5行|type|unstyled|depth|inlineStyleRanges|entityRanges|data|24716|+df.select(df.columns[:2]).take(5)|code-block|syntax|javascript|24717|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

First two columns and 5 rows

<pre><code> df.select(df.columns[:2]).take(5)
</code></pre>

blocks|key|24672|text|您可以使用一个数组，并在select中将其解包：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|24673|cols+=+['_2','_4','_5']
df.select(*cols).show()|code-block|syntax|javascript|24674|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

You can use an array and unpack it inside the select:

<pre><code>cols = ['_2','_4','_5']
df.select(*cols).show()
</code></pre>

blocks|key|26120|text|使用df.schema.names：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|26121|spark.version
#+u'2.2.0'

df+=+spark.createDataFrame([("foo",+1),+("bar",+2)])
df.show()
#+%2B---%2B---%2B+
#+%7C+_1%7C+_2%7C
#+%2B---%2B---%2B
#+%7Cfoo%7C++1%7C+
#+%7Cbar%7C++2%7C
#+%2B---%2B---%2B

df.schema.names
#+['_1',+'_2']

for+i+in+df.schema.names:
++#+df_new+=+df.withColumn(i,+[do-something])
++print+i
#+_1
#+_2|code-block|syntax|javascript|26122|entityMap^0|2|F|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@$9|N|A|O|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|P|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|Q|8|@]|D|@]|E|$]]]|L|$]]

Use <code>df.schema.names</code>: 



<pre class="lang-python prettyprint-override"><code>spark.version
# u'2.2.0'

df = spark.createDataFrame([("foo", 1), ("bar", 2)])
df.show()
# +---+---+ 
# | _1| _2|
# +---+---+
# |foo| 1| 
# |bar| 2|
# +---+---+

df.schema.names
# ['_1', '_2']

for i in df.schema.names:
 # df_new = df.withColumn(i, [do-something])
 print i
# _1
# _2
</code></pre>

blocks|key|24742|text|方法select接受列名(字符串)或表达式(列)的列表作为参数。要选择列，可以使用：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|24743|--列名(字符串)：|24744|df.select('col_1','col_2','col_3')|code-block|syntax|javascript|24745|--列对象：|24746|import+pyspark.sql.functions+as+F

df.select(F.col('col_1'),+F.col('col_2'),+F.col('col_3'))

#+or

df.select(df.col_1,+df.col_2,+df.col_3)

#+or

df.select(df['col_1'],+df['col_2'],+df['col_3'])|24747|--列名或列对象的列表：|24748|df.select(*['col_1','col_2','col_3'])

#or

df.select(*[F.col('col_1'),+F.col('col_2'),+F.col('col_3')])

#or+

df.select(*[df.col_1,+df.col_2,+df.col_3])|24749|星形运算符*可以省略，因为它用于保持与drop等不接受列表作为参数的其他函数的一致性。|24750|entityMap|0|LINK|mutability|MUTABLE|url|https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=select#pyspark.sql.DataFrame.select^0|2|6|2|6|0|0|0|0|0|0|0|0|5|1|J|4|0^^$0|@$1|2|3|4|5|6|7|14|8|@$9|15|A|16|B|C]]|D|@$9|17|A|18|1|19]]|E|$]]|$1|F|3|G|5|6|7|1A|8|@]|D|@]|E|$]]|$1|H|3|I|5|J|7|1B|8|@]|D|@]|E|$K|L]]|$1|M|3|N|5|6|7|1C|8|@]|D|@]|E|$]]|$1|O|3|P|5|J|7|1D|8|@]|D|@]|E|$K|L]]|$1|Q|3|R|5|6|7|1E|8|@]|D|@]|E|$]]|$1|S|3|T|5|J|7|1F|8|@]|D|@]|E|$K|L]]|$1|U|3|V|5|6|7|1G|8|@$9|1H|A|1I|B|C]|$9|1J|A|1K|B|C]]|D|@]|E|$]]|$1|W|3|-4|5|6|7|1L|8|@]|D|@]|E|$]]]|X|$Y|$5|Z|10|11|E|$12|13]]]]

The method <a href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=select#pyspark.sql.DataFrame.select" rel="noreferrer"><code>select</code></a> accepts a list of column names (string) or expressions (Column) as a parameter. To select columns you can use:
-- column names (strings):
<pre><code>df.select('col_1','col_2','col_3')
</code></pre>
-- column objects:
<pre><code>import pyspark.sql.functions as F

df.select(F.col('col_1'), F.col('col_2'), F.col('col_3'))

# or

df.select(df.col_1, df.col_2, df.col_3)

# or

df.select(df['col_1'], df['col_2'], df['col_3'])
</code></pre>
-- a list of column names or column objects:
<pre><code>df.select(*['col_1','col_2','col_3'])

#or

df.select(*[F.col('col_1'), F.col('col_2'), F.col('col_3')])

#or 

df.select(*[df.col_1, df.col_2, df.col_3])
</code></pre>
The star operator <code>*</code> can be omitted as it's used to keep it consistent with other functions like <code>drop</code> that don't accept a list as a parameter.

blocks|key|2481072|text|ss.csv中的dataset包含一些我感兴趣的列：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2481073|ss_+=+spark.read.csv("ss.csv",+header=+True,+
++++++++++++++++++++++inferSchema+=+True)
ss_.columns|code-block|syntax|javascript|2481074|['Reporting+Area',+'MMWR+Year',+'MMWR+Week',+'Salmonellosis+(excluding+Paratyphoid+fever+andTyphoid+fever)†,+Current+week',+'Salmonellosis+(excluding+Paratyphoid+fever+andTyphoid+fever)†,+Current+week,+flag',+'Salmonellosis+(excluding+Paratyphoid+fever+andTyphoid+fever)†,+Previous+52+weeks+Med',+'Salmonellosis+(excluding+Paratyphoid+fever+andTyphoid+fever)†,+Previous+52+weeks+Med,+flag',+'Salmonellosis+(excluding+Paratyphoid+fever+andTyphoid+fever)†,+Previous+52+weeks+Max',+'Salmonellosis+(excluding+Paratyphoid+fever+andTyphoid+fever)†,+Previous+52+weeks+Max,+flag',+'Salmonellosis+(excluding+Paratyphoid+fever+andTyphoid+fever)†,+Cum+2018',+'Salmonellosis+(excluding+Paratyphoid+fever+andTyphoid+fever)†,+Cum+2018,+flag',+'Salmonellosis+(excluding+Paratyphoid+fever+andTyphoid+fever)†,+Cum+2017',+'Salmonellosis+(excluding+Paratyphoid+fever+andTyphoid+fever)†,+Cum+2017,+flag',+'Shiga+toxin-producing+Escherichia+coli,+Current+week',+'Shiga+toxin-producing+Escherichia+coli,+Current+week,+flag',+'Shiga+toxin-producing+Escherichia+coli,+Previous+52+weeks+Med',+'Shiga+toxin-producing+Escherichia+coli,+Previous+52+weeks+Med,+flag',+'Shiga+toxin-producing+Escherichia+coli,+Previous+52+weeks+Max',+'Shiga+toxin-producing+Escherichia+coli,+Previous+52+weeks+Max,+flag',+'Shiga+toxin-producing+Escherichia+coli,+Cum+2018',+'Shiga+toxin-producing+Escherichia+coli,+Cum+2018,+flag',+'Shiga+toxin-producing+Escherichia+coli,+Cum+2017',+'Shiga+toxin-producing+Escherichia+coli,+Cum+2017,+flag',+'Shigellosis,+Current+week',+'Shigellosis,+Current+week,+flag',+'Shigellosis,+Previous+52+weeks+Med',+'Shigellosis,+Previous+52+weeks+Med,+flag',+'Shigellosis,+Previous+52+weeks+Max',+'Shigellosis,+Previous+52+weeks+Max,+flag',+'Shigellosis,+Cum+2018',+'Shigellosis,+Cum+2018,+flag',+'Shigellosis,+Cum+2017',+'Shigellosis,+Cum+2017,+flag']|2481075|但我只需要几个：|2481076|columns_lambda+=+lambda+k:+k.endswith(',+Current+week')+or+k+==+'Reporting+Area'+or+k+==+'MMWR+Year'+or++k+==+'MMWR+Week'|2481077|筛选器返回所需列的列表，list的计算结果为：|2481078|sss+=+filter(columns_lambda,+ss_.columns)
to_keep+=+list(sss)|2481079|所需列的列表被解包为dataframe+select函数的参数，该函数返回的数据集仅包含列表中的列：|2481080|dfss+=+ss_.select(*to_keep)
dfss.columns|2481081|结果是：|2481082|['Reporting+Area',
+'MMWR+Year',
+'MMWR+Week',
+'Salmonellosis+(excluding+Paratyphoid+fever+andTyphoid+fever)†,+Current+week',
+'Shiga+toxin-producing+Escherichia+coli,+Current+week',
+'Shigellosis,+Current+week']|2481083|df.select()有一个互补的对：http://spark.apache.org/docs/2.4.1/api/python/pyspark.sql.html#pyspark.sql.DataFrame.drop|2481084|若要删除列的列表，请执行以下操作。|2481085|entityMap|0|LINK|mutability|MUTABLE|url|http://spark.apache.org/docs/2.4.1/api/python/pyspark.sql.html#pyspark.sql.DataFrame.drop^0|0|6|0|0|0|0|0|0|0|0|0|0|0|0|B|J|2H|0|0|0^^$0|@$1|2|3|4|5|6|7|1E|8|@$9|1F|A|1G|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|1H|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|H|7|1I|8|@]|D|@]|E|$I|J]]|$1|M|3|N|5|6|7|1J|8|@]|D|@]|E|$]]|$1|O|3|P|5|H|7|1K|8|@]|D|@]|E|$I|J]]|$1|Q|3|R|5|6|7|1L|8|@]|D|@]|E|$]]|$1|S|3|T|5|H|7|1M|8|@]|D|@]|E|$I|J]]|$1|U|3|V|5|6|7|1N|8|@]|D|@]|E|$]]|$1|W|3|X|5|H|7|1O|8|@]|D|@]|E|$I|J]]|$1|Y|3|Z|5|6|7|1P|8|@]|D|@]|E|$]]|$1|10|3|11|5|H|7|1Q|8|@]|D|@]|E|$I|J]]|$1|12|3|13|5|6|7|1R|8|@$9|1S|A|1T|B|C]]|D|@$9|1U|A|1V|1|1W]]|E|$]]|$1|14|3|15|5|6|7|1X|8|@]|D|@]|E|$]]|$1|16|3|-4|5|6|7|1Y|8|@]|D|@]|E|$]]]|17|$18|$5|19|1A|1B|E|$1C|1D]]]]

The dataset in <code>ss.csv</code> contains some columns I am interested in:

<pre class="lang-py prettyprint-override"><code>ss_ = spark.read.csv("ss.csv", header= True, 
 inferSchema = True)
ss_.columns
</code></pre>

<pre><code>['Reporting Area', 'MMWR Year', 'MMWR Week', 'Salmonellosis (excluding Paratyphoid fever andTyphoid fever)†, Current week', 'Salmonellosis (excluding Paratyphoid fever andTyphoid fever)†, Current week, flag', 'Salmonellosis (excluding Paratyphoid fever andTyphoid fever)†, Previous 52 weeks Med', 'Salmonellosis (excluding Paratyphoid fever andTyphoid fever)†, Previous 52 weeks Med, flag', 'Salmonellosis (excluding Paratyphoid fever andTyphoid fever)†, Previous 52 weeks Max', 'Salmonellosis (excluding Paratyphoid fever andTyphoid fever)†, Previous 52 weeks Max, flag', 'Salmonellosis (excluding Paratyphoid fever andTyphoid fever)†, Cum 2018', 'Salmonellosis (excluding Paratyphoid fever andTyphoid fever)†, Cum 2018, flag', 'Salmonellosis (excluding Paratyphoid fever andTyphoid fever)†, Cum 2017', 'Salmonellosis (excluding Paratyphoid fever andTyphoid fever)†, Cum 2017, flag', 'Shiga toxin-producing Escherichia coli, Current week', 'Shiga toxin-producing Escherichia coli, Current week, flag', 'Shiga toxin-producing Escherichia coli, Previous 52 weeks Med', 'Shiga toxin-producing Escherichia coli, Previous 52 weeks Med, flag', 'Shiga toxin-producing Escherichia coli, Previous 52 weeks Max', 'Shiga toxin-producing Escherichia coli, Previous 52 weeks Max, flag', 'Shiga toxin-producing Escherichia coli, Cum 2018', 'Shiga toxin-producing Escherichia coli, Cum 2018, flag', 'Shiga toxin-producing Escherichia coli, Cum 2017', 'Shiga toxin-producing Escherichia coli, Cum 2017, flag', 'Shigellosis, Current week', 'Shigellosis, Current week, flag', 'Shigellosis, Previous 52 weeks Med', 'Shigellosis, Previous 52 weeks Med, flag', 'Shigellosis, Previous 52 weeks Max', 'Shigellosis, Previous 52 weeks Max, flag', 'Shigellosis, Cum 2018', 'Shigellosis, Cum 2018, flag', 'Shigellosis, Cum 2017', 'Shigellosis, Cum 2017, flag']

</code></pre>

but I only need a few:

<pre><code>columns_lambda = lambda k: k.endswith(', Current week') or k == 'Reporting Area' or k == 'MMWR Year' or k == 'MMWR Week'
</code></pre>

The filter returns the list of desired columns, list is evaluated:

<pre><code>sss = filter(columns_lambda, ss_.columns)
to_keep = list(sss)
</code></pre>

the list of desired columns is unpacked as arguments to dataframe select function that return dataset containing only columns in the list:

<pre class="lang-py prettyprint-override"><code>dfss = ss_.select(*to_keep)
dfss.columns
</code></pre>

The result:

<pre><code>['Reporting Area',
 'MMWR Year',
 'MMWR Week',
 'Salmonellosis (excluding Paratyphoid fever andTyphoid fever)†, Current week',
 'Shiga toxin-producing Escherichia coli, Current week',
 'Shigellosis, Current week']
</code></pre>

The <code>df.select()</code> has a complementary pair: <a href="http://spark.apache.org/docs/2.4.1/api/python/pyspark.sql.html#pyspark.sql.DataFrame.drop" rel="nofollow noreferrer">http://spark.apache.org/docs/2.4.1/api/python/pyspark.sql.html#pyspark.sql.DataFrame.drop</a>

to drop the list of columns.

I am looking for a way to select columns of my dataframe in PySpark. For the first row, I know I can use <code>df.first()</code>, but not sure about columns given that they do not have column names.
I have 5 columns and want to loop through each one of them.
<pre><code>+--+---+---+---+---+---+---+
|_1| _2| _3| _4| _5| _6| _7|
+--+---+---+---+---+---+---+
|1 |0.0|0.0|0.0|1.0|0.0|0.0|
|2 |1.0|0.0|0.0|0.0|0.0|0.0|
|3 |0.0|0.0|1.0|0.0|0.0|0.0|
</code></pre>

Select columns in PySpark dataframe

我正在寻找一种在PySpark中选择数据帧列的方法。对于第一行，我知道我可以使用df.first()，但不确定列，因为它们没有列名。我有5列，我想遍历每一列。+--+---+---+---+---+---+---+|_1| _2| _3| _4| _5| _6| _7|+--+---+---+---+---+---+---+|1 |0.0|0.0|0.0|1.0|0.0|0.0||2 |1.0|0

问选择PySpark数据框中的列
EN

回答 6

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问选择PySpark数据框中的列EN

回答 6

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问选择PySpark数据框中的列
EN