blocks|key|2709629|text|最简单的方法就是：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2709630|df.groupBy().sum().collect()|code-block|syntax|javascript|2709631|但是这是非常慢的操作：避免groupByKey，您应该使用RDD和reduceByKey：|offset|length|2709632|df.rdd.map(lambda+x:+(1,x[1])).reduceByKey(lambda+x,y:+x+%2B+y).collect()[0][1]|2709633|我尝试了一个更大的数据集，并测量了处理时间：|2709634|RDD和ReduceByKey：2.23+s|style|BOLD|2709635|GroupByKey:+30.5秒|2709636|entityMap|0|LINK|mutability|MUTABLE|url|https://databricks.gitbooks.io/databricks-spark-knowledge-base/content/best_practices/prefer_reducebykey_over_groupbykey.html^0|0|0|B|C|0|0|0|0|G|6|0|0^^$0|@$1|2|3|4|5|6|7|12|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|13|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|14|8|@]|9|@$I|15|J|16|1|17]]|A|$]]|$1|K|3|L|5|D|7|18|8|@]|9|@]|A|$E|F]]|$1|M|3|N|5|6|7|19|8|@]|9|@]|A|$]]|$1|O|3|P|5|6|7|1A|8|@$I|1B|J|1C|Q|R]]|9|@]|A|$]]|$1|S|3|T|5|6|7|1D|8|@]|9|@]|A|$]]|$1|U|3|-4|5|6|7|1E|8|@]|9|@]|A|$]]]|V|$W|$5|X|Y|Z|A|$10|11]]]]

The simplest way really : 

<pre><code>df.groupBy().sum().collect()
</code></pre>

But it is very slow operation: <a href="https://databricks.gitbooks.io/databricks-spark-knowledge-base/content/best_practices/prefer_reducebykey_over_groupbykey.html" rel="noreferrer">Avoid groupByKey</a>, you should use RDD and reduceByKey: 

<pre><code>df.rdd.map(lambda x: (1,x[1])).reduceByKey(lambda x,y: x + y).collect()[0][1]
</code></pre>

I tried on a bigger dataset and i measured the processing time: 

RDD and ReduceByKey : 2.23 s

GroupByKey: 30.5 s

blocks|key|1075581|text|我认为最简单的方法是：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1075582|df.groupBy().sum().collect()|code-block|syntax|javascript|1075583|将返回一个列表。在你的例子中：|1075584|In+[9]:+df.groupBy().sum().collect()[0][0]
Out[9]:+130|1075585|entityMap^0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|N|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|O|8|@]|9|@]|A|$]]|$1|I|3|J|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|K|3|-4|5|6|7|Q|8|@]|9|@]|A|$]]]|L|$]]

I think the simplest way:

<pre><code>df.groupBy().sum().collect()
</code></pre>

will return a list.
In your example:

<pre><code>In [9]: df.groupBy().sum().collect()[0][0]
Out[9]: 130
</code></pre>

blocks|key|1075671|text|如果您想要一个特定的列：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1075672|import+pyspark.sql.functions+as+F+++++

df.agg(F.sum("my_column")).collect()[0][0]|code-block|syntax|javascript|1075673|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

If you want a specific column : 

<pre><code>import pyspark.sql.functions as F 

df.agg(F.sum("my_column")).collect()[0][0]
</code></pre>

blocks|key|542551|text|这是另一种你能做到的方法。使用agg和collect|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|542552|sum_number+=+df.agg({"Number":"sum"}).collect()[0]

result+=+sum_number["sum(Number)"]|code-block|syntax|javascript|542553|entityMap^0|F|3|J|7|0|0^^$0|@$1|2|3|4|5|6|7|M|8|@$9|N|A|O|B|C]|$9|P|A|Q|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|R|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|S|8|@]|D|@]|E|$]]]|L|$]]

This is another way you can do this. using <code>agg</code> and <code>collect</code>:

<pre><code>sum_number = df.agg({"Number":"sum"}).collect()[0]

result = sum_number["sum(Number)"]
</code></pre>

blocks|key|1075745|text|类似于其他答案，但没有使用组或agg。我只需选择所讨论的列，将其和起来，收集它，然后获取前两个索引返回一个int。我之所以选择这样的答案而不是接受的答案，是因为我对这个问题并不熟悉，并且对“Number”列没有在接受的答案中显式地求和感到困惑。如果我不得不在某个时候回来，并试图了解正在发生的事情，下面这样的语法对我来说会更容易理解。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1075746|import+pyspark.sql.functions+as+f+++

df.select(f.sum('Number')).collect()[0][0]|code-block|syntax|javascript|1075747|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Similar to other answers, but without the use of a groupby or agg. I just select the column in question, sum it, collect it, and then grab the first two indices to return an int. The only reason I chose this over the accepted answer is I am new to pyspark and was confused that the 'Number' column was not explicitly summed in the accepted answer. If I had to come back after sometime and try to understand what was happening, syntax such as below would be easier for me to follow.
<pre><code>import pyspark.sql.functions as f 

df.select(f.sum('Number')).collect()[0][0]
</code></pre>

blocks|key|1075675|text|您也可以尝试使用first()函数。它从dataframe返回第一行，您可以使用索引访问各个列的值。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1075676|df.groupBy().sum().first()[0]|code-block|syntax|javascript|1075677|在您的示例中，结果是一个带有单行和列的dataframe，因此上面的代码片段可以工作。|1075678|entityMap^0|8|7|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@$9|P|A|Q|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|R|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|S|8|@]|D|@]|E|$]]|$1|M|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|N|$]]

You can also try using <code>first()</code> function. It returns the first row from the dataframe, and you can access values of respective columns using indices.
<pre><code>df.groupBy().sum().first()[0]
</code></pre>
In your case, the result is a dataframe with single row and column, so above snippet works.

blocks|key|1075772|text|选择列作为RDD，使用keys()在行中获取值(或使用.map(lambda+x:+x[0]))，然后使用RDD：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1075773|df.select("Number").rdd.keys().sum()|code-block|syntax|javascript|1075774|使用selectExpr的SQL和|1075775|df.selectExpr("sum(Number)").first()[0]|1075776|entityMap|0|LINK|mutability|MUTABLE|url|https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.selectExpr.html^0|B|6|R|K|0|0|2|A|2|A|0|0|0^^$0|@$1|2|3|4|5|6|7|W|8|@$9|X|A|Y|B|C]|$9|Z|A|10|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|11|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|12|8|@$9|13|A|14|B|C]]|D|@$9|15|A|16|1|17]]|E|$]]|$1|M|3|N|5|H|7|18|8|@]|D|@]|E|$I|J]]|$1|O|3|-4|5|6|7|19|8|@]|D|@]|E|$]]]|P|$Q|$5|R|S|T|E|$U|V]]]]

Select column as RDD, abuse <code>keys()</code> to get value in Row (or use <code>.map(lambda x: x[0])</code>), then use RDD sum:
<pre><code>df.select(&quot;Number&quot;).rdd.keys().sum()
</code></pre>
SQL sum using <a href="https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.selectExpr.html" rel="nofollow noreferrer"><code>selectExpr</code></a>:
<pre><code>df.selectExpr(&quot;sum(Number)&quot;).first()[0]
</code></pre>

blocks|key|2709567|text|下列措施应能发挥作用：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2709568|df.groupBy().sum().rdd.map(lambda+x:+x[0]).collect()|code-block|syntax|javascript|2709569|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

The following should work:

<pre><code>df.groupBy().sum().rdd.map(lambda x: x[0]).collect()
</code></pre>

blocks|key|1075653|text|有时读取csv文件来生成pyspark，也许数字列更改为字符串类型'23'，就像这样，您应该使用pyspark.sql.functions.sum作为int，而不是sum()来获得结果。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1075654|import+pyspark.sql.functions+as+F++++++++++++++++++++++++++++++++++++++++++++++++++++
df.groupBy().agg(F.sum('Number')).show()|code-block|syntax|javascript|1075655|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

sometimes read a csv file to pyspark Dataframe, maybe the numeric column change to string type '23',like this, you should use pyspark.sql.functions.sum to get the result as int , not sum() 

<pre><code>import pyspark.sql.functions as F 
df.groupBy().agg(F.sum('Number')).show()
</code></pre>

I have a pyspark dataframe with a column of numbers. I need to sum that column and then have the result return as an int in a python variable. 

<pre><code>df = spark.createDataFrame([("A", 20), ("B", 30), ("D", 80)],["Letter", "Number"])
</code></pre>

I do the following to sum the column. 

<pre><code>df.groupBy().sum()
</code></pre>

But I get a dataframe back. 

<pre><code>+-----------+
|sum(Number)|
+-----------+
| 130|
+-----------+
</code></pre>

I would 130 returned as an int stored in a variable to be used else where in the program. 

<pre><code>result = 130
</code></pre>

PySpark - Sum a column in dataframe and return results as int

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我有一列数字的电火花数据。我需要对该列进行求和，然后在python变量中将结果作为int返回。df = spark.createDataFrame([("A", 20), ("B", 30), ("D", 80)],["Letter", "Number"])我做下面的总结这一栏。df.groupBy().sum()但我得到了一份数据。+-----------+|sum(Number)|+-----

问PySpark -对数据格式中的列进行求和，并以int形式返回结果
EN

回答 9

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问PySpark -对数据格式中的列进行求和，并以int形式返回结果EN

回答 9

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问PySpark -对数据格式中的列进行求和，并以int形式返回结果
EN