blocks|key|643749|text|我认为所有的预测都是0.0或1.0。那么中间阈值将不会有任何影响。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|643750|考虑BinaryClassificationMetrics的numBins参数|offset|length|style|CODE|643751|ROC+numBins**:**如果大于0，则内部计算的曲线(ROC曲线、PR曲线)将向下采样到如此多的“|BOLD|643752|”。如果为0，则不会进行下采样。这很有用，因为对于输入中的每个不同分数，曲线都包含一个点，这可能与输入本身一样大--几百万个点或更多，而几千个点可能完全足以汇总曲线。下采样后，曲线将改为由近似的numBins点组成。点是由相同数量的连续点构成的。每个存储箱的大小是floor(scoreAndLabels.count()+/+numBins)，这意味着最终的存储箱数量可能不完全等于numBins。因此，每个分区中的最后一个仓位可能更小，这意味着在分区边界处可能有额外的样本。|blockquote|643753|643754|643755|因此，如果您不设置numBins，则将在所有不同的预测值上计算精度。在您的情况下，这似乎只有0.0和1.0。|643756|entityMap^0|0|2|R|U|7|0|4|9|4|7|0|0|0|0|9|7|0^^$0|@$1|2|3|4|5|6|7|T|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|U|8|@$D|V|E|W|F|G]|$D|X|E|Y|F|G]]|9|@]|A|$]]|$1|H|3|I|5|6|7|Z|8|@$D|10|E|11|F|J]|$D|12|E|13|F|G]]|9|@]|A|$]]|$1|K|3|L|5|M|7|14|8|@]|9|@]|A|$]]|$1|N|3|-4|5|6|7|15|8|@]|9|@]|A|$]]|$1|O|3|-4|5|6|7|16|8|@]|9|@]|A|$]]|$1|P|3|Q|5|6|7|17|8|@$D|18|E|19|F|G]]|9|@]|A|$]]|$1|R|3|-4|5|6|7|1A|8|@]|9|@]|A|$]]]|S|$]]

I think what happens is that all the predictions are 0.0 or 1.0. Then the intermediate threshold values make no difference.

Consider the <code>numBins</code> argument of <code>BinaryClassificationMetrics</code>:

<blockquote>
 <code>numBins</code>:
 if greater than 0, then the curves (ROC curve, PR curve) computed internally will be down-sampled to this many "bins". If 0, no down-sampling will occur. This is useful because the curve contains a point for each distinct score in the input, and this could be as large as the input itself -- millions of points or more, when thousands may be entirely sufficient to summarize the curve. After down-sampling, the curves will instead be made of approximately numBins points instead. Points are made from bins of equal numbers of consecutive points. The size of each bin is floor(scoreAndLabels.count() / numBins), which means the resulting number of bins may not exactly equal numBins. The last bin in each partition may be smaller as a result, meaning there may be an extra sample at partition boundaries.
</blockquote>

So if you don't set <code>numBins</code>, then precision will be calculated at all the different prediction values. In your case this seems to be just 0.0 and 1.0.

blocks|key|325578|text|在进行预测之前，您需要清除模型阈值。清除阈值使您的预测返回一个分数，而不是分类标签。否则，您将只有两个阈值，即您的标签0.0和1.0。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|325579|model.clearThreshold()|code-block|syntax|javascript|325580|来自predictionsAndLabels的元组应该类似于(0.6753421,1.0)，而不是(1.0,1.0)|offset|length|style|CODE|325581|看一看https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassificationMetricsExample.scala|325582|如果输入很大，您可能仍然希望设置numBins来控制点的数量。|325583|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassificationMetricsExample.scala^0|0|0|U|F|1D|9|0|3|3W|0|0|0^^$0|@$1|2|3|4|5|6|7|Y|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|Z|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|10|8|@$I|11|J|12|K|L]|$I|13|J|14|K|L]]|9|@]|A|$]]|$1|M|3|N|5|6|7|15|8|@]|9|@$I|16|J|17|1|18]]|A|$]]|$1|O|3|P|5|6|7|19|8|@]|9|@]|A|$]]|$1|Q|3|-4|5|6|7|1A|8|@]|9|@]|A|$]]]|R|$S|$5|T|U|V|A|$W|X]]]]

You need to clear the model threshold before you make predictions. Clearing threshold makes your predictions return a score and not the classified label. If not you will only have two thresholds, i.e. your labels 0.0 and 1.0.

<pre><code>model.clearThreshold()
</code></pre>

A tuple from predictionsAndLabels should look like <code>(0.6753421,1.0)</code> and not <code>(1.0,1.0)</code>

Take a look at <a href="https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassificationMetricsExample.scala" rel="noreferrer">https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassificationMetricsExample.scala</a>

You probably still want to set numBins to control the number of points if the input is large.

blocks|key|637951|text|首先，尝试像这样添加更多的numBins+(这里的bin是10)：|type|unstyled|depth|inlineStyleRanges|entityRanges|data|637952|val+metrics+=+new+BinaryClassificationMetrics(probabilitiesAndLabels,10);|code-block|syntax|javascript|637953|如果您仍然只有0和1两个阈值，那么请检查以确保您定义predictionAndLabels的方式。如果你不小心提供了(label,+prediction)而不是(prediction,+label)，你可能会遇到这个问题。|offset|length|style|CODE|637954|entityMap^0|0|0|1M|J|28|J|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@$I|R|J|S|K|L]|$I|T|J|U|K|L]]|9|@]|A|$]]|$1|M|3|-4|5|6|7|V|8|@]|9|@]|A|$]]]|N|$]]

First, try adding more bins like this (here numBins is 10):

<pre><code>val metrics = new BinaryClassificationMetrics(probabilitiesAndLabels,10);
</code></pre>

If you still only have two thresholds of 0 and 1, then check to make sure the way you have defined your predictionAndLabels. You many be having this problem if you have accidentally provided <code>(label, prediction)</code> instead of <code>(prediction, label)</code>.

I have logistic regression mode, where I explicitly set the threshold to 0.5.

<pre><code>model.setThreshold(0.5)
</code></pre>

I train the model and then I want to get basic stats -- precision, recall etc.

This is what I do when I evaluate the model:

<pre><code>val metrics = new BinaryClassificationMetrics(predictionAndLabels)

val precision = metrics.precisionByThreshold


precision.foreach { case (t, p) =&gt;

 println(s"Threshold is: $t, Precision is: $p")

 }
</code></pre>

I get results with only 0.0 and 1.0 as values of threshold and 0.5 is completely ignored. 

Here is the output of the above loop:

Threshold is: 1.0, Precision is: 0.8571428571428571

Threshold is: 0.0, Precision is: 0.3005181347150259

When I call metrics.thresholds() it also returns only two values, 0.0 and 1.0. 

How do I get the precision and recall values with threshold as 0.5?

Spark : regression model threshold and precision

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我有逻辑回归模式，其中我显式地将阈值设置为0.5。model.setThreshold(0.5)我训练模型，然后我想获得基本的统计数据--精度、召回率等。这是我在评估模型时所做的：val metrics = new BinaryClassificationMetrics(predictionAndLabels)val ...

问Spark :回归模型阈值和精度
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问Spark :回归模型阈值和精度EN