blocks|key|2987152|text|据我所知没有。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|2987153|下面是两个PyTorch实现的示例：|2987154|机器翻译的模块在OpenNMT框架中的应用|unordered-list-item|offset|length|2987155|attention-is-all-you-need-pytorch，重新实现Google的注意就是你所需要的纸|style|CODE|2987156|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/OpenNMT/OpenNMT-py/blob/e8622eb5c6117269bb3accd8eb6f66282b5e67d9/onmt/utils/loss.py#L186|1|https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/train.py#L38|2|https://arxiv.org/abs/1706.03762^0|0|0|5|2|0|0|0|X|0|X|1|19|A|2|0^^$0|@$1|2|3|4|5|6|7|Y|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|Z|8|@]|9|@]|A|$]]|$1|D|3|E|5|F|7|10|8|@]|9|@$G|11|H|12|1|13]]|A|$]]|$1|I|3|J|5|F|7|14|8|@$G|15|H|16|K|L]]|9|@$G|17|H|18|1|19]|$G|1A|H|1B|1|1C]]|A|$]]|$1|M|3|-4|5|6|7|1D|8|@]|9|@]|A|$]]]|N|$O|$5|P|Q|R|A|$S|T]]|U|$5|P|Q|R|A|$S|V]]|W|$5|P|Q|R|A|$S|X]]]]

None that I know of.

Here are two examples of PyTorch implementation:

<ul>
<li><a href="https://github.com/OpenNMT/OpenNMT-py/blob/e8622eb5c6117269bb3accd8eb6f66282b5e67d9/onmt/utils/loss.py#L186" rel="noreferrer"><code>LabelSmoothingLoss</code> module</a> in OpenNMT framework for machine translation</li>
<li><a href="https://github.com/jadore801120/attention-is-all-you-need-pytorch/blob/master/train.py#L38" rel="noreferrer"><code>attention-is-all-you-need-pytorch</code></a>, re-implementation of Google's <a href="https://arxiv.org/abs/1706.03762" rel="noreferrer">Attention is all you need paper</a></li>
</ul>

blocks|key|1436963|text|我一直在寻找从_Loss派生的选项，就像PyTorch中的其他损失类一样，并且尊重基本参数，比如reduction。不幸的是，我找不到直接的替代品，所以最终写了我自己的。不过，我还没有对此进行充分的测试：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1436964|import+torch
from+torch.nn.modules.loss+import+_WeightedLoss
import+torch.nn.functional+as+F

class+SmoothCrossEntropyLoss(_WeightedLoss):
++++def+__init__(self,+weight=None,+reduction='mean',+smoothing=0.0):
++++++++super().__init__(weight=weight,+reduction=reduction)
++++++++self.smoothing+=+smoothing
++++++++self.weight+=+weight
++++++++self.reduction+=+reduction

++++@staticmethod
++++def+_smooth_one_hot(targets:torch.Tensor,+n_classes:int,+smoothing=0.0):
++++++++assert+0+<=+smoothing+<+1
++++++++with+torch.no_grad():
++++++++++++targets+=+torch.empty(size=(targets.size(0),+n_classes),
++++++++++++++++++++device=targets.device)+\
++++++++++++++++.fill_(smoothing+/(n_classes-1))+\
++++++++++++++++.scatter_(1,+targets.data.unsqueeze(1),+1.-smoothing)
++++++++return+targets

++++def+forward(self,+inputs,+targets):
++++++++targets+=+SmoothCrossEntropyLoss._smooth_one_hot(targets,+inputs.size(-1),
++++++++++++self.smoothing)
++++++++lsm+=+F.log_softmax(inputs,+-1)

++++++++if+self.weight+is+not+None:
++++++++++++lsm+=+lsm+*+self.weight.unsqueeze(0)

++++++++loss+=+-(targets+*+lsm).sum(-1)

++++++++if++self.reduction+==+'sum':
++++++++++++loss+=+loss.sum()
++++++++elif++self.reduction+==+'mean':
++++++++++++loss+=+loss.mean()

++++++++return+loss|code-block|syntax|javascript|1436965|其他备选方案：|1436966|utils.pytorch实现|unordered-list-item|1436967|DeepMatch实现|1436968|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/eladhoffer/utils.pytorch/blob/master/cross_entropy.py#L70|1|http://pages.cs.wisc.edu/~sidharth/deepmatcher/_modules/deepmatcher/optim.html^0|7|5|1C|9|0|0|0|0|D|0|0|0|9|1|0^^$0|@$1|2|3|4|5|6|7|11|8|@$9|12|A|13|B|C]|$9|14|A|15|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|16|8|@]|D|@]|E|$I|J]]|$1|K|3|L|5|6|7|17|8|@]|D|@]|E|$]]|$1|M|3|N|5|O|7|18|8|@]|D|@$9|19|A|1A|1|1B]]|E|$]]|$1|P|3|Q|5|O|7|1C|8|@]|D|@$9|1D|A|1E|1|1F]]|E|$]]|$1|R|3|-4|5|6|7|1G|8|@]|D|@]|E|$]]]|S|$T|$5|U|V|W|E|$X|Y]]|Z|$5|U|V|W|E|$X|10]]]]

I've been looking for options that derives from <code>_Loss</code> like other loss classes in PyTorch and respects basic parameters such as <code>reduction</code>. Unfortunately I can't find straight forward replacement so ended up writing my own. I haven't fully tested this yet, however:

<pre><code>import torch
from torch.nn.modules.loss import _WeightedLoss
import torch.nn.functional as F

class SmoothCrossEntropyLoss(_WeightedLoss):
 def __init__(self, weight=None, reduction='mean', smoothing=0.0):
 super().__init__(weight=weight, reduction=reduction)
 self.smoothing = smoothing
 self.weight = weight
 self.reduction = reduction

 @staticmethod
 def _smooth_one_hot(targets:torch.Tensor, n_classes:int, smoothing=0.0):
 assert 0 &lt;= smoothing &lt; 1
 with torch.no_grad():
 targets = torch.empty(size=(targets.size(0), n_classes),
 device=targets.device) \
 .fill_(smoothing /(n_classes-1)) \
 .scatter_(1, targets.data.unsqueeze(1), 1.-smoothing)
 return targets

 def forward(self, inputs, targets):
 targets = SmoothCrossEntropyLoss._smooth_one_hot(targets, inputs.size(-1),
 self.smoothing)
 lsm = F.log_softmax(inputs, -1)

 if self.weight is not None:
 lsm = lsm * self.weight.unsqueeze(0)

 loss = -(targets * lsm).sum(-1)

 if self.reduction == 'sum':
 loss = loss.sum()
 elif self.reduction == 'mean':
 loss = loss.mean()

 return loss
</code></pre>

Other options:

<ul>
<li><a href="https://github.com/eladhoffer/utils.pytorch/blob/master/cross_entropy.py#L70" rel="noreferrer">utils.pytorch</a> Implementation</li>
<li><a href="http://pages.cs.wisc.edu/~sidharth/deepmatcher/_modules/deepmatcher/optim.html" rel="noreferrer">DeepMatch</a> implementation</li>
</ul>

blocks|key|1437023|text|标签平滑PyTorch实现参考：https://github.com/wangleiofficial/label-smoothing-pytorch|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|1437024|import+torch.nn.functional+as+F

def+linear_combination(x,+y,+epsilon):
++++return+epsilon+*+x+%2B+(1+-+epsilon)+*+y


def+reduce_loss(loss,+reduction='mean'):
++++return+loss.mean()+if+reduction+==+'mean'+else+loss.sum()+if+reduction+==+'sum'+else+loss


class+LabelSmoothingCrossEntropy(nn.Module):
++++def+__init__(self,+epsilon:+float+=+0.1,+reduction='mean'):
++++++++super().__init__()
++++++++self.epsilon+=+epsilon
++++++++self.reduction+=+reduction

++++def+forward(self,+preds,+target):
++++++++n+=+preds.size()[-1]
++++++++log_preds+=+F.log_softmax(preds,+dim=-1)
++++++++loss+=+reduce_loss(-log_preds.sum(dim=-1),+self.reduction)
++++++++nll+=+F.nll_loss(log_preds,+target,+reduction=self.reduction)
++++++++return+linear_combination(loss+/+n,+nll,+self.epsilon)|code-block|syntax|javascript|1437025|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/wangleiofficial/label-smoothing-pytorch^0|0|D|G|1M|0|0|0^^$0|@$1|2|3|4|5|6|7|S|8|@$9|T|A|U|B|C]]|D|@$9|V|A|W|1|X]]|E|$]]|$1|F|3|G|5|H|7|Y|8|@]|D|@]|E|$I|J]]|$1|K|3|-4|5|6|7|Z|8|@]|D|@]|E|$]]]|L|$M|$5|N|O|P|E|$Q|R]]]]

label smoothing PyTorch implementation
Ref: <a href="https://github.com/wangleiofficial/label-smoothing-pytorch" rel="nofollow noreferrer">https://github.com/wangleiofficial/label-smoothing-pytorch</a>
<pre><code>import torch.nn.functional as F

def linear_combination(x, y, epsilon):
 return epsilon * x + (1 - epsilon) * y


def reduce_loss(loss, reduction='mean'):
 return loss.mean() if reduction == 'mean' else loss.sum() if reduction == 'sum' else loss


class LabelSmoothingCrossEntropy(nn.Module):
 def __init__(self, epsilon: float = 0.1, reduction='mean'):
 super().__init__()
 self.epsilon = epsilon
 self.reduction = reduction

 def forward(self, preds, target):
 n = preds.size()[-1]
 log_preds = F.log_softmax(preds, dim=-1)
 loss = reduce_loss(-log_preds.sum(dim=-1), self.reduction)
 nll = F.nll_loss(log_preds, target, reduction=self.reduction)
 return linear_combination(loss / n, nll, self.epsilon)
</code></pre>

blocks|key|1437236|text|使用软目标，即硬目标的加权平均和标签上的均匀分布，可以显著提高多类神经网络的泛化和学习速度。通过这种方式平滑标签可以防止网络变得过于自信，标签平滑已经应用于许多最先进的模型中，包括图像分类、语言翻译和语音识别。|type|unstyled|depth|inlineStyleRanges|offset|length|style|BOLD|entityRanges|data|1437237|标签平滑已经在交叉熵损失函数的Tensorflow中实现.BinaryCrossentropy，CategoricalCrossentropy.但是目前还没有标签平滑在PyTorch中的正式实现。然而，目前正在积极讨论这一问题，希望能向它提供一个正式的一揽子方案。下面是讨论主题：第7455期。|CODE|1437238|在这里，我们将介绍一些标签平滑(LS)的可用最佳实现。基本上，有许多实现LS的方法。请参阅这方面的具体讨论，其中之一是这里和另一个在这里。在这里，我们将以2的独特方式实现，每个版本都有两个版本；因此，总4。|1437239|备选案文1:+CrossEntropyLossWithProbs|1437240|通过这种方式，它接受one-hot目标向量。用户必须手动平滑目标向量。它可以在with+torch.no_grad()范围内完成，因为它暂时将所有requires_grad标志设置为false。|1437241|杨德文：来源|ordered-list-item|1437242|import+torch
import+numpy+as+np
import+torch.nn+as+nn
import+torch.nn.functional+as+F
from+torch.autograd+import+Variable
from+torch.nn.modules.loss+import+_WeightedLoss


class+LabelSmoothingLoss(nn.Module):
++++def+__init__(self,+classes,+smoothing=0.0,+dim=-1,+weight+=+None):
++++++++"""if+smoothing+==+0,+it's+one-hot+method
+++++++++++if+0+<+smoothing+<+1,+it's+smooth+method
++++++++"""
++++++++super(LabelSmoothingLoss,+self).__init__()
++++++++self.confidence+=+1.0+-+smoothing
++++++++self.smoothing+=+smoothing
++++++++self.weight+=+weight
++++++++self.cls+=+classes
++++++++self.dim+=+dim

++++def+forward(self,+pred,+target):
++++++++assert+0+<=+self.smoothing+<+1
++++++++pred+=+pred.log_softmax(dim=self.dim)

++++++++if+self.weight+is+not+None:
++++++++++++pred+=+pred+*+self.weight.unsqueeze(0)+++

++++++++with+torch.no_grad():
++++++++++++true_dist+=+torch.zeros_like(pred)
++++++++++++true_dist.fill_(self.smoothing+/+(self.cls+-+1))
++++++++++++true_dist.scatter_(1,+target.data.unsqueeze(1),+self.confidence)
++++++++return+torch.mean(torch.sum(-true_dist+*+pred,+dim=self.dim))|code-block|syntax|javascript|1437243|此外，我们还在self.+smoothing上添加了一个断言检查点，并在此实现上添加了损失加权支持。|1437244|沙阿：来源|1437245|已经把答案发到这里了。这里我们指出，这个实现类似于杨德文的上面的实现，但是，这里我们提到了他的代码，同时最小化了一些code+syntax。|1437246|class+SmoothCrossEntropyLoss(_WeightedLoss):
++++def+__init__(self,+weight=None,+reduction='mean',+smoothing=0.0):
++++++++super().__init__(weight=weight,+reduction=reduction)
++++++++self.smoothing+=+smoothing
++++++++self.weight+=+weight
++++++++self.reduction+=+reduction

++++def+k_one_hot(self,+targets:torch.Tensor,+n_classes:int,+smoothing=0.0):
++++++++with+torch.no_grad():
++++++++++++targets+=+torch.empty(size=(targets.size(0),+n_classes),
++++++++++++++++++++++++++++++++++device=targets.device)+\
++++++++++++++++++++++++++++++++++.fill_(smoothing+/(n_classes-1))+\
++++++++++++++++++++++++++++++++++.scatter_(1,+targets.data.unsqueeze(1),+1.-smoothing)
++++++++return+targets

++++def+reduce_loss(self,+loss):
++++++++return+loss.mean()+if+self.reduction+==+'mean'+else+loss.sum()+\
++++++++if+self.reduction+==+'sum'+else+loss

++++def+forward(self,+inputs,+targets):
++++++++assert+0+<=+self.smoothing+<+1

++++++++targets+=+self.k_one_hot(targets,+inputs.size(-1),+self.smoothing)
++++++++log_preds+=+F.log_softmax(inputs,+-1)

++++++++if+self.weight+is+not+None:
++++++++++++log_preds+=+log_preds+*+self.weight.unsqueeze(0)

++++++++return+self.reduce_loss(-(targets+*+log_preds).sum(dim=-1))|1437247|检查|1437248|import+torch
import+numpy+as+np
import+torch.nn+as+nn
import+torch.nn.functional+as+F
from+torch.autograd+import+Variable
from+torch.nn.modules.loss+import+_WeightedLoss


if+__name__=="__main__":
++++#+1.+Devin+Yang
++++crit+=+LabelSmoothingLoss(classes=5,+smoothing=0.5)
++++predict+=+torch.FloatTensor([[0,+0.2,+0.7,+0.1,+0],
+++++++++++++++++++++++++++++++++[0,+0.9,+0.2,+0.2,+1],+
+++++++++++++++++++++++++++++++++[1,+0.2,+0.7,+0.9,+1]])
++++v+=+crit(Variable(predict),
+++++++++++++Variable(torch.LongTensor([2,+1,+0])))
++++print(v)

++++#+2.+Shital+Shah
++++crit+=+SmoothCrossEntropyLoss(smoothing=0.5)
++++predict+=+torch.FloatTensor([[0,+0.2,+0.7,+0.1,+0],
+++++++++++++++++++++++++++++++++[0,+0.9,+0.2,+0.2,+1],+
+++++++++++++++++++++++++++++++++[1,+0.2,+0.7,+0.9,+1]])
++++v+=+crit(Variable(predict),
+++++++++++++Variable(torch.LongTensor([2,+1,+0])))
++++print(v)

tensor(1.4178)
tensor(1.4178)|1437249|备选案文2:+LabelSmoothingCrossEntropyLoss|1437250|这样，它接受目标向量，并且使用不手动平滑目标向量，而是内建模块负责标签平滑。它允许我们根据F.nll_loss实现标签平滑。|1437251|(a)。王雷官：来源+-+(AFAIK)，原版海报|1437252|(b)。数据龙：添加了来源的加权支持|1437253|此外，我们还稍微最小化了编码编写，使其更加简洁。|1437254|class+LabelSmoothingLoss(torch.nn.Module):
++++def+__init__(self,+smoothing:+float+=+0.1,+
+++++++++++++++++reduction="mean",+weight=None):
++++++++super(LabelSmoothingLoss,+self).__init__()
++++++++self.smoothing+++=+smoothing
++++++++self.reduction+=+reduction
++++++++self.weight++++=+weight

++++def+reduce_loss(self,+loss):
++++++++return+loss.mean()+if+self.reduction+==+'mean'+else+loss.sum()+\
+++++++++if+self.reduction+==+'sum'+else+loss

++++def+linear_combination(self,+x,+y):
++++++++return+self.smoothing+*+x+%2B+(1+-+self.smoothing)+*+y

++++def+forward(self,+preds,+target):
++++++++assert+0+<=+self.smoothing+<+1

++++++++if+self.weight+is+not+None:
++++++++++++self.weight+=+self.weight.to(preds.device)

++++++++n+=+preds.size(-1)
++++++++log_preds+=+F.log_softmax(preds,+dim=-1)
++++++++loss+=+self.reduce_loss(-log_preds.sum(dim=-1))
++++++++nll+=+F.nll_loss(
++++++++++++log_preds,+target,+reduction=self.reduction,+weight=self.weight
++++++++)
++++++++return+self.linear_combination(loss+/+n,+nll)|1437255|NVIDIA/深造-实例：来源|1437256|class+LabelSmoothing(nn.Module):
++++"""NLL+loss+with+label+smoothing.
++++"""
++++def+__init__(self,+smoothing=0.0):
++++++++"""Constructor+for+the+LabelSmoothing+module.
++++++++:param+smoothing:+label+smoothing+factor
++++++++"""
++++++++super(LabelSmoothing,+self).__init__()
++++++++self.confidence+=+1.0+-+smoothing
++++++++self.smoothing+=+smoothing

++++def+forward(self,+x,+target):
++++++++logprobs+=+torch.nn.functional.log_softmax(x,+dim=-1)
++++++++nll_loss+=+-logprobs.gather(dim=-1,+index=target.unsqueeze(1))
++++++++nll_loss+=+nll_loss.squeeze(1)
++++++++smooth_loss+=+-logprobs.mean(dim=-1)
++++++++loss+=+self.confidence+*+nll_loss+%2B+self.smoothing+*+smooth_loss
++++++++return+loss.mean()|1437257|1437258|if+__name__=="__main__":
++++#+Wangleiofficial
++++crit+=+LabelSmoothingLoss(smoothing=0.3,+reduction="mean")
++++predict+=+torch.FloatTensor([[0,+0.2,+0.7,+0.1,+0],
+++++++++++++++++++++++++++++++++[0,+0.9,+0.2,+0.2,+1],+
+++++++++++++++++++++++++++++++++[1,+0.2,+0.7,+0.9,+1]])

++++v+=+crit(Variable(predict),
+++++++++++++Variable(torch.LongTensor([2,+1,+0])))
++++print(v)

++++#+NVIDIA
++++crit+=+LabelSmoothing(smoothing=0.3)
++++predict+=+torch.FloatTensor([[0,+0.2,+0.7,+0.1,+0],
+++++++++++++++++++++++++++++++++[0,+0.9,+0.2,+0.2,+1],+
+++++++++++++++++++++++++++++++++[1,+0.2,+0.7,+0.9,+1]])
++++v+=+crit(Variable(predict),
+++++++++++++Variable(torch.LongTensor([2,+1,+0])))
++++print(v)

tensor(1.3883)
tensor(1.3883)|1437259|更新：正式加入|1437260|torch.nn.CrossEntropyLoss(weight=None,+size_average=None,+
++++++++++++++++++++++++++ignore_index=-+100,+reduce=None,+
++++++++++++++++++++++++++reduction='mean',+label_smoothing=0.0)|1437261|entityMap|0|LINK|mutability|MUTABLE|url|https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy|1|https://www.tensorflow.org/api_docs/python/tf/keras/losses/CategoricalCrossentropy|2|https://github.com/pytorch/pytorch/issues/7455|3|https://github.com/pytorch/pytorch/issues/7455#issuecomment-746999397|4|https://github.com/pytorch/pytorch/issues/7455#issuecomment-747769787|5|https://github.com/PistonY|6|https://github.com/pytorch/pytorch/issues/7455#issuecomment-513062631|7|https://stackoverflow.com/users/207661/shital-shah|8|https://stackoverflow.com/a/59264908/9215780|9|10|https://github.com/pytorch/pytorch/blob/1442a92741d8f39e3d6228af2ed8800cc29ed16f/torch/nn/functional.py#L2429|11|https://github.com/wangleiofficial|12|https://github.com/pytorch/pytorch/issues/7455#issuecomment-720100742|13|https://www.kaggle.com/anjum48|14|https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/166833#930136|15|https://github.com/NVIDIA/DeepLearningExamples|16|https://github.com/NVIDIA/DeepLearningExamples/blob/8d8b21a933fff3defb692e0527fca15532da5dc6/PyTorch/Classification/ConvNets/image_classification/smoothing.py#L18|17|https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html^0|2|3|7|3|B|4|K|4|0|0|4|27|4|F|A|2C|7|T|I|0|1C|N|1|3W|6|2|0|B|8|10|2|25|1|2T|1|1N|2|3|1Q|6|4|0|0|A|7|13|K|21|D|0|0|3|5|4|2|6|0|0|7|F|0|0|2|7|3|2|8|0|1M|B|P|3|9|0|0|0|0|0|19|A|19|A|A|0|4|3|B|8|2|C|0|4|3|D|B|2|E|0|0|0|0|C|F|D|2|G|0|0|0|0|3|4|H|0|0^^$0|@$1|2|3|4|5|6|7|30|8|@$9|31|A|32|B|C]|$9|33|A|34|B|C]|$9|35|A|36|B|C]|$9|37|A|38|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|39|8|@$9|3A|A|3B|B|C]|$9|3C|A|3D|B|C]|$9|3E|A|3F|B|H]|$9|3G|A|3H|B|H]]|D|@$9|3I|A|3J|1|3K]|$9|3L|A|3M|1|3N]|$9|3O|A|3P|1|3Q]]|E|$]]|$1|I|3|J|5|6|7|3R|8|@$9|3S|A|3T|B|C]|$9|3U|A|3V|B|C]|$9|3W|A|3X|B|C]|$9|3Y|A|3Z|B|C]]|D|@$9|40|A|41|1|42]|$9|43|A|44|1|45]]|E|$]]|$1|K|3|L|5|6|7|46|8|@]|D|@]|E|$]]|$1|M|3|N|5|6|7|47|8|@$9|48|A|49|B|H]|$9|4A|A|4B|B|H]|$9|4C|A|4D|B|H]]|D|@]|E|$]]|$1|O|3|P|5|Q|7|4E|8|@]|D|@$9|4F|A|4G|1|4H]|$9|4I|A|4J|1|4K]]|E|$]]|$1|R|3|S|5|T|7|4L|8|@]|D|@]|E|$U|V]]|$1|W|3|X|5|6|7|4M|8|@$9|4N|A|4O|B|H]]|D|@]|E|$]]|$1|Y|3|Z|5|Q|7|4P|8|@]|D|@$9|4Q|A|4R|1|4S]|$9|4T|A|4U|1|4V]]|E|$]]|$1|10|3|11|5|6|7|4W|8|@$9|4X|A|4Y|B|H]]|D|@$9|4Z|A|50|1|51]]|E|$]]|$1|12|3|13|5|T|7|52|8|@]|D|@]|E|$U|V]]|$1|14|3|15|5|6|7|53|8|@]|D|@]|E|$]]|$1|16|3|17|5|T|7|54|8|@]|D|@]|E|$U|V]]|$1|18|3|19|5|6|7|55|8|@]|D|@]|E|$]]|$1|1A|3|1B|5|6|7|56|8|@$9|57|A|58|B|H]]|D|@$9|59|A|5A|1|5B]]|E|$]]|$1|1C|3|1D|5|6|7|5C|8|@]|D|@$9|5D|A|5E|1|5F]|$9|5G|A|5H|1|5I]]|E|$]]|$1|1E|3|1F|5|6|7|5J|8|@]|D|@$9|5K|A|5L|1|5M]|$9|5N|A|5O|1|5P]]|E|$]]|$1|1G|3|1H|5|6|7|5Q|8|@]|D|@]|E|$]]|$1|1I|3|1J|5|T|7|5R|8|@]|D|@]|E|$U|V]]|$1|1K|3|1L|5|Q|7|5S|8|@]|D|@$9|5T|A|5U|1|5V]|$9|5W|A|5X|1|5Y]]|E|$]]|$1|1M|3|1N|5|T|7|5Z|8|@]|D|@]|E|$U|V]]|$1|1O|3|15|5|6|7|60|8|@]|D|@]|E|$]]|$1|1P|3|1Q|5|T|7|61|8|@]|D|@]|E|$U|V]]|$1|1R|3|1S|5|6|7|62|8|@]|D|@$9|63|A|64|1|65]]|E|$]]|$1|1T|3|1U|5|T|7|66|8|@]|D|@]|E|$U|V]]|$1|1V|3|-4|5|6|7|67|8|@]|D|@]|E|$]]]|1W|$1X|$5|1Y|1Z|20|E|$21|22]]|23|$5|1Y|1Z|20|E|$21|24]]|25|$5|1Y|1Z|20|E|$21|26]]|27|$5|1Y|1Z|20|E|$21|28]]|29|$5|1Y|1Z|20|E|$21|2A]]|2B|$5|1Y|1Z|20|E|$21|2C]]|2D|$5|1Y|1Z|20|E|$21|2E]]|2F|$5|1Y|1Z|20|E|$21|2G]]|2H|$5|1Y|1Z|20|E|$21|2I]]|2J|$5|1Y|1Z|20|E|$21|2E]]|2K|$5|1Y|1Z|20|E|$21|2L]]|2M|$5|1Y|1Z|20|E|$21|2N]]|2O|$5|1Y|1Z|20|E|$21|2P]]|2Q|$5|1Y|1Z|20|E|$21|2R]]|2S|$5|1Y|1Z|20|E|$21|2T]]|2U|$5|1Y|1Z|20|E|$21|2V]]|2W|$5|1Y|1Z|20|E|$21|2X]]|2Y|$5|1Y|1Z|20|E|$21|2Z]]]]

The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels. Smoothing the labels in this way prevents the network from becoming over-confident and label smoothing has been used in many state-of-the-art models, including image classification, language translation, and speech recognition.
<hr />
Label Smoothing is already implemented in <code>Tensorflow</code> within the cross-entropy loss functions. <a href="https://www.tensorflow.org/api_docs/python/tf/keras/losses/BinaryCrossentropy" rel="noreferrer">BinaryCrossentropy</a>, <a href="https://www.tensorflow.org/api_docs/python/tf/keras/losses/CategoricalCrossentropy" rel="noreferrer">CategoricalCrossentropy</a>. But currently, there is no official implementation of Label Smoothing in <code>PyTorch</code>. However, there is going an active discussion on it and hopefully, it will be provided with an official package. Here is that discussion thread: <a href="https://github.com/pytorch/pytorch/issues/7455" rel="noreferrer">Issue #7455</a>.
Here We will bring some available best implementation of Label Smoothing (LS) from <code>PyTorch</code> practitioner. Basically, there are many ways to implement the LS. Please refer to this specific discussion on this, one is <a href="https://github.com/pytorch/pytorch/issues/7455#issuecomment-746999397" rel="noreferrer">here</a>, and <a href="https://github.com/pytorch/pytorch/issues/7455#issuecomment-747769787" rel="noreferrer">another here</a>. Here we will bring implementation in 2 unique ways with two versions of each; so total 4.
<h2>Option 1: CrossEntropyLossWithProbs</h2>
In this way, it accepts the <code>one-hot</code> target vector. The user must manually smooth their target vector. And it can be done within <code>with torch.no_grad()</code> scope, as it temporarily sets all of the <code>requires_grad</code> flags to false.
<ol>
<li><a href="https://github.com/PistonY" rel="noreferrer">Devin Yang</a>: <a href="https://github.com/pytorch/pytorch/issues/7455#issuecomment-513062631" rel="noreferrer">Source</a></li>
</ol>
<pre><code>import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
from torch.nn.modules.loss import _WeightedLoss


class LabelSmoothingLoss(nn.Module):
 def __init__(self, classes, smoothing=0.0, dim=-1, weight = None):
 &quot;&quot;&quot;if smoothing == 0, it's one-hot method
 if 0 &lt; smoothing &lt; 1, it's smooth method
 &quot;&quot;&quot;
 super(LabelSmoothingLoss, self).__init__()
 self.confidence = 1.0 - smoothing
 self.smoothing = smoothing
 self.weight = weight
 self.cls = classes
 self.dim = dim

 def forward(self, pred, target):
 assert 0 &lt;= self.smoothing &lt; 1
 pred = pred.log_softmax(dim=self.dim)

 if self.weight is not None:
 pred = pred * self.weight.unsqueeze(0) 

 with torch.no_grad():
 true_dist = torch.zeros_like(pred)
 true_dist.fill_(self.smoothing / (self.cls - 1))
 true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence)
 return torch.mean(torch.sum(-true_dist * pred, dim=self.dim))
</code></pre>
Additionally, we've added an assertion checkmark on <code>self. smoothing</code> and added loss weighting support on this implementation.
<ol start="2">
<li><a href="https://stackoverflow.com/users/207661/shital-shah">Shital Shah</a>: <a href="https://stackoverflow.com/a/59264908/9215780">Source</a></li>
</ol>
Shital already posted the answer here. Here we're pointing out that this implementation is similar to<a href="https://github.com/pytorch/pytorch/issues/7455#issuecomment-513062631" rel="noreferrer"> Devin Yang</a>'s above implementation. However, here we're mentioning his code with minimizing a bit of <code>code syntax</code>.
<pre><code>class SmoothCrossEntropyLoss(_WeightedLoss):
 def __init__(self, weight=None, reduction='mean', smoothing=0.0):
 super().__init__(weight=weight, reduction=reduction)
 self.smoothing = smoothing
 self.weight = weight
 self.reduction = reduction

 def k_one_hot(self, targets:torch.Tensor, n_classes:int, smoothing=0.0):
 with torch.no_grad():
 targets = torch.empty(size=(targets.size(0), n_classes),
 device=targets.device) \
 .fill_(smoothing /(n_classes-1)) \
 .scatter_(1, targets.data.unsqueeze(1), 1.-smoothing)
 return targets

 def reduce_loss(self, loss):
 return loss.mean() if self.reduction == 'mean' else loss.sum() \
 if self.reduction == 'sum' else loss

 def forward(self, inputs, targets):
 assert 0 &lt;= self.smoothing &lt; 1

 targets = self.k_one_hot(targets, inputs.size(-1), self.smoothing)
 log_preds = F.log_softmax(inputs, -1)

 if self.weight is not None:
 log_preds = log_preds * self.weight.unsqueeze(0)

 return self.reduce_loss(-(targets * log_preds).sum(dim=-1))
</code></pre>
Check
<pre><code>import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
from torch.nn.modules.loss import _WeightedLoss


if __name__==&quot;__main__&quot;:
 # 1. Devin Yang
 crit = LabelSmoothingLoss(classes=5, smoothing=0.5)
 predict = torch.FloatTensor([[0, 0.2, 0.7, 0.1, 0],
 [0, 0.9, 0.2, 0.2, 1], 
 [1, 0.2, 0.7, 0.9, 1]])
 v = crit(Variable(predict),
 Variable(torch.LongTensor([2, 1, 0])))
 print(v)

 # 2. Shital Shah
 crit = SmoothCrossEntropyLoss(smoothing=0.5)
 predict = torch.FloatTensor([[0, 0.2, 0.7, 0.1, 0],
 [0, 0.9, 0.2, 0.2, 1], 
 [1, 0.2, 0.7, 0.9, 1]])
 v = crit(Variable(predict),
 Variable(torch.LongTensor([2, 1, 0])))
 print(v)

tensor(1.4178)
tensor(1.4178)
</code></pre>
<hr />
<h2>Option 2: LabelSmoothingCrossEntropyLoss</h2>
By this, it accepts the target vector and uses doesn't manually smooth the target vector, rather the built-in module takes care of the label smoothing. It allows us to implement label smoothing in terms of <a href="https://github.com/pytorch/pytorch/blob/1442a92741d8f39e3d6228af2ed8800cc29ed16f/torch/nn/functional.py#L2429" rel="noreferrer"><code>F.nll_loss</code></a>.
(a). <a href="https://github.com/wangleiofficial" rel="noreferrer">Wangleiofficial</a>: <a href="https://github.com/pytorch/pytorch/issues/7455#issuecomment-720100742" rel="noreferrer">Source</a> - (AFAIK), Original Poster
(b). <a href="https://www.kaggle.com/anjum48" rel="noreferrer">Datasaurus</a>: <a href="https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/166833#930136" rel="noreferrer">Source</a> - Added Weighting Support
Further, we slightly minimize the coding write-up to make it more concise.
<pre><code>class LabelSmoothingLoss(torch.nn.Module):
 def __init__(self, smoothing: float = 0.1, 
 reduction=&quot;mean&quot;, weight=None):
 super(LabelSmoothingLoss, self).__init__()
 self.smoothing = smoothing
 self.reduction = reduction
 self.weight = weight

 def reduce_loss(self, loss):
 return loss.mean() if self.reduction == 'mean' else loss.sum() \
 if self.reduction == 'sum' else loss

 def linear_combination(self, x, y):
 return self.smoothing * x + (1 - self.smoothing) * y

 def forward(self, preds, target):
 assert 0 &lt;= self.smoothing &lt; 1

 if self.weight is not None:
 self.weight = self.weight.to(preds.device)

 n = preds.size(-1)
 log_preds = F.log_softmax(preds, dim=-1)
 loss = self.reduce_loss(-log_preds.sum(dim=-1))
 nll = F.nll_loss(
 log_preds, target, reduction=self.reduction, weight=self.weight
 )
 return self.linear_combination(loss / n, nll)
</code></pre>
<ol start="2">
<li><a href="https://github.com/NVIDIA/DeepLearningExamples" rel="noreferrer">NVIDIA/DeepLearningExamples</a>: <a href="https://github.com/NVIDIA/DeepLearningExamples/blob/8d8b21a933fff3defb692e0527fca15532da5dc6/PyTorch/Classification/ConvNets/image_classification/smoothing.py#L18" rel="noreferrer">Source</a></li>
</ol>
<pre><code>class LabelSmoothing(nn.Module):
 &quot;&quot;&quot;NLL loss with label smoothing.
 &quot;&quot;&quot;
 def __init__(self, smoothing=0.0):
 &quot;&quot;&quot;Constructor for the LabelSmoothing module.
 :param smoothing: label smoothing factor
 &quot;&quot;&quot;
 super(LabelSmoothing, self).__init__()
 self.confidence = 1.0 - smoothing
 self.smoothing = smoothing

 def forward(self, x, target):
 logprobs = torch.nn.functional.log_softmax(x, dim=-1)
 nll_loss = -logprobs.gather(dim=-1, index=target.unsqueeze(1))
 nll_loss = nll_loss.squeeze(1)
 smooth_loss = -logprobs.mean(dim=-1)
 loss = self.confidence * nll_loss + self.smoothing * smooth_loss
 return loss.mean()
</code></pre>
Check
<pre><code>if __name__==&quot;__main__&quot;:
 # Wangleiofficial
 crit = LabelSmoothingLoss(smoothing=0.3, reduction=&quot;mean&quot;)
 predict = torch.FloatTensor([[0, 0.2, 0.7, 0.1, 0],
 [0, 0.9, 0.2, 0.2, 1], 
 [1, 0.2, 0.7, 0.9, 1]])

 v = crit(Variable(predict),
 Variable(torch.LongTensor([2, 1, 0])))
 print(v)

 # NVIDIA
 crit = LabelSmoothing(smoothing=0.3)
 predict = torch.FloatTensor([[0, 0.2, 0.7, 0.1, 0],
 [0, 0.9, 0.2, 0.2, 1], 
 [1, 0.2, 0.7, 0.9, 1]])
 v = crit(Variable(predict),
 Variable(torch.LongTensor([2, 1, 0])))
 print(v)

tensor(1.3883)
tensor(1.3883)
</code></pre>
<hr />
<h2>Update: <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noreferrer">Officially Added</a></h2>
<pre><code>torch.nn.CrossEntropyLoss(weight=None, size_average=None, 
 ignore_index=- 100, reduce=None, 
 reduction='mean', label_smoothing=0.0)
</code></pre>

blocks|key|2987538|text|从1.10.0版本开始，Pytorch正式支持torch.nn.CrossEntropyLoss中的标签平滑和软目标。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|2987539|entityMap|0|LINK|mutability|MUTABLE|url|https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html^0|N|P|N|P|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@$9|O|A|P|B|C]]|D|@$9|Q|A|R|1|S]]|E|$]]|$1|F|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|G|$H|$5|I|J|K|E|$L|M]]]]

From version 1.10.0 Pytorch officially supports label smoothing and soft targets in <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="nofollow noreferrer"><code>torch.nn.CrossEntropyLoss</code></a>.

blocks|key|806672|text|当重量通过时，目前的最高职位并不完全正确。请注意，PyTorch文档指定在传递权重和reduction='mean'时，对权重进行规范化处理：|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|806673|📷|atomic|806674|​|806675|所以削减应该是|806676|if+self.weight+is+not+None:
++++redw+=+1.0/(true_dist*self.weight.unsqueeze(0)).sum()
else:
++++redw+=+1.0/prediction.shape[0]+#+1/n
++++++++
return+torch.sum(redw*torch.sum(-true_dist+*+pred,+dim=self.dim))|code-block|syntax|javascript|806677|而不是|806678|return+torch.mean(torch.sum(-true_dist+*+pred,+dim=self.dim))|806679|完整地，使用此修补程序：|806680|import+torch
import+numpy+as+np
import+torch.nn+as+nn
import+torch.nn.functional+as+F
from+torch.autograd+import+Variable
from+torch.nn.modules.loss+import+_WeightedLoss


class+LabelSmoothingLoss(nn.Module):
++++def+__init__(self,+classes,+smoothing=0.0,+dim=-1,+weight+=+None):
++++++++"""if+smoothing+==+0,+it's+one-hot+method
+++++++++++if+0+<+smoothing+<+1,+it's+smooth+method
++++++++"""
++++++++super(LabelSmoothingLoss,+self).__init__()
++++++++self.confidence+=+1.0+-+smoothing
++++++++self.smoothing+=+smoothing
++++++++self.weight+=+weight
++++++++self.cls+=+classes
++++++++self.dim+=+dim

++++def+forward(self,+pred,+target):
++++++++assert+0+<=+self.smoothing+<+1
++++++++pred+=+pred.log_softmax(dim=self.dim)

++++++++if+self.weight+is+not+None:
++++++++++++pred+=+pred+*+self.weight.unsqueeze(0)+++

++++++++with+torch.no_grad():
++++++++++++true_dist+=+torch.zeros_like(pred)
++++++++++++true_dist.fill_(self.smoothing+/+(self.cls+-+1))
++++++++++++true_dist.scatter_(1,+target.data.unsqueeze(1),+self.confidence)
++++++++if+self.weight+is+not+None:
++++++++++++redw+=+1.0/(true_dist*self.weight.unsqueeze(0)).sum()
++++++++else:
++++++++++++redw+=+1.0/prediction.shape[0]+#+1/n
++++++++
++++++++return+torch.sum(redw*torch.sum(-true_dist+*+pred,+dim=self.dim))|806681|entityMap|0|IMAGE|mutability|IMMUTABLE|imageUrl|https://developer.qcloudimg.com/http-save/yehe-900000/5b0d53a42ec0570731bc358d171fd32d.png|imageAlt^0|16|G|0|0|1|0|0|0|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|18|8|@$9|19|A|1A|B|C]]|D|@]|E|$]]|$1|F|3|G|5|H|7|1B|8|@]|D|@$9|1C|A|1D|1|1E]]|E|$]]|$1|I|3|J|5|6|7|1F|8|@]|D|@]|E|$]]|$1|K|3|L|5|6|7|1G|8|@]|D|@]|E|$]]|$1|M|3|N|5|O|7|1H|8|@]|D|@]|E|$P|Q]]|$1|R|3|S|5|6|7|1I|8|@]|D|@]|E|$]]|$1|T|3|U|5|O|7|1J|8|@]|D|@]|E|$P|Q]]|$1|V|3|W|5|6|7|1K|8|@]|D|@]|E|$]]|$1|X|3|Y|5|O|7|1L|8|@]|D|@]|E|$P|Q]]|$1|Z|3|-4|5|6|7|1M|8|@]|D|@]|E|$]]]|10|$11|$5|12|13|14|E|$15|16|17|-4]]]]

The current top post isn't entirely correct when weights are passed. Note that the PyTorch documentation specifies that when weights are passed and <code>reduction='mean'</code>, the normalization is done accounting for the weights:
<a href="https://i.stack.imgur.com/YcFdW.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YcFdW.png" alt="normalization when weights are passed" /></a>
So the reduction should be
<pre class="lang-py prettyprint-override"><code>if self.weight is not None:
 redw = 1.0/(true_dist*self.weight.unsqueeze(0)).sum()
else:
 redw = 1.0/prediction.shape[0] # 1/n
 
return torch.sum(redw*torch.sum(-true_dist * pred, dim=self.dim))
</code></pre>
instead of
<pre class="lang-py prettyprint-override"><code>return torch.mean(torch.sum(-true_dist * pred, dim=self.dim))
</code></pre>
In full, with this fix:
<pre class="lang-py prettyprint-override"><code>import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
from torch.nn.modules.loss import _WeightedLoss


class LabelSmoothingLoss(nn.Module):
 def __init__(self, classes, smoothing=0.0, dim=-1, weight = None):
 &quot;&quot;&quot;if smoothing == 0, it's one-hot method
 if 0 &lt; smoothing &lt; 1, it's smooth method
 &quot;&quot;&quot;
 super(LabelSmoothingLoss, self).__init__()
 self.confidence = 1.0 - smoothing
 self.smoothing = smoothing
 self.weight = weight
 self.cls = classes
 self.dim = dim

 def forward(self, pred, target):
 assert 0 &lt;= self.smoothing &lt; 1
 pred = pred.log_softmax(dim=self.dim)

 if self.weight is not None:
 pred = pred * self.weight.unsqueeze(0) 

 with torch.no_grad():
 true_dist = torch.zeros_like(pred)
 true_dist.fill_(self.smoothing / (self.cls - 1))
 true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence)
 if self.weight is not None:
 redw = 1.0/(true_dist*self.weight.unsqueeze(0)).sum()
 else:
 redw = 1.0/prediction.shape[0] # 1/n
 
 return torch.sum(redw*torch.sum(-true_dist * pred, dim=self.dim))
</code></pre>

I'm building a <code>ResNet-18</code> classification model for the Stanford Cars dataset using transfer learning. I would like to implement <a href="https://arxiv.org/pdf/1701.06548.pdf" rel="noreferrer">label smoothing</a> to penalize overconfident predictions and improve generalization.
<code>TensorFlow</code> has a simple keyword argument in <a href="https://www.tensorflow.org/api_docs/python/tf/losses/softmax_cross_entropy" rel="noreferrer"><code>CrossEntropyLoss</code></a>. Has anyone built a similar function for <code>PyTorch</code> that I could plug-and-play with?

Label Smoothing in PyTorch

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

我正在使用传输学习为斯坦福汽车数据集构建一个分类模型。我想要实现来惩罚过度自信的预测和改进泛化。TensorFlow在中有一个简单的关键字参数。有人为PyTorch构建了类似的功能，我可以用它进行即插即用吗？

问PyTorch中的标号平滑
EN

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问PyTorch中的标号平滑EN