pytorch中的nn.CrossEntropyLoss()计算原理
pytorch中的nn.CrossEntropyLoss()计算原理
生成随机矩阵
x = np.random.rand(2,3) array([[0.10786477, 0.56611762, 0.10557245], [0.4596513 , 0.13174377, 0.82373043]])
计算softmax
在numpy中
y = np.exp(x) / np.sum(np.exp(x), axis=1, keepdims=True)array([[0.27940617, 0.44182742, 0.27876641], [0.31649398, 0.22801164, 0.45549437]])
在pytorch中
torch_x = torch.from_numpy(x) torch_y = nn.Softmax(dim=-1)(torch_x)tensor([[0.2794, 0.4418, 0.2788], [0.3165, 0.2280, 0.4555]], dtype=torch.float64)
计算log_softmax
在numpy中
import numpy as np
x = np.array([[-0.7715, -0.6205,-0.2562]])
y = np.exp(x) / np.sum(np.exp(x), axis=1, keepdims=True)
y = np.log(y) array([[-1.27508877, -0.81683591, -1.27738109], [-1.15045104, -1.47835858, -0.78637192]])
在pytorch中
torch_x = torch.from_numpy(x)
torch_y = nn.LogSoftmax(dim=-1)(torch_x) tensor([[-1.2751, -0.8168, -1.2774], [-1.1505, -1.4784, -0.7864]], dtype=torch.float64)
计算NLLLoss
说明，就是在计算log_softmax之后，根据每个样本的真实标签取得其对应的值。默认权重都是1，而且采取求均值的方式。这里就是-(-1.27508877 + -0.78637192) / 2，即取出第0行的第0个和第1行的第2个，正好对应[0, 2]。
在numpy中
targets = np.array([0, 2])
nll_loss = -(np.sum(np.choose(targets, y.T)) / y.shape[0]) 1.0307303437846973
在pytorch中
首先我们来看下官方代码：
| >>> m = nn.LogSoftmax(dim=1)
| >>> loss = nn.NLLLoss()
| >>> # input is of size N x C = 3 x 5
| >>> input = torch.randn(3, 5, requires_grad=True)
| >>> # each element in target has to have 0 <= value < C
| >>> target = torch.tensor([1, 0, 4])
| >>> output = loss(m(input), target)
| >>> output.backward()发现其也是在计算LogSoftmax之后计算NLLLoss()。 我们在看下pytorch的计算结果：
torch_targets = torch.tensor([0, 2])
torch_nll_loss = nn.NLLLoss()(torch_y, torch_targets)tensor(1.0307, dtype=torch.float64) 与我们一步一步利用numpy计算的保持一致。 最后我们在利用更直观的一种形式来看看：
import torch.nn.functional as F
output = F.nll_loss(F.log_softmax(torch_x, dim=1), torch_targets, reduction='mean')tensor(1.0307, dtype=torch.float64) 结果也符合我们的预期。
https://www.gentlecp.com/articles/874.html https://blog.csdn.net/qq_28418387/article/details/95918829 https://blog.csdn.net/yyhhlancelot/article/details/83142255 https://blog.csdn.net/Jeremy_lf/article/details/102725285