entityMap|blocks|key|8uhtb|text|通常，当遇到cuda+runtine+error时，建议使用CUDA_LAUNCH_BLOCKING=1标志再次运行程序，以获得准确的堆栈跟踪。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|1t5cs|在您的特定情况下，数据的目标对于指定数量的类来说太高(或太低)。|6mhos^0|6|I|U|M|0|0^^$0|$]|1|@$2|3|4|5|6|7|8|J|9|@$A|K|B|L|C|D]|$A|M|B|N|C|D]]|E|@]|F|$]]|$2|G|4|H|6|7|8|O|9|@]|E|@]|F|$]]|$2|I|4|-4|6|7|8|P|9|@]|E|@]|F|$]]]]

In general, when encountering <code>cuda runtine error</code>s, it is advisable to run your program again using the <code>CUDA_LAUNCH_BLOCKING=1</code> flag to obtain an accurate stack trace. 

In your specific case, the targets of your data were too high (or low) for the specified number of classes.

entityMap|blocks|key|b73qh|text|这个问题我已经遇到过好几次了。我发现这是一个索引问题。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|5i3rh|例如，如果您的地面实况标签从1：target+=+[1,2,3,4,5]开始，那么您应该为每个标签减去1，将其更改为：[0,1,2,3,4]。|offset|length|style|CODE|85m71|这每次都能解决我的问题。|24nnm^0|0|G|K|1F|1|1N|B|0|0^^$0|$]|1|@$2|3|4|5|6|7|8|L|9|@]|A|@]|B|$]]|$2|C|4|D|6|7|8|M|9|@$E|N|F|O|G|H]|$E|P|F|Q|G|H]|$E|R|F|S|G|H]]|A|@]|B|$]]|$2|I|4|J|6|7|8|T|9|@]|A|@]|B|$]]|$2|K|4|-4|6|7|8|U|9|@]|A|@]|B|$]]]]

I have encountered this problem several times. And I find it to be an index issue.
For example, if your ground truth label starts at 1: <code>target = [1,2,3,4,5]</code>, then you should subtract <code>1</code> for every label, change it to: <code>[0,1,2,3,4]</code>.
This solves my problem every time.

entityMap|0|type|LINK|mutability|MUTABLE|data|url|https://github.com/huggingface/transformers/issues/5611#issuecomment-666231984|blocks|key|9nrgp|text|我在运行BertModel.from_pretrained('bert-base-uncased').时遇到此错误当错误信息变为“IndexError:+index+out+of+range+in+self”时，我找到了解决方案。这让我找到了this帖子。解决方案是将句子截短到512个长度。|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|1i5g9^0|3D|4|0|0^^$0|$1|$2|3|4|5|6|$7|8]]]|9|@$A|B|C|D|2|E|F|L|G|@]|H|@$I|M|J|N|A|O]]|6|$]]|$A|K|C|-4|2|E|F|P|G|@]|H|@]|6|$]]]]

I encountered this error when running BertModel.from_pretrained('bert-base-uncased'). I found the solution by moving to the CPU when the error message changed to 'IndexError: index out of range in self'. Which led me to <a href="https://github.com/huggingface/transformers/issues/5611#issuecomment-666231984" rel="noreferrer">this</a> post. The solution was to truncate sentences to length 512.

entityMap|blocks|key|8h9tt|text|如果你先切换到CPU，这个错误可能会变得更加复杂。一旦您切换到CPU，它将显示确切的错误，这很可能与索引问题有关，即IndexError:在我的情况下，目标2越界，在您的情况下可能与此相关。问题是“您当前使用了多少个类，输出的形式是什么？”，您可以找到这样的类|type|unstyled|depth|inlineStyleRanges|entityRanges|data|1e00n|max(train_labels)
min(train_labels)|code-block|syntax|javascript|6l58g|在我的例子中，我得到了2和0，这个问题是由于缺少1索引引起的，所以一个快速的技巧是用1快速地将所有2替换为1，这可以通过下面的代码来完成：|cpn8v|train_=train.copy()
train_['label']+=train_['label'].replace(2,1)|583mc|然后运行相同的代码并查看结果，它应该可以工作|dgccv|class+NDataset(torch.utils.data.Dataset):
++++def+__init__(self,+encodings,+labels):
++++++++self.encodings+=+encodings
++++++++self.labels+=+labels

++++def+__getitem__(self,+idx):
++++++++item+=+{key:+torch.tensor(val[idx])+for+key,+val+in+self.encodings.items()}
++++++++item['labels']+=+torch.tensor(self.labels[idx])
++++++++return+item

++++def+__len__(self):
++++++++return+len(self.labels)

train_dataset+=+NDataset(train_encodings,+train_labels)
val_dataset+=+NDataset(val_encodings,+val_labels)
test_dataset+=+NDataset(test_encodings,+test_labels)|1dh62^0|0|0|0|0|0|0^^$0|$]|1|@$2|3|4|5|6|7|8|Q|9|@]|A|@]|B|$]]|$2|C|4|D|6|E|8|R|9|@]|A|@]|B|$F|G]]|$2|H|4|I|6|7|8|S|9|@]|A|@]|B|$]]|$2|J|4|K|6|E|8|T|9|@]|A|@]|B|$F|G]]|$2|L|4|M|6|7|8|U|9|@]|A|@]|B|$]]|$2|N|4|O|6|E|8|V|9|@]|A|@]|B|$F|G]]|$2|P|4|-4|6|7|8|W|9|@]|A|@]|B|$]]]]

This error can be made more elaborative if you switch to CPU first. Once you switch to CPU, it will show the exact error, which is most probably related to the indexing problem, which is IndexError: Target 2 is out of bounds in my case and could be related in yours case.
The issue is &quot;How many classes are you currently using and what is the shape of your output?&quot;, you can find the classes like this
<pre><code>max(train_labels)
min(train_labels)
</code></pre>
which in my case gave me 2 and 0, the problem is caused by missing 1 index, so a quick hack is to quickly replace all 2s with 1s , which can be done through this code:
<pre><code>train_=train.copy()
train_['label'] =train_['label'].replace(2,1)
</code></pre>
then you run the same code and see the results, it should work
<pre><code>class NDataset(torch.utils.data.Dataset):
 def __init__(self, encodings, labels):
 self.encodings = encodings
 self.labels = labels

 def __getitem__(self, idx):
 item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
 item['labels'] = torch.tensor(self.labels[idx])
 return item

 def __len__(self):
 return len(self.labels)

train_dataset = NDataset(train_encodings, train_labels)
val_dataset = NDataset(val_encodings, val_labels)
test_dataset = NDataset(test_encodings, test_labels)
</code></pre>

entityMap|blocks|key|320tn|text|提出"CUDA错误:设备端断言触发“RuntimeError的一种方式是通过使用具有外维索引的list索引到GPU+torch.Tensor中。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|dq09k|因此，此代码片段将引发一个IndexError，并显示消息"IndexError:索引3已超出尺寸为3的维度0的边界“，而不是CUDA错误|4jldk|data+=+torch.randn((3,10),+device=torch.device("cuda"))
data[3,:]|code-block|syntax|javascript|fgua4|然而，这将引发CUDA“设备端断言触发”RuntimeError|b1dbj|data+=+torch.randn((3,10),+device=torch.device("cuda"))
indices+=+[1,3]
data[indices,:]|ecof6|这可能意味着在类标签的情况下，当标签从1而不是0开始时，是最终的类标签(即当label+==+num_classes时)导致错误。|4nnpj|此外，当device为"cpu"时，抛出的错误是IndexError，比如第一个代码段抛出的错误。|8gur2^0|I|C|1B|4|1M|C|0|D|A|0|0|K|C|0|0|12|K|0|B|5|O|A|0^^$0|$]|1|@$2|3|4|5|6|7|8|W|9|@$A|X|B|Y|C|D]|$A|Z|B|10|C|D]|$A|11|B|12|C|D]]|E|@]|F|$]]|$2|G|4|H|6|7|8|13|9|@$A|14|B|15|C|D]]|E|@]|F|$]]|$2|I|4|J|6|K|8|16|9|@]|E|@]|F|$L|M]]|$2|N|4|O|6|7|8|17|9|@$A|18|B|19|C|D]]|E|@]|F|$]]|$2|P|4|Q|6|K|8|1A|9|@]|E|@]|F|$L|M]]|$2|R|4|S|6|7|8|1B|9|@$A|1C|B|1D|C|D]]|E|@]|F|$]]|$2|T|4|U|6|7|8|1E|9|@$A|1F|B|1G|C|D]|$A|1H|B|1I|C|D]]|E|@]|F|$]]|$2|V|4|-4|6|7|8|1J|9|@]|E|@]|F|$]]]]

One way to raise the &quot;CUDA error: device-side assert triggered&quot; <code>RuntimeError</code>, is by indexing into a GPU <code>torch.Tensor</code> using a <code>list</code> having out of dimension indices.
So, this snippet would raise an <code>IndexError</code> with the message &quot;IndexError: index 3 is out of bounds for dimension 0 with size 3&quot;, not the CUDA error
<pre class="lang-py prettyprint-override"><code>data = torch.randn((3,10), device=torch.device(&quot;cuda&quot;))
data[3,:]
</code></pre>
whereas, this one would raise the CUDA &quot;device-side assert triggered&quot; <code>RuntimeError</code>
<pre class="lang-py prettyprint-override"><code>data = torch.randn((3,10), device=torch.device(&quot;cuda&quot;))
indices = [1,3]
data[indices,:]
</code></pre>
which could mean that in case of class labels, such as in the answer by @Rainy, it's the final class label (i.e. when <code>label == num_classes</code>) that is causing the error, when the labels start from 1 rather than 0.
Also, when device is <code>&quot;cpu&quot;</code> the error thrown is <code>IndexError</code> such as the one thrown by the first snippet.

entityMap|blocks|key|buqub|text|当我有一个带有无效值的标签时，我发现我得到了这个错误。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|92bld^0|0^^$0|$]|1|@$2|3|4|5|6|7|8|D|9|@]|A|@]|B|$]]|$2|C|4|-4|6|7|8|E|9|@]|A|@]|B|$]]]]

I found I got this error when I had a label with an invalid value.

I have access to Tesla K20c, I am running ResNet50 on CIFAR10 dataset...
Then I get the error as:
<pre class="lang-none prettyprint-override"><code>THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu line=265 error=59 : device-side assert triggered
Traceback (most recent call last):
 File &quot;main.py&quot;, line 109, in &lt;module&gt;
 train(loader_train, model, criterion, optimizer)
 File &quot;main.py&quot;, line 54, in train
 optimizer.step()
 File &quot;/usr/local/anaconda35/lib/python3.6/site-packages/torch/optim/sgd.py&quot;, line 93, in step
 d_p.add_(weight_decay, p.data)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu:265
</code></pre>
How to resolve this error?

CUDA runtime error (59) : device-side assert triggered

 我可以访问特斯拉K20c，我正在CIFAR10数据集上运行ResNet50 ...然后我得到的错误是： THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generated/../generic/THC...

问CUDA运行时错误(59)：设备端断言已触发
EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问CUDA运行时错误(59)：设备端断言已触发EN

社区

活动

资源

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问CUDA运行时错误(59)：设备端断言已触发
EN