AutoTokenizer.from_pretrained无法加载本地保存的预训练令牌器(PyTorch)
AutoTokenizer.from_pretrained无法加载本地保存的预训练令牌器(PyTorch)
提问于 2020-06-19 14:17:45
我是PyTorch的新手，最近，我一直在尝试与变形金刚合作。我使用的是由HuggingFace提供的预先训练的令牌。
我成功地下载并运行了它们。但是，如果我试图保存它们并再次加载，则会发生一些错误。
如果我使用AutoTokenizer.from_pretrained下载令牌程序，那么它就能工作。
[1]: tokenizer = AutoTokenizer.from_pretrained('distilroberta-base')
text = "Hello there"
enc = tokenizer.encode_plus(text)
enc.keys()
Out[1]: dict_keys(['input_ids', 'attention_mask'])但是，如果我使用tokenizer.save_pretrained("distilroberta-tokenizer")保存它并尝试在本地加载它，那么它就失败了。
[2]: tmp = AutoTokenizer.from_pretrained('distilroberta-tokenizer')
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
238 resume_download=resume_download,
--> 239 local_files_only=local_files_only,
240 )
/opt/conda/lib/python3.7/site-packages/transformers/file_utils.py in cached_path(url_or_filename, cache_dir, force_download, proxies, resume_download, user_agent, extract_compressed_file, force_extract, local_files_only)
266 # File, but it doesn't exist.
--> 267 raise EnvironmentError("file {} not found".format(url_or_filename))
268 else:
OSError: file distilroberta-tokenizer/config.json not found
During handling of the above exception, another exception occurred:
OSError Traceback (most recent call last)
<ipython-input-25-3bd2f7a79271> in <module>
----> 1 tmp = AutoTokenizer.from_pretrained("distilroberta-tokenizer")
/opt/conda/lib/python3.7/site-packages/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
193 config = kwargs.pop("config", None)
194 if not isinstance(config, PretrainedConfig):
--> 195 config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
196
197 if "bert-base-japanese" in pretrained_model_name_or_path:
/opt/conda/lib/python3.7/site-packages/transformers/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
194
195 """
--> 196 config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
197
198 if "model_type" in config_dict:
/opt/conda/lib/python3.7/site-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
250 f"- or '{pretrained_model_name_or_path}' is the correct path to a directory containing a {CONFIG_NAME} file\n\n"
251 )
--> 252 raise EnvironmentError(msg)
253
254 except json.JSONDecodeError:
OSError: Can't load config for 'distilroberta-tokenizer'. Make sure that:
- 'distilroberta-tokenizer' is a correct model identifier listed on 'https://huggingface.co/models'
- or 'distilroberta-tokenizer' is the correct path to a directory containing a config.json file它的意思是“config.josn”不在目录中。在检查目录时，我将得到以下文件的列表：
[3]: !ls distilroberta-tokenizer
Out[3]: merges.txt special_tokens_map.json tokenizer_config.json vocab.json我知道这个问题之前已经发布过了，但似乎都没有效果。我也试图跟随文档，但仍然无法使它发挥作用。
任何帮助都将不胜感激。
Stack Overflow用户
发布于 2020-06-21 21:54:12
我在您的代码中看到几个问题，我在下面列出了这些问题：
- 蒸馏器-令牌器是一个目录，包含了containing等文件。请确保首先创建这个dir。
- 如果这个dir包含config.json而不是tokenizer_config.json，那么使用tokenizer_config.json是有效的。所以请重命名这个文件。
我在下面修改了您的代码，它起作用了。
dir_name = "distilroberta-tokenizer"
if os.path.isdir(dir_name) == False:
os.mkdir(dir_name)
tokenizer.save_pretrained(dir_name)
#Rename config file now
#tmp = AutoTokenizer.from_pretrained(dir_name) 我希望这能帮到你!
谢谢!
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/62472238
复制相关文章
相似问题
腾讯云开发者
