磁盘空间不足
磁盘空间不足
提问于 2019-09-12 04:14:26
当在AML计算上运行AML管道时，我得到这样的错误：
我可以尝试重新启动集群，但这可能无法解决问题(如果存储累积没有节点，则应该进行清理。
Session ID: 933fc468-7a22-425d-aa1b-94eba5784faa
{"error":{"code":"ServiceError","message":"Job preparation failed: [Errno 28] No space left on device","detailsUri":null,"target":null,"details":[],"innerError":null,"debugInfo":{"type":"OSError","message":"[Errno 28] No space left on device","stackTrace":" File \"/mnt/batch/tasks/shared/LS_root/jobs/jj2/azureml/piperun-20190911_1568231788841835_1/mounts/workspacefilestore/azureml/PipeRun-20190911_1568231788841835_1-setup/job_prep.py\", line 126, in <module>\n invoke()\n File \"/mnt/batch/tasks/shared/LS_root/jobs/jj2/azureml/piperun-20190911_1568231788841835_1/mounts/workspacefilestore/azureml/PipeRun-20190911_1568231788841835_1-setup/job_prep.py\", line 97, in invoke\n extract_project(project_dir, options.project_zip, options.snapshots)\n File \"/mnt/batch/tasks/shared/LS_root/jobs/jj2/azureml/piperun-20190911_1568231788841835_1/mounts/workspacefilestore/azureml/PipeRun-20190911_1568231788841835_1-setup/job_prep.py\", line 60, in extract_project\n project_fetcher.fetch_project_snapshot(snapshot[\"Id\"], snapshot[\"PathStack\"])\n File \"/mnt/batch/tasks/shared/LS_root/jobs/jj2/azureml/piperun-20190911_1568231788841835_1/mounts/workspacefilestore/azureml/PipeRun-20190911_1568231788841835_1/azureml-setup/project_fetcher.py\", line 72, in fetch_project_snapshot\n _download_tree(sas_tree, path_stack)\n File \"/mnt/batch/tasks/shared/LS_root/jobs/jj2/azureml/piperun-20190911_1568231788841835_1/mounts/workspacefilestore/azureml/PipeRun-20190911_1568231788841835_1/azureml-setup/project_fetcher.py\", line 106, in _download_tree\n _download_tree(child, path_stack)\n File \"/mnt/batch/tasks/shared/LS_root/jobs/jj2/azureml/piperun-20190911_1568231788841835_1/mounts/workspacefilestore/azureml/PipeRun-20190911_1568231788841835_1/azureml-setup/project_fetcher.py\", line 106, in _download_tree\n _download_tree(child, path_stack)\n File \"/mnt/batch/tasks/shared/LS_root/jobs/jj2/azureml/piperun-20190911_1568231788841835_1/mounts/workspacefilestore/azureml/PipeRun-20190911_1568231788841835_1/azureml-setup/project_fetcher.py\", line 98, in _download_tree\n fh.write(response.read())\n","innerException":null,"data":null,"errorResponse":null}},"correlation":null,"environment":null,"location":null,"time":"0001-01-01T00:00:00+00:00"}我希望作业能按其应有的方式运行。事实上，我已经检查了该节点，该节点确实有大量可用硬盘空间：
root@4f57957ac829466a86bad4d4dc51fadd000001:~# df -kh Filesystem Size Used Avail Use% Mounted on
udev 28G 0 28G 0% /dev
tmpfs 5.6G 9.0M 5.5G 1% /run
/dev/sda1 125G 2.8G 122G 3% /
tmpfs 28G 0 28G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 28G 0 28G 0% /sys/fs/cgroup
/dev/sdb1 335G 6.7G 311G 3% /mnt
tmpfs 5.6G 0 5.6G 0% /run/user/1002关于我应该检查什么的建议？
回答 2
Stack Overflow用户
回答已采纳
发布于 2019-09-13 01:26:46
你似乎遇到了Azure文件共享限制。您可以使用以下示例代码来更改您的运行，以使用blob存储，它可以扩展到大量并行运行的作业：
Stack Overflow用户
发布于 2019-09-14 06:02:52
我们还在开发一个功能，用于在运行作业之前或之后清理磁盘。目前还没有针对它的ETA。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/57896195
复制相关文章
相似问题
腾讯云开发者
