Dataproc群集创建失败，PIP错误“无法生成车轮”
Dataproc群集创建失败，PIP错误“无法生成车轮”
提问于 2022-01-17 15:17:53
我们用下面的配置旋转星系团。它过去一直运行良好，直到上周，但是现在由于error ERROR: Failed cleaning build dir for libcst Failed to build libcst ERROR: Could not build wheels for libcst which use PEP 517 and cannot be installed directly失败了。
Building wheels for collected packages: pynacl, libcst
Building wheel for pynacl (PEP 517): started
Building wheel for pynacl (PEP 517): still running...
Building wheel for pynacl (PEP 517): finished with status 'done'
Created wheel for pynacl: filename=PyNaCl-1.5.0-cp37-cp37m-linux_x86_64.whl size=201317 sha256=4e5897bc415a327f6b389b864940a8c1dde9448017a2ce4991517b30996acb71
Stored in directory: /root/.cache/pip/wheels/2f/01/7f/11d382bf954a093a55ed9581fd66c3b45b98769f292367b4d3
Building wheel for libcst (PEP 517): started
Building wheel for libcst (PEP 517): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /opt/conda/anaconda/bin/python /opt/conda/anaconda/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpon3bonqi
cwd: /tmp/pip-install-9ozf4fcp/libcst群集配置命令：
gcloud dataproc clusters create cluster-test \
--enable-component-gateway \
--region us-east1 \
--zone us-east1-b \
--master-machine-type n1-highmem-32 \
--master-boot-disk-size 500 \
--num-workers 3 \
--worker-machine-type n1-highmem-16 \
--worker-boot-disk-size 500 \
--optional-components ANACONDA,JUPYTER,ZEPPELIN \
--image-version 1.5.54-ubuntu18 \
--tags <tag-name> \
--bucket '<cloud storage bucket>' \
--initialization-actions 'gs://goog-dataproc-initialization-actions-us-east1/connectors/connectors.sh','gs://goog-dataproc-initialization-actions-us-east1/python/pip-install.sh' \
--metadata='PIP_PACKAGES=wheel datalab xgboost==1.3.3 shap oyaml click apache-airflow apache-airflow-providers-google' \
--initialization-action-timeout 30m \
--metadata gcs-connector-version=2.1.1,bigquery-connector-version=1.1.1,spark-bigquery-connector-version=0.17.2 \
--project <project-name>我尝试过这样做： a)我试图显式地安装轮式软件包，作为pip包的一部分，但是问题没有解决。
( b)带有升级pip的Gcloud命令t：
gcloud dataproc clusters create cluster-test \
--enable-component-gateway \
--region us-east1 \
--zone us-east1-b \
--master-machine-type n1-highmem-32 \
--master-boot-disk-size 500 \
--num-workers 3 \
--worker-machine-type n1-highmem-16 \
--worker-boot-disk-size 500 \
--optional-components ANACONDA,JUPYTER,ZEPPELIN \
--image-version 1.5.54-ubuntu18 \
--tags <tag-name> \
--bucket '<cloud storage bucket>' \
--initialization-actions 'gs://goog-dataproc-initialization-actions-us-east1/connectors/connectors.sh','gs://<bucket-path>/upgrade-pip.sh','gs://goog-dataproc-initialization-actions-us-east1/python/pip-install.sh' \
--metadata='PIP_PACKAGES=wheel datalab xgboost==1.3.3 shap oyaml click apache-airflow apache-airflow-providers-google' \
--initialization-action-timeout 30m \
--metadata gcs-connector-version=2.1.1,bigquery-connector-version=1.1.1,spark-bigquery-connector-version=0.17.2 \
--project <project-name>回答 1
Stack Overflow用户
回答已采纳
发布于 2022-01-19 21:50:31
似乎你需要升级pip，看看这个question。
但是在Dataproc集群中可以有多个pip，您需要选择正确的一个。
- For init action，在集群创建时，
/opt/conda/default是指向/opt/conda/miniconda3或/opt/conda/anaconda的符号链接，取决于您选择的Conda env，默认值是Miniconda3，但在您的示例中是Anaconda。所以您可以运行/opt/conda/anaconda/bin/pip install --upgrade pip.
或/opt/conda/default/bin/pip install --upgrade pip。
- For 自定义图像，在图像创建时，您希望使用显式完整路径，即Anaconda的
/opt/conda/anaconda/bin/pip install --upgrade pip或Miniconda3的/opt/conda/miniconda3/bin/pip install --upgrade pip.
。
因此，您可以简单地将/opt/conda/anaconda/bin/pip install --upgrade pip用于init操作和自定义映像。
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/70743642
复制相关文章
相似问题
腾讯云开发者
