Mrjob无法在dataproc上创建集群：__init__()获得意外的关键字参数'channel‘
Mrjob无法在dataproc上创建集群：__init__()获得意外的关键字参数'channel‘
提问于 2020-11-01 13:11:57
我正在尝试使用Python mrjob库在Google Cloud Dataproc上运行Hadoop Map Reduce字数计算示例。但是，mrjob会失败，并出现以下异常：
TypeError: __init__() got an unexpected keyword argument 'channel'
Traceback (most recent call last):
File "freq.py", line 21, in <module>
MRWordFreqCount.run()
File "/usr/local/lib/python3.8/dist-packages/mrjob/job.py", line 616, in run
cls().execute()
File "/usr/local/lib/python3.8/dist-packages/mrjob/job.py", line 687, in execute
self.run_job()
File "/usr/local/lib/python3.8/dist-packages/mrjob/job.py", line 636, in run_job
runner.run()
File "/usr/local/lib/python3.8/dist-packages/mrjob/runner.py", line 503, in run
self._run()
File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 468, in _run
self._launch()
File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 473, in _launch
self._launch_cluster()
File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 637, in _launch_cluster
self._get_cluster(self._cluster_id)
File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 1188, in _get_cluster
return self.cluster_client.get_cluster(
File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 376, in cluster_client
return google.cloud.dataproc_v1beta2.ClusterControllerClient(
TypeError: __init__() got an unexpected keyword argument 'channel'我检查了是否正确设置了GOOGLE_APPLICATION_CREDENTIALS，在Google Cloud上启用了API，并为服务帐户设置了所有必需的角色。
mrjob成功地将文件上传到谷歌云存储。但是一旦它试图创建一个新的Dataproc集群就失败了。
可能会出什么问题呢？
在Dataproc上启动mrjob的命令行：
$ python3 freq.py -r dataproc words.txt当前Python环境：
$ python3 -VV
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0]
$ pip3 list | grep google
google-api-core 1.23.0
google-auth 1.23.0
google-auth-oauthlib 0.4.2
google-cloud-core 1.4.3
google-cloud-dataproc 2.0.2
google-cloud-logging 1.15.1
google-cloud-storage 1.32.0
google-crc32c 1.0.0
google-pasta 0.2.0
google-resumable-media 1.1.0
googleapis-common-protos 1.52.0
$ pip3 list | grep mrjob
mrjob 0.7.4Stack Overflow用户
发布于 2020-11-01 13:18:13
解决方案是将google-cloud-dataproc降级到1.1.1。
在调试到mrjob实现之后，我发现mrjob版本0.7.4使用一个参数调用google.cloud.dataproc_v1beta2.ClusterControllerClient的构造函数，这个参数在google-cloud-dataproc库中从版本2.0.0开始被重命名。
如何使用pip3降级：
$ pip3 install --force-reinstall --no-deps google-cloud-dataproc==1.1.1页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/64628864
复制相关文章
相似问题
腾讯云开发者
