在电子病历上运行稀疏的DocumentAssembler
在电子病历上运行稀疏的DocumentAssembler
提问于 2021-02-04 08:20:23
我在试着在EMR上运行sparknlp。我登录了我的齐柏林飞艇笔记本，并运行以下命令
import sparknlp
spark = SparkSession.builder \
.appName("BBC Text Categorization")\
.config("spark.driver.memory","8G")\
.config("spark.memory.offHeap.enabled",True)\
.config("spark.memory.offHeap.size","8G") \
.config("spark.driver.maxResultSize", "2G") \
.config("spark.jars.packages", "JohnSnowLabs:spark-nlp:2.4.5")\
.config("spark.kryoserializer.buffer.max", "1000M")\
.config("spark.network.timeout","3600s")\
.getOrCreate()
from sparknlp.base import DocumentAssembler
documentAssembler = DocumentAssembler()\
.setInputCol("description") \
.setOutputCol('document')这导致以下错误：
Fail to execute line 1: documentAssembler = DocumentAssembler()\
Traceback (most recent call last):
File "/tmp/zeppelin_pyspark-4581426413302524147.py", line 380, in <module>
exec(code, _zcUserQueryNameSpace)
File "<stdin>", line 1, in <module>
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 110, in wrapper
return func(self, **kwargs)
File "/usr/local/lib/python3.6/site-packages/sparknlp/base.py", line 148, in __init__
super(DocumentAssembler, self).__init__(classname="com.johnsnowlabs.nlp.DocumentAssembler")
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 110, in wrapper
return func(self, **kwargs)
File "/usr/local/lib/python3.6/site-packages/sparknlp/internal.py", line 72, in __init__
self._java_obj = self._new_java_obj(classname, self.uid)
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/wrapper.py", line 67, in _new_java_obj
return java_obj(*java_args)
TypeError: 'JavaPackage' object is not callable为了理解这个问题，我尝试登录到主服务器，并在pyspark控制台中运行上面的命令。所有操作都很好，如果我使用命令：pyspark --packages JohnSnowLabs:spark-nlp:2.4.5启动pyspark控制台，就不会得到上面的错误
但是，在使用命令pyspark时，我得到了与前面相同的错误
我怎样才能在齐柏林飞艇笔记本上完成这个工作呢？
设置详细信息：
EMR 5.27.0
spark 2.4.4
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)下面是我的引导脚本：
#!/bin/bash
sudo yum install -y python36-devel python36-pip python36-setuptools python36-virtualenv
sudo python36 -m pip install --upgrade pip
sudo python36 -m pip install pandas
sudo python36 -m pip install boto3
sudo python36 -m pip install re
sudo python36 -m pip install spark-nlp==2.7.2Stack Overflow用户
发布于 2022-05-17 14:32:48
- 请确保使用受支持的电子病历版本，支持的版本为请看这里。
- 引导脚本应该包含
#!/bin/bash
set -x -e
echo -e 'export PYSPARK_PYTHON=/usr/bin/python3
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_JARS_DIR=/usr/lib/spark/jars
export SPARK_HOME=/usr/lib/spark' >> $HOME/.bashrc && source $HOME/.bashrc
sudo python3 -m pip install awscli boto spark-nlp
set +x
exit 0- 提供conf文件，您可以将其存储在S3中并传递给集群。
[{
"Classification": "spark-env",
"Configurations": [{
"Classification": "export",
"Properties": {
"PYSPARK_PYTHON": "/usr/bin/python3"
}
}]
},
{
"Classification": "spark-defaults",
"Properties": {
"spark.yarn.stagingDir": "hdfs:///tmp",
"spark.yarn.preserve.staging.files": "true",
"spark.kryoserializer.buffer.max": "2000M",
"spark.serializer": "org.apache.spark.serializer.KryoSerializer",
"spark.driver.maxResultSize": "0",
"spark.jars.packages": "com.johnsnowlabs.nlp:spark-nlp_2.12:3.4.4"
}
}
]- 最后，启动MER集群，即从CLI启动。
aws emr create-cluster \
--name "Spark NLP 3.4.4" \
--release-label emr-6.2.0 \
--applications Name=Hadoop Name=Spark Name=Hive \
--instance-type m4.4xlarge \
--instance-count 3 \
--use-default-roles \
--log-uri "s3://<S3_BUCKET>/" \
--bootstrap-actions Path=s3://<S3_BUCKET>/emr-bootstrap.sh,Name=custome \
--configurations "https://<public_access>/sparknlp-config.json" \
--ec2-attributes KeyName=<your_ssh_key>,EmrManagedMasterSecurityGroup=<security_group_with_ssh>,EmrManagedSlaveSecurityGroup=<security_group_with_ssh> \
--profile <aws_profile_credentials>页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/66041769
复制相关文章
相似问题
腾讯云开发者
