spark-3.1.1 on yarn setup on linux

原创

esse LL

修改于 2023-11-05 16:10:00

2630

修改于 2023-11-05 16:10:00

文章被收录于专栏：操作系统实验操作系统实验

1. before start

follow all steps in hadoop-3.1.3 cluster setup on linux

and then switch to root user:

su

2. cp spark and extract

tar -xvzf /opt/software/spark-3.1.1-bin-hadoop3.2.tgz -C /opt/module

3. set env variables

vi /etc/profile

add the following 4 lines:

export SPARK_HOME="/opt/module/spark-3.1.1-bin-hadoop3.2"
export PATH=$PATH:$SPARK_HOME/bin
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native

source or re-login:

source /etc/profile

4. test spark-submit

cd /opt
spark-submit --version

5. config yarn

cd $SPARK_HOME
cp conf/spark-defaults.conf.template conf/spark-defaults.conf
vi conf/spark-defaults.conf

add:

spark.master yarn

6. run on yarn

start hdfs and yarn:

$HADOOP_HOME/sbin/start-dfs.sh
$HADOOP_HOME/sbin/start-yarn.sh

use jps to check ResourceManager process

spark-submit --master yarn --class org.apache.spark.examples.SparkPi  $SPARK_HOME/examples/jars/spark-examples_2.12-3.1.1.jar

7. test hdfs

put test file in hdfs:

cd ~
wget -O alice.txt https://www.gutenberg.org/files/11/11-0.txt
hdfs dfs -mkdir inputs
hdfs dfs -put alice.txt inputs

run spark-shell and read the test file:

spark-shell --master yarn --deploy-mode client

val input = sc.textFile("inputs/alice.txt")
// Count the number of non blank lines
input.filter(line => line.length()>0).count()

8. optional ops

8.1 config workers memory

vi $SPARK_HOME/conf/spark-defaults.conf

add following 3 lines:

spark.driver.memory 512m
spark.yarn.am.memory 512m
spark.executor.memory 512m

if necessary:

8.2 switch to jdk 1.8

try following cmds:

which java
ls -l /usr/bin/java
mv /usr/bin/java /usr/bin/java2
java -version

jdk should be 1.8 now.

9. more information

Spark web UI at http://master:4040

Yarn web UI at http://master:8088/

download archive version of pkgs, view http://archive.apache.org/dist/spark/

for more information, view

https://spark.apache.org/docs/latest/running-on-yarn.html

https://www.linode.com/docs/guides/install-configure-run-spark-on-top-of-hadoop-yarn-cluster/

https://sparkbyexamples.com/spark/spark-setup-on-hadoop-yarn/

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

apache-spark

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

apache-spark

登录后参与评论

0 条评论

热度