生活离不开水,正如现代生活离不开数据。欢迎学习Spark框架的知识体系。今天主要介绍Spark框架的环境搭建。
搭建Spark环境前首先需要来搭建Hadoop的环境以及Scala的环境,先来搭建Scala的环境,执行命令:brew install scala,就会自动下载以及安装Scala,安装后需要配置到环境变量.bash_profile,配置信息如下:
export SCALA_HOME=/usr/local/Cellar/scala/2.13.3
export PATH=$PATH:$SCALA_HOME/bin
在控制台里面输入scala,显示如下说明scala的环境搭建是OK的,如下所示:
Welcome to Scala 2.13.3 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_241).
Type in expressions for evaluation. Or try :help.
scala> print("Hello World")
Hello World
下来需要下载hadoop的安装包,安装成功后,对hadoop的安装包进行解压,解压后,把它配置到.bash_profile的环境变量,具体如下:
export HADOOP_HOME=/Applications/devOps/bigData/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
配置环境变量后,记得使用source来刷新环境变量。
下来开始配置hadoop的各个文件的参数,首先来hadoop-env.sh里面指定jdk的目录,具体如下:
cd /hadoop/etc/hadoop
打开hadoop-env.sh文件后,填写正确的jdk的目录,如下所示:
# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_241.jdk/Contents/Home
下来依次的配置core-site.xml,hdfs-site.xml,mapred-site.xml,yarn-site.xml的文件,具体如下:
在core-site.xml的文件里面新增如下内容:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/Applications/devOps/bigData/hadoop/data</value>
</property>
</configuration>
在hdfs-site.xml文件里面新增如下内容:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/Applications/devOps/bigData/hadoop/data/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/Applications/devOps/bigData/hadoop/data/datanode</value>
</property>
<property>
<name>dfs.http.address</name>
<value>localhost:9870</value>
</property>
</configuration>
在mapred-site.xml文件里面新增如下内容:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>
在yarn-site.xml文件里面新增如下内容:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>20480</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2048</value>
</property>
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>2.1</value>
</property>
</configuration>
下来需要配置在本地免密码登录,在控制台里面输入命令:
ssh-keygen -t rs 然后按下回车键,如图所示:
对公钥复制到免密登录的机器上,命令为:ssh-copy-id localhost 然后在控制台输入ssh localhost就会直接登录成功。
配置成功后,在控制台输入hadoop,显示如下信息说明配置成功,如下所示:
下来在hadoop的sbin的目录下启动服务,执行命令具体为./start-dfs.sh,和./start-yarn.sh,具体如下所示:
./start-dfs.sh
Starting namenodes on [localhost]
localhost: namenode is running as process 1139. Stop it first.
Starting datanodes
localhost: datanode is running as process 1249. Stop it first.
Starting secondary namenodes [localhost]
localhost: secondarynamenode is running as process 1396. Stop it first.
2020-11-15 23:30:26,676 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
在浏览器中访问http://localhost:9870/,显示如下说明服务启动OK,如下所示:
在浏览器中输入http://localhost:8088/cluster就会显示hadoop的界面,如下所示:
至此,基于hadoop的伪分布式环境已搭建完成。感谢您的阅读和关注,后续会逐步的介绍PySpark在数据分析和大数据处理方面的案例应用。