“纸上得来终觉浅,觉知此事要躬行”
本系列文章主要针对腾讯云上进行大数据系统化操作讲解,在互联网盛行的今日,站在巨人头上的我们。一门技术得来,百度一下终得解决。然而互联网上的文章零零碎碎,达不到强度系统化,以及方便性,快捷性,和简洁性,与针对性准则,这给云上大数据爱好者们带来困扰,使适应腾讯云平台需要花费大量的精力与时间。开发成本大幅增加,然而这些对生产项目尤为重要,
本文章就是为了弥补这些而写,借助腾讯云平台进行一些实战性,选择性讲解。并希望与一些志同道合的小伙伴一起来攻克难关,共同促进云计算,大数据发展。
1.添加hadoop用户组
groupadd hadoop
2.创建hadoop用户并添加到用户组中
useradd -m -g hadoop hadoop
3.修改 hadoop 用户的密码为hadoop
rmp -ivh java.rpm
java -version
vi /home/hadoop/.bashrc
export JAVA_HOME=/usr/java/1.8.0_131
export HADOOP_HOME=/home/hadoop/bigdate/hadoop
export HADOOP_USER_NAME=hadoop
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
vim /etc/hosts
vim /etc/ssh/sshd_config
service sshd restart
tar -zxf hadoop-2.7.1.tar.gz
。mkdir bigdata
mv hadoop-2.7.1 bigdata/
cd bigdata
mv hadoop-2.7.1 hadoop
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. -->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop:tmp.dir</name>
<value>/home/hadoop/bigdate/data/hadoop/tmp</value>
</property>
</configuration>
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. -->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name> dfs:namenode.secondary.http-adress</name>
<value>master:9001</value>
</property>
<property>
<name>dfs:namenode.name.dir</name>
<value>file:/home/hadoop/bigdate/date/hadoop/hdfs/datanode</value>
</property>
<property>
<name>dfs:namenode.name.dir</name>
<value>file:/home/hadoop/bigdate/date/hadoop/hdfs/namenode</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. -->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
<?xml version="1.0"?>
<!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. -->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.nodemanager.aux-serivices</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
slave01 slave02
vim /etc/sysconfig/network
。
reboot
su - hadoop
cd ~
ssh-keygen -t rsa
cd .ssh/
cat id_rsa.pub >>authorized_keys
8.在master主机中将authorized_keys文件传到slave01主机hadoop家目录.shh目录下:scp authorized_keys hadoop@slave01:/home/hadoop/.ssh/
scp authorized_keys hadoop@slave02:/home/hadoop/.ssh/
10.目录权限设置,hadoop用户目录权限为755或者700,.ssh目录是755.id_rsa.pub和authorized_keys权限为644,如下图
17.使用shh+主机名进行验证
hadoop namenode -format
namenode格式化,如不能操作检查环境变量或./hadoop namenode -format
进行namenode格式化cd /home/hadoop/bigdata/hadoop/sbin
sh start-all.sh
jps
四个进程运行中
ssh slave01
下篇文章,笔者将介绍如何在腾讯云上完成 Hive 安装及配置。
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。
原创声明:本文系作者授权腾讯云开发者社区发表,未经许可,不得转载。
如有侵权,请联系 cloudcommunity@tencent.com 删除。