spark单机模式简单搭建

待安装列表 hadoop hive scala spark 一.环境变量配置： ~/.bash_profile PATH=$PATH:$HOME/bin

export PATH

JAVA_HOME=/usr/local/jdk export SCALA_HOME=/usr/local/scala export SPARK_HOME=/usr/local/spark export PATH=.:$JAVA_HOME/bin:$SCALA_HOME/bin:$PATH

HADOOP_HOME=/usr/local/hadoop export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH export HADOOP_HOME PATH

HIVE_HOME=/usr/local/hive PATH=$HIVE_HOME/bin:$PATH export HIVE_HOME PATH

二.hadoop 安装搭建 1.配置ssh互信 ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys

2.修改hostname 为yul32 vi/etc/hosts vi /etc/sysconfig/network （3.修改hadoop-env.sh export JAVA_HOME=/usr/local/jdk）

(4.修改core-site.xml) <property> <name>fs.defaultFS</name> <value>hdfs://yul32:9000</value> </property>

(5.修改hdfs-site.xml) （/usr/hadoop-2.3.0/etc/hadoop) <property> <name>dfs.namenode.name.dir</name> <value>/usr/local/hadoop/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/usr/local/hadoop/dfs/data</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.permission</name> <value>false</value> </property>

(5.修改mapred-site.xml) (mapred-site.xml.template ?) （/usr/hadoop-2.3.0/etc/hadoop) <property> <name>mapreduce.cluster.temp.dir</name> <value></value> <description>No description</description> <final>true</final> </property> <property> <name>mapreduce.cluster.local.dir</name> <value></value> <description>No description</description> <final>true</final> </property>

（6.修改yarn-site.xml） （/usr/hadoop-2.3.0/etc/hadoop) <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property>

7.修改slaves.sh ？？ yul32 8.namenode format 输入命令hadoop namenode –format 9.启动hadoop cd hadoop/sbin start-all.sh ifup ifdown 三.spark 搭建 (/usr/spark-1.1.0-bin-hadoop2.3/conf) <报错 readonly> 1.修改conf/slaves yul32

(2.修改spark-env.sh (/usr/spark-1.1.0-bin-hadoop2.3/conf)) export SCALA_HOME=/usr/local/scala export JAVA_HOME=/usr/local/jdk export SPARK_MASTER_IP=yul32 export SPARK_WORKER_CORES=1 export SPARK_WORKER_INSTANCES=1 export SPARK_MASTER_PORT=7077 export SPARK_WORKER_MEMORY=1g export MASTER=spark://${SPARK_MASTER_IP}:${SPARK_MASTER_PORT} 3.启动spark ./sbin/start-all.sh 4.运行spark例子 ./bin/run-example org.apache.spark.examples.JavaSparkPi 2 5.运行scala-shell ./bin/spark-shell --master local[2] 6.python ./bin/pyspark --master local[2] 7.启动spark sql ./sbin/start-thriftserver.sh(./sbin/start-thriftserver.sh --master yarn) 在后台运行命令： nohup ./sbin/start-thriftserver.sh --master yarn & 查看后台运行进程命令： jobs -l 启动后jps 中包含 SparkSubmit 8.spark sql 客户端连接 ./bin/beeline -u jdbc:hive2://yul32:10000 -n spark -p spark 说明 -n 用户名 -p 密码 或者输入命令 ./bin/beeline beeline> !connect jdbc:hive2://yul32:10000 用户名 密码

上传文件，创建表； 1.hadoop fs -ls /user/ocdc/coc hadoop fs -put /home/ocdc/CI_CUSER_20141104112305197.csv /user/ocdc/coc 2.shark> create table CI_CUSER_20141104112305196( PRODUCT_NO string)ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' ; shark> load data inpath '/user/ocdc/coc/CI_CUSER_20141104112305197.csv' into table CI_CUSER_20141104112305196; shark> create table CI_CUSER_20141104112305197( PRODUCT_NO string)ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' stored as rcfile; shark> insert into table CI_CUSER_20141104112305197 select * from CI_CUSER_20141104112305196;

四.hive 安装配置（非必须） 1.修改hive-env.sh export HADOOP_HOME=/usr/local/hadoop export HIVE_CONF_DIR=/usr/local/hive/conf 2.hive 远程服务 (端口号10000) 启动方式 hive --service hiveserver & 连接Hive JDBC URL：jdbc:hive://ip:10000/default （Hive默认端口：10000 默认数据库名：default） hive数据仓库的位置 hive/conf/hive-site.xml hive.metastroe.warehouse.dir：数据仓库的位置，默认是/user/hive/warehouse； <property> <name>hive.metastore.warehouse.dir</name> <value>/user/hive/warehouse</value> <description>location of default database for the warehouse</description> </property>

shark　jdbc 连接 1.查看SharServer 是否启动 [ocdc@oc98 conf]$ jps 7983 Kafka 8803 SharkCliDriver 7377 ResourceManager 16894 SharkServer 6925 JournalNode 12601 CoarseGrainedExecutorBackend 17056 CoarseGrainedExecutorBackend 18424 Jps 14486 Master 4108 QuorumPeerMain 23408 HRegionServer 17655 RunJar 6727 DataNode 7132 DFSZKFailoverController 7510 NodeManager 12553 WorkerLauncher 6614 NameNode 23268 HMaster 12415 SharkCliDriver 2.查看SharkServer端口 [ocdc@oc98 conf]$ netstat -apn | grep 16894 tcp 0 0 ::ffff:10.1.251.98:57902 :::* LISTEN 16894/java tcp 0 0 :::52309 :::* LISTEN 16894/java tcp 0 0 :::9977 :::* LISTEN 16894/java tcp 0 0 :::41222 :::* LISTEN 16894/java tcp 0 0 :::4040 :::* LISTEN 16894/java tcp 0 0 :::45192 :::* LISTEN 16894/java tcp 0 0 ::ffff:10.1.251.98:35289 ::ffff:10.1.251.98:3306 ESTABLISHED 16894/java tcp 0 0 ::ffff:10.1.251.98:57902 ::ffff:10.1.251.104:41877 ESTABLISHED 16894/java tcp 0 0 ::ffff:10.1.251.98:57902 ::ffff:10.1.251.98:53176 ESTABLISHED 16894/java tcp 0 0 ::ffff:10.1.251.98:9977 ::ffff:10.1.48.20:60586 ESTABLISHED 16894/java tcp 1 0 ::ffff:10.1.251.98:57320 ::ffff:10.1.251.98:50012 CLOSE_WAIT 16894/java tcp 0 0 ::ffff:10.1.251.98:9977 ::ffff:10.1.48.20:59756 ESTABLISHED 16894/java tcp 0 0 ::ffff:10.1.251.98:57902 ::ffff:10.1.251.101:50160 ESTABLISHED 16894/java tcp 0 0 ::ffff:10.1.251.98:57902 ::ffff:10.1.251.98:53172 ESTABLISHED 16894/java tcp 0 0 ::ffff:10.1.251.98:57902 ::ffff:10.1.251.101:50159 ESTABLISHED 16894/java unix 2 [ ] STREAM CONNECTED 8889813 16894/java unix 2 [ ] STREAM CONNECTED 8889793 16894/java 端口为9977 即shark服务启动端口 nohup ./bin/shark –-service sharkserver –-p 9977 &

3.jdbc连接 public class SharkTest { private static String driverName = "org.apache.hadoop.hive.jdbc.HiveDriver"; public static void main(String args[]) throws SQLException { try { Class.forName(driverName); } catch (ClassNotFoundException e) { e.printStackTrace(); System.exit(1); } Connection con = DriverManager.getConnection( "jdbc:hive://10.1.251.98:9977/default", "ocdc", "asiainfo"); Statement stmt = con.createStatement(); ResultSet res = stmt.executeQuery("select * from src "); if (res.next()) { System.out.println(res.getString(1)+ " " + res.getString(2)); } } }

Sparksql Sever启动命令 ./sbin/start-thriftserver.sh --master yarn 客户端连接 ./bin/beeline -u jdbc:hive2://10.1.251.98:10000 -n ocdc -p asiainfo 让配置文件立即生效 source /etc/profile

依赖jar包 hive-common-0.8.1.jar hive-exec-0.8.1.jar hive-jdbc-0.8.1.jar hive-metastore-0.8.1.jar hive-service-0.8.1.jar libfb303.jar slf4j-api-1.4.3.jar slf4j-log4j12-1.4.3.jar httpclient-4.2.5.jar hadoop-common-2.3.0.jar

wq 是保存 i 是编辑 q 是强制退出

（赋权） 1、到你想要赋权的文件夹路径下 2、使用 chmod 777 slaves（为这个文件赋权） 3、赋权给ysy(用户）写的权限 chown -R ysy132:ysy132 dfs

切换用户 使用 su - ysy

(hadoop报错日志位置为 /usr/hadoop-2.3.0/logs) tail -500 hadoop-root-namenode-ysy0915.log 查看500行报错日志

(启动hadoop） 在hadoop-2.3.0目录下 输入./sbin/start-dfs.sh 停止 .sbin/stop-dfs.sh ./sbin/stop-dfs.sh

查看启动的节点 jps (回退到上一个目录下)

eg:spark SQL (select a+b from table) val a:Int = inputRow.getInt(0) val b:Int = inputRow.getInt(1) val result:Int = a + b resultRow.setInt(0,result)

def generateCode(e: Expression): Tree = e match{ case Attribute(ordinal) => q"inputRow.getInt($ordinal)" case Add(left,right)=> q""" { val leftResult = ${generateCode(left)} val rightResult = ${generateCode(right)} leftResult + rightResult } """ }