写个yarn的监控
在星球里和微信群里很多朋友都有疑惑，如何监控 yarn 上 spark 或者 mr 应用的存活状态，浪尖今天在这里分享一下实现方法，实际上只需要简单的几行代码即可。
首先是，pom文件，添加yarn相关的配置
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.7.4</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.4</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-yarn-api -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-api</artifactId>
<version>2.7.4</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-yarn-client -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-client</artifactId>
<version>2.7.4</version>
</dependency>接着将yarn-site.xml配置文件放到resources目录下：
具体实现代码其实很简单就是，通过yarnclient获取resourcemanager上 spark 或者 mapreduce的状态。
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.api.records.YarnApplicationState;
import org.apache.hadoop.yarn.client.api.YarnClient;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.YarnException;
import java.io.IOException;
import java.util.EnumSet;
import java.util.List;
public class client {
public static void main(String[] args){
Configuration conf = new YarnConfiguration();
YarnClient yarnClient = YarnClient.createYarnClient();
yarnClient.init(conf);
yarnClient.start();
try {
List<ApplicationReport> applications = yarnClient.getApplications(EnumSet.of(YarnApplicationState.RUNNING, YarnApplicationState.FINISHED));
System.out.println("ApplicationId ============> "+applications.get(0).getApplicationId());
System.out.println("name ============> "+applications.get(0).getName());
System.out.println("queue ============> "+applications.get(0).getQueue());
System.out.println("queue ============> "+applications.get(0).getUser());
} catch (YarnException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
yarnClient.stop();
}
}可以看到，我们可以通过YarnApplicationState设置状态，来过滤调一些我们不需要的任务状态。状态列表如下：
public enum YarnApplicationState {
/** Application which was just created. */
NEW,
/** Application which is being saved. */
NEW_SAVING,
/** Application which has been submitted. */
SUBMITTED,
/** Application has been accepted by the scheduler */
ACCEPTED,
/** Application which is currently running. */
RUNNING,
/** Application which finished successfully. */
FINISHED,
/** Application which failed. */
FAILED,
/** Application which was terminated by a user or admin. */
KILLED
}上述demo监控的是spark streaming 的状态，运行结果如下：
这样，我们通过app name字段可以获取到存活的 spark 等任务，然后通过比对我们要监控的任务列表，不存在的发出告警即可。
对于 spark streaming 或者 spark其他任务，可以通过一个配置来制定spark 任务在yarn上显示的name，设置的参数是
new SparkConf().setAppName(this.getClass.getName)this.getClass.getName该方式在yarn-client和 yarn-cluster有稍微的区别，具体可以测一下。
类名作为spark app的 name 还是很方便及便于识别的。
本文参与 腾讯云自媒体同步曝光计划，分享自微信公众号。
原始发表：2018-10-25，如有侵权请联系 cloudcommunity@tencent.com 删除
评论
登录后参与评论
推荐阅读
腾讯云开发者
