Cloudera Agent服务异常分析
1.异常描述
在Cloudera Manager的主机列表界面查看cdh05.fayson.com节点显示异常，节点上一次检测时间超过15s
运行主机检查提示该节点显示如下异常
登录cdh05.fayson.com节点查看cloudera-scm-agent服务状态
在命令行重启cloudera-scm-agent服务无任何提示，/var/log/cloudera-scm-agent目录下的log也未正常输出
无最新的日志输出
2.异常分析及处理
1.在解决故障的过程中，检查了该节点的防火墙、SELinux、磁盘空间大小，防火墙和SELinux处于关闭状态，磁盘空间充足。
排除防火墙、SELinux和磁盘空间不足导致Cloudera-scm-agent服务异常启动原因。
2.由于cloudera-scm-agent服务是被systemctl管理，需要检查系统的日志文件(/var/log/messages)，查看是否有关服务启动失败的异常信息
Aug 30 15:33:44 cdh05 systemd: Stopping LSB: Cloudera SCM Agent...
Aug 30 15:33:44 cdh05 cloudera-scm-agent: which: no python in (/sbin:/usr/sbin:/bin:/usr/bin)
Aug 30 15:33:44 cdh05 cloudera-scm-agent: Stopping cloudera-scm-agent: /usr/bin/dirname: invalid option -- 'd'
Aug 30 15:33:44 cdh05 cloudera-scm-agent: Try '/usr/bin/dirname --help' for more information.
Aug 30 15:33:44 cdh05 cloudera-scm-agent: [FAILED]#015Usage: grep [OPTION]... PATTERN [FILE]...
Aug 30 15:33:44 cdh05 cloudera-scm-agent: Try 'grep --help' for more information.
Aug 30 15:33:44 cdh05 cloudera-scm-agent: /etc/rc.d/init.d/cloudera-scm-agent: line 163: $local_pidfile: ambiguous redirect
Aug 30 15:33:44 cdh05 systemd: cloudera-scm-agent.service: control process exited, code=exited status=1
Aug 30 15:33:44 cdh05 systemd: Unit cloudera-scm-agent.service entered failed state.
Aug 30 15:33:44 cdh05 systemd: cloudera-scm-agent.service failed.
Aug 30 15:33:44 cdh05 systemd: Starting LSB: Cloudera SCM Agent...
Aug 30 15:33:44 cdh05 cloudera-scm-agent: which: no python in (/sbin:/usr/sbin:/bin:/usr/bin)
Aug 30 15:33:44 cdh05 cloudera-scm-agent: cloudera-scm-agent is already running
Aug 30 15:33:44 cdh05 systemd: Started LSB: Cloudera SCM Agent.
Aug 30 15:34:01 cdh05 systemd: Started Session 3154 of user root.
Aug 30 15:34:01 cdh05 systemd: Starting Session 3154 of user root.（可左右滑动）
在系统日志中可以看到在/usr/bin;/sbin;/bin;/usr/sbin等目录下未找到python命令导致。
3.查看/usr/bin目录下python
可以看到该目录下有python2的软连并未有python的命令。查看其它正常节点该目录的python
4.在cdh05.fayson.com节点上修复python的命令，执行如下命令创建软连
[root@cdh05 bin]# ln -s /usr/bin/python2 /usr/bin/python
[root@cdh05 bin]# ll /usr/bin/python*（可左右滑动）
5.修复python命令后，重启cloudera-scm-agent服务
[root@cdh05 bin]# systemctl restart cloudera-scm-agent
[root@cdh05 bin]# systemctl status cloudera-scm-agent（可左右滑动）
服务启动正常，关于cloudera-scm-agent服务状态显示“active(exited)”问题可参考Fayson前面的文章《Cloudera Manager Server服务在RedHat7状态显示异常分析》
6.查看Cloudera Manager的主机列表显示cdh05.fayson.com状态正常
7.主机检查运行成功
3.总结
cloudera-scm-agent服务依赖python环境，默认使用会在/usr/bin;/usr/sbin;/sbin;/bin目录下查找python的命令，如果未找到会到最后cloudera-scm-agent服务运行失败。