Feature Overview
The log function offers the capability to gather, search, and export operational logs of components, supporting keyword searches for core service logs and node system logs within the current cluster. This allows for swift examination of crucial service logs without the necessity of logging into the node.
Instructions
1. Log in to the EMR Console, and in the cluster list, click on the corresponding cluster ID/Name to access the cluster details page.
2. In the cluster details page, select Cluster Monitoring > Logs to filter and view log content based on the current cluster, log files, node IP, and time range.
Alternatively, in the cluster details page, select Cluster Services>Component Card>Role Management, view the role list, select a specific node IP and click to jump to the node monitoring metrics display page. Click on Role Logs to navigate to the log search page.
Click on Node IP to navigate to the corresponding node status page. Click on Log Source to navigate to the corresponding node monitoring metrics display page.
Keyword Explanation:
Supports full-text keyword search.
Supports search using special characters - . * > < = ! () {} /.
Supports phrase search, for example:
address=/ip:port.3. When troubleshooting, it is often necessary to pay attention to the context of the keyword in the logs. On the log search page, click on View Context to access the log context page.
4. Supports filtering log results and exporting source logs, which can be downloaded locally.
Supported service types for log search.
Note
Currently, only a 30-day log search is supported.
If the cluster has not enabled log collection and you wish to enable it, please contact your dedicated after-sales service.
Only a select few services have audit log collection enabled by default. If your cluster requires the activation of audit log collection, please reach out to your dedicated after-sales support.
Component | Role | Logs | Note |
HDFS | NameNode | /data/emr/hdfs/logs/hadoop-hadoop-namenode.log | Runtime logs of NameNode |
| NNAudit | /data/emr/hdfs/logs/nn-audit.log | Audit logs of NameNode |
| ZKFC | /data/emr/hdfs/logs/hadoop-hadoop-zkfc.log | Runtime logs of ZKFC |
| ZKFCAudit | /data/emr/hdfs/logs/SecurityAuth-zkfc-audit.log | Audit logs of ZKFC |
| DataNode | /data/emr/hdfs/logs/hadoop-hadoop-datanode.log | Runtime logs of DataNode |
| DNAudit | /data/emr/hdfs/logs/SecurityAuth-datanode-audit.log | Audit logs of DataNode |
| JournalNode | /data/emr/hdfs/logs/hadoop-hadoop-journalnode.log | Runtime logs of JournalNode |
| JNAudit | /data/emr/hdfs/logs/SecurityAuth-journalnode-audit.log | Audit logs of JournalNode |
| DFSRouter | /data/emr/hdfs/logs/hadoop-hadoop-dfsrouter.log | Runtime logs of DFSRouter |
YARN | ResourceManager | /data/emr/yarn/logs/yarn-hadoop-resourcemanager.log | Runtime logs of ResourceManager |
| RMAudit | /data/emr/yarn/logs/ rm-audit.log | Audit logs of ResourceManager |
| NodeManager | /data/emr/yarn/logs/yarn-hadoop-nodemanager.log | Runtime logs of NodeManager |
| NMAudit | /data/emr/yarn/logs/nm-audit.log | Audit logs of NodeManager |
| JobHistoryServer | /data/emr/yarn/logs/mapred-hadoop-historyserver.log | Runtime logs of JobHistoryServer |
| TimelineServer | /data/emr/yarn/logs/yarn-hadoop-timelineserver.log | Runtime logs of TimelineServer |
HBase | HMaster | /data/emr/hbase/logs/hbase-hadoop-master.log | Runtime logs of HMaster |
| HMAudit | /data/emr/hbase/logs/SecurityAuth-hmaster-audit.log | Audit logs of HMaster |
| ThriftServer | /data/emr/hbase/logs/hbase-hadoop-thrift.log | Runtime logs of ThriftServer |
| RegionServer | /data/emr/hbase/logs/hbase-hadoop-regionserver.log | Runtime logs of RegionServer |
| RSAudit | /data/emr/hbase/logs/SecurityAuth-regionserver-audit.log | Audit logs of RegionServer |
ClickHouse | ClickHouse-server | /data/clickhouse/clickhouse-server/logs/clickhouse-server.log | Runtime logs of ClickHouse-server |
Druid | Broker | /data/emr/druid/var/log/druid/broker.log | Runtime logs of Broker |
| Coordinator | /data/emr/druid/var/log/druid/coordinator.log | Runtime logs of Coordinator |
| Router | /data/emr/druid/var/log/druid/router.log | Runtime logs of Router |
| Overload | /data/emr/druid/var/log/druid/overload.log | Runtime logs of Overload |
| Historical | /data/emr/druid/var/log/druid/historical.log | Runtime logs of Historical |
| MiddleManager | /data/emr/druid/var/log/druid/middleManager.log | Runtime logs of MiddleManager |
Zookeeper | Zookeeper | /data/emr/zookeeper/logs/zookeeper-root-server.log | Runtime logs of Zookeeper |
| ZKAudit | /data/emr/zookeeper/logs/zookeeper-audit.log | Audit logs of Zookeeper |
Hive | HiveServer2 | /data/emr/hive/logs/hadoop-hive | Runtime logs of HiveServer2 |
| HS2Audit | /data/emr/hive/logs/hadoop_hive_server2_audit | Audit logs of HiveServer2 |
| HMSAudit | /data/emr/hive/logs/hadoop-hivemetastore | Audit logs of HiveMetaStore |
KUDU | KuduMaster | /data/emr/kudu/logs/kudu-master.WARNING | Runtime logs of KuduMaster |
| KuduServer | /data/emr/kudu/logs/kudu-tserver.WARNING | Runtime logs of KuduServer |
Alluxio | AlluxioMaster | /data/emr/alluxio/logs/master.log | Runtime logs of AlluxioMaster |
| AlluxioWorker | /data/emr/alluxio/logs/worker.log | Runtime logs of AlluxioWorker |
Ranger | EmbeddedServer | /data/emr/ranger/logs/ranger-admin.log | Runtime logs of EmbeddedServer |
CosRanger | CosRangerServer | /usr/local/service/cosranger/log/info.log | Runtime logs of CosRanger |
Impala | Catalogd | /data/emr/impala/logs/catalogd.INFO | Runtime logs of Catalogd |
| Statestored | /data/emr/impala/logs/statestored.INFO | Runtime logs of Statestored |
| Impalad | /data/emr/impala/logs/impalad.INFO | Runtime logs of Impalad |
Spark | HistoryServer | /data/emr/spark/logs/spark-hadoop.log | Runtime logs of HistoryServer |
Kylin | Kylin | /data/emr/kylin/logs/kylin.log | Runtime logs of Kylin |
Zeppelin | ZeppelinServer | /data/emr/zepplin/logs/zeppelin-hadoop.log | Runtime logs of ZeppelinServer |
Knox | Gateway | /data/emr/knox/logs/gateway.log | Runtime logs of Gateway |
Doris | BrokerBootstrap | /data/emr/doris/broker/log/apache_hdfs_broker.log | Runtime logs of BrokerBootstrap |
| PaloFe | /data/emr/doris/fe/log/fe.log | Runtime logs of PaloFe |
| PaloBe | /data/emr/doris/be/log/be.INFO | Runtime logs of PaloBe |
Kafka | Kafka | /user/local/service/kafka/logs/server.log | Runtime logs of Kafka |
Kyuubi | KyuubiServer | /data/emr/kyuubi/logs/kyuubi.log /data/emr/kyuubi/logs/kyuubi-hadoop-org.apache.kyuubi.server.KyuubiServer-hostIp.out | Runtime logs of KyuubiServer |
Trino | Coordinator | /data/emr/trino/logs/coordinator.log | Runtime logs of Coordinator |
| Worker | /data/emr/trino/logs/worker.log | Runtime logs of the Worker |
StarRocks | FE | /data/emr/starrocks/fe/log/fe.log | Runtime logs of FE |
| BE | /data/emr/starrocks/be/log/be.INFO | Runtime logs of BE |
| Broker | /data/emr/starrocks/broker/log/apache_hdfs_broker.log | Runtime logs of Broker |
| CN | /data/emr/starrocks/be/log/cn.INFO | Runtime logs of CN |
The service supports the minimum log level.
Sevice | Default collection of the lowest log level. |
Impala、Kudu | INFO |
Additional Services | WARN |
Query rules for the lowest log level.
Minimum log level | Queryable log levels. |
INFO | INFO、WARN、ERROR、FATAL |
WARN | WARN、ERROR、FATAL |
ERROR | ERROR、FATAL |
FATAL | FATAL |