首页
学习
活动
专区
工具
TVP
发布
社区首页 >问答首页 >如何配置Spark 的log4j的日志?

如何配置Spark 的log4j的日志?

提问于 2020-05-28 10:34:15
回答 1关注 0查看 1.7K

我使用的是CDH5.16安装的集群。跑的是spark程序,想要自定义log4j输出日志到客户端目录

配置的log4j如下

代码语言:javascript
复制
log4j.rootCategory=INFO, console,FILE
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=WARN
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=WARN
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

log4j.appender.FILE = org.apache.log4j.RollingFileAppender
log4j.appender.FILE.Threshold=DEBUG
#log4j.appender.FILE.File=./log/rzp.log
log4j.appender.FILE.File=/home/rzpt/logs/spark.log
log4j.appender.logFile.Encoding = UTF-8
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.ConversionPattern=[%-5p] [%d{yyyy-MM-dd HH:mm:ss}] [%C{1}:%M:%L] %m%n
log4j.appender.FILE.MaxFileSize=10mb

这个log4j配置文件我在项目中放在了resource文件下,服务器上也放了一份(因为网上找了答案两种都有),然后这些命令我都试过了,但是全部都没有日志输出

代码语言:js
复制
spark2-submit \
    --class com.nari.sgp.amc.measStandAssess.aurSum.AurSumMain \
    --master yarn \
    --deploy-mode cluster \
    --files "./log4j.properties" \
    ./sgp-1.0.jar
    
    spark2-submit --class com.nari.sgp.amc.measStandAssess.aurSum.AurSumMain --master local[*]  \
    --deploy-mode client sgp-1.0.jar

spark2-submit --class com.nari.sgp.amc.measStandAssess.aurSum.AurSumMain \
--files "./log4j.properties" \
--driver-java-options "-Dlog4j.debug=true -Dlog4j.configuration=log4j.properties" \
--conf "spark.executor.extraJavaOptions=-Dlog4j.debug=true -Dlog4j.configuration=log4j.properties" \
--master yarn --deploy-mode cluster  sgp-1.0.jar

spark2-submit --class com.nari.sgp.amc.measStandAssess.aurSum.AurSumMain \
--files "./log4j.properties" \
--driver-java-options "-Dlog4j.debug=true -Dlog4j.configuration=log4j.properties" \
--conf spark.driver.extraJavaOptions='-Dlog4j.configuration=file:/opt/centralLogs/conf/log4j.properties' \
--master yarn --deploy-mode cluster  \
sgp-1.0.jar
相关文章

相似问题

相关问答用户
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档