渣渣菜鸡的 ElasticSearch 源码解析 —— 启动流程(上)

前提

上篇文章写了 ElasticSearch 源码解析 —— 环境搭建 ,其中里面说了启动 打开 server 模块下的 Elasticsearch 类:org.elasticsearch.bootstrap.Elasticsearch,运行里面的 main 函数就可以启动 ElasticSearch 了,这篇文章讲讲启动流程,因为篇幅会很多,所以分了两篇来写。

启动流程

main 方法入口

可以看到入口其实是一个 main 方法,方法里面先是检查权限,然后是一个错误日志监听器(确保在日志配置之前状态日志没有出现 error),然后是 Elasticsearch 对象的创建,然后调用了静态方法 main 方法(18 行),并把创建的对象和参数以及 Terminal 默认值传进去。静态的 main 方法里面调用 elasticsearch.main 方法。

 1public static void main(final String[] args) throws Exception {         //1、入口
 2    // we want the JVM to think there is a security manager installed so that if internal policy decisions that would be based on the
 3    // presence of a security manager or lack thereof act as if there is a security manager present (e.g., DNS cache policy)
 4    System.setSecurityManager(new SecurityManager() {
 5        @Override
 6        public void checkPermission(Permission perm) {
 7            // grant all permissions so that we can later set the security manager to the one that we want
 8        }
 9    });
10    LogConfigurator.registerErrorListener();                            //
11    final Elasticsearch elasticsearch = new Elasticsearch();
12    int status = main(args, elasticsearch, Terminal.DEFAULT); //2、调用Elasticsearch.main方法
13    if (status != ExitCodes.OK) {
14        exit(status);
15    }
16}
17
18static int main(final String[] args, final Elasticsearch elasticsearch, final Terminal terminal) throws Exception {
19    return elasticsearch.main(args, terminal);  //3、command main
20}

因为 Elasticsearch 类是继承了 EnvironmentAwareCommand 类,EnvironmentAwareCommand 类继承了 Command 类,但是 Elasticsearch 类并没有重写 main 方法,所以上面调用的 elasticsearch.main 其实是调用了 Command 的 main 方法,代码如下:

 1/** Parses options for this command from args and executes it. */
 2public final int main(String[] args, Terminal terminal) throws Exception {
 3    if (addShutdownHook()) {                                                //利用Runtime.getRuntime().addShutdownHook方法加入一个Hook,在程序退出时触发该Hook
 4        shutdownHookThread = new Thread(() -> {
 5            try {
 6                this.close();
 7            } catch (final IOException e) {
 8                try (
 9                    StringWriter sw = new StringWriter();
10                    PrintWriter pw = new PrintWriter(sw)) {
11                    e.printStackTrace(pw);
12                    terminal.println(sw.toString());
13                } catch (final IOException impossible) {
14                    // StringWriter#close declares a checked IOException from the Closeable interface but the Javadocs for StringWriter
15                    // say that an exception here is impossible
16                    throw new AssertionError(impossible);
17                }
18            }
19        });
20        Runtime.getRuntime().addShutdownHook(shutdownHookThread);
21    }
22
23    beforeMain.run();
24
25    try {
26        mainWithoutErrorHandling(args, terminal);//4、mainWithoutErrorHandling
27    } catch (OptionException e) {
28        printHelp(terminal);
29        terminal.println(Terminal.Verbosity.SILENT, "ERROR: " + e.getMessage());
30        return ExitCodes.USAGE;
31    } catch (UserException e) {
32        if (e.exitCode == ExitCodes.USAGE) {
33            printHelp(terminal);
34        }
35        terminal.println(Terminal.Verbosity.SILENT, "ERROR: " + e.getMessage());
36        return e.exitCode;
37    }
38    return ExitCodes.OK;
39}

上面代码一开始利用一个勾子函数,在程序退出时触发该 Hook,该方法主要代码是 mainWithoutErrorHandling() 方法,然后下面的是 catch 住方法抛出的异常,方法代码如下:

 1/*** Executes the command, but all errors are thrown. */
 2void mainWithoutErrorHandling(String[] args, Terminal terminal) throws Exception {
 3    final OptionSet options = parser.parse(args);
 4    if (options.has(helpOption)) {
 5        printHelp(terminal);
 6        return;
 7    }
 8    if (options.has(silentOption)) {
 9        terminal.setVerbosity(Terminal.Verbosity.SILENT);
10    } else if (options.has(verboseOption)) {
11        terminal.setVerbosity(Terminal.Verbosity.VERBOSE);
12    } else {
13        terminal.setVerbosity(Terminal.Verbosity.NORMAL);
14    }
15    execute(terminal, options);//5、执行 EnvironmentAwareCommand 中的 execute(),(重写了command里面抽象的execute方法)
16}

上面的代码从 3 ~ 14 行是解析传进来的参数并配置 terminal,重要的 execute() 方法,执行的是 EnvironmentAwareCommand 中的 execute() (重写了 Command 类里面的抽象 execute 方法),从上面那个继承图可以看到 EnvironmentAwareCommand 继承了 Command,重写的 execute 方法代码如下:

 1@Override
 2protected void execute(Terminal terminal, OptionSet options) throws Exception {
 3    final Map<String, String> settings = new HashMap<>();
 4    for (final KeyValuePair kvp : settingOption.values(options)) {
 5        if (kvp.value.isEmpty()) {
 6            throw new UserException(ExitCodes.USAGE, "setting [" + kvp.key + "] must not be empty");
 7        }
 8        if (settings.containsKey(kvp.key)) {
 9            final String message = String.format(
10                Locale.ROOT, "setting [%s] already set, saw [%s] and [%s]",
11                kvp.key, settings.get(kvp.key), kvp.value);
12            throw new UserException(ExitCodes.USAGE, message);
13        }
14        settings.put(kvp.key, kvp.value);
15    }
16    //6、根据我们ide配置的 vm options 进行设置path.data、path.home、path.logs
17    putSystemPropertyIfSettingIsMissing(settings, "path.data", "es.path.data");
18    putSystemPropertyIfSettingIsMissing(settings, "path.home", "es.path.home");
19    putSystemPropertyIfSettingIsMissing(settings, "path.logs", "es.path.logs");
20
21    execute(terminal, options, createEnv(terminal, settings));//7、先调用 createEnv 创建环境
22    //9、执行elasticsearch的execute方法,elasticsearch中重写了EnvironmentAwareCommand中的抽象execute方法
23}

方法前面是根据传参去判断配置的,如果配置为空,就会直接跳到执行 putSystemPropertyIfSettingIsMissing 方法,这里会配置三个属性:path.data、path.home、path.logs 设置 es 的 data、home、logs 目录,它这里是根据我们 ide 配置的 vm options 进行设置的,这也是为什么我们上篇文章说的配置信息,如果不配置的话就会直接报错。下面看看 putSystemPropertyIfSettingIsMissing 方法代码里面怎么做到的:

 1/** Ensure the given setting exists, reading it from system properties if not already set. */
 2private static void putSystemPropertyIfSettingIsMissing(final Map<String, String> settings, final String setting, final String key) {
 3    final String value = System.getProperty(key);//获取key(es.path.data)找系统设置
 4    if (value != null) {
 5        if (settings.containsKey(setting)) {
 6            final String message =
 7                String.format(
 8                Locale.ROOT,
 9                "duplicate setting [%s] found via command-line [%s] and system property [%s]",
10                setting, settings.get(setting), value);
11            throw new IllegalArgumentException(message);
12        } else {
13            settings.put(setting, value);
14        }
15    }
16}

执行这三个方法后:

跳出此方法,继续看会发现 execute 方法调用了方法,

1 execute(terminal, options, createEnv(terminal, settings));

这里我们先看看 createEnv(terminal, settings) 方法:

1protected Environment createEnv(final Terminal terminal, final Map<String, String> settings) throws UserException {
2    final String esPathConf = System.getProperty("es.path.conf");//8、读取我们 vm options 中配置的 es.path.conf
3    if (esPathConf == null) {
4        throw new UserException(ExitCodes.CONFIG, "the system property [es.path.conf] must be set");
5    }
6    return InternalSettingsPreparer.prepareEnvironment(Settings.EMPTY, terminal, settings, getConfigPath(esPathConf));  //8、准备环境 prepareEnvironment
7}

读取我们 ide vm options 中配置的 es.path.conf,同上篇文章也讲了这个一定要配置的,因为 es 启动的时候会加载我们的配置和一些插件。这里继续看下上面代码第 6 行的 prepareEnvironment 方法:

 1public static Environment prepareEnvironment(Settings input, Terminal terminal, Map<String, String> properties, Path configPath) {
 2    // just create enough settings to build the environment, to get the config dir
 3    Settings.Builder output = Settings.builder();
 4    initializeSettings(output, input, properties);
 5    Environment environment = new Environment(output.build(), configPath);
 6
 7    //查看 es.path.conf 目录下的配置文件是不是 yml 格式的,如果不是则抛出一个异常
 8    if (Files.exists(environment.configFile().resolve("elasticsearch.yaml"))) {
 9        throw new SettingsException("elasticsearch.yaml was deprecated in 5.5.0 and must be renamed to elasticsearch.yml");
10    }
11
12    if (Files.exists(environment.configFile().resolve("elasticsearch.json"))) {
13        throw new SettingsException("elasticsearch.json was deprecated in 5.5.0 and must be converted to elasticsearch.yml");
14    }
15
16    output = Settings.builder(); // start with a fresh output
17    Path path = environment.configFile().resolve("elasticsearch.yml");
18    if (Files.exists(path)) {
19        try {
20            output.loadFromPath(path);  //加载文件并读取配置文件内容
21        } catch (IOException e) {
22            throw new SettingsException("Failed to load settings from " + path.toString(), e);
23        }
24    }
25
26    // re-initialize settings now that the config file has been loaded
27    initializeSettings(output, input, properties);          //再一次初始化设置
28    finalizeSettings(output, terminal);
29
30    environment = new Environment(output.build(), configPath);
31
32    // we put back the path.logs so we can use it in the logging configuration file
33    output.put(Environment.PATH_LOGS_SETTING.getKey(), environment.logsFile().toAbsolutePath().normalize().toString());
34    return new Environment(output.build(), configPath);
35}

准备的环境如上图,通过构建的环境查看配置文件 elasticsearch.yml 是不是以 yml 结尾,如果是 yaml 或者 json 结尾的则抛出异常(在 5.5.0 版本其他两种格式过期了,只能使用 yml 格式),然后加载该配置文件并读取里面的内容(KV结构)。

跳出 createEnv 方法,我们继续看 execute 方法吧。

EnvironmentAwareCommand 类的 execute 方法代码如下:

1protected abstract void execute(Terminal terminal, OptionSet options, Environment env) throws Exception;

这是个抽象方法,那么它的实现方法在 Elasticsearch 类中,代码如下:

 1@Override
 2protected void execute(Terminal terminal, OptionSet options, Environment env) throws UserException {
 3    if (options.nonOptionArguments().isEmpty() == false) {
 4        throw new UserException(ExitCodes.USAGE, "Positional arguments not allowed, found " + options.nonOptionArguments());
 5    }
 6    if (options.has(versionOption)) {
 7        final String versionOutput = String.format(
 8            Locale.ROOT,
 9            "Version: %s, Build: %s/%s/%s/%s, JVM: %s",
10            Version.displayVersion(Version.CURRENT, Build.CURRENT.isSnapshot()),
11            Build.CURRENT.flavor().displayName(),
12            Build.CURRENT.type().displayName(),
13            Build.CURRENT.shortHash(),
14            Build.CURRENT.date(),
15            JvmInfo.jvmInfo().version());
16        terminal.println(versionOutput);
17        return;
18    }
19
20    final boolean daemonize = options.has(daemonizeOption);
21    final Path pidFile = pidfileOption.value(options);
22    final boolean quiet = options.has(quietOption);
23
24    // a misconfigured java.io.tmpdir can cause hard-to-diagnose problems later, so reject it immediately
25    try {
26        env.validateTmpFile();
27    } catch (IOException e) {
28        throw new UserException(ExitCodes.CONFIG, e.getMessage());
29    }
30    try {
31        init(daemonize, pidFile, quiet, env);    //10、初始化
32    } catch (NodeValidationException e) {
33        throw new UserException(ExitCodes.CONFIG, e.getMessage());
34    }
35}

上面代码里主要还是看看 init(daemonize, pidFile, quiet, env); 初始化方法吧。

 1void init(final boolean daemonize, final Path pidFile, final boolean quiet, Environment initialEnv)
 2    throws NodeValidationException, UserException {
 3    try {
 4        Bootstrap.init(!daemonize, pidFile, quiet, initialEnv); //11、执行 Bootstrap 中的 init 方法
 5    } catch (BootstrapException | RuntimeException e) {
 6        // format exceptions to the console in a special way
 7        // to avoid 2MB stacktraces from guice, etc.
 8        throw new StartupException(e);
 9    }
10}

init 方法

Bootstrap 中的静态 init 方法如下:

  1static void init(
  2    final boolean foreground,
  3    final Path pidFile,
  4    final boolean quiet,
  5    final Environment initialEnv) throws BootstrapException, NodeValidationException, UserException {
  6    // force the class initializer for BootstrapInfo to run before
  7    // the security manager is installed
  8    BootstrapInfo.init();
  9
 10    INSTANCE = new Bootstrap();   //12、创建一个 Bootstrap 实例
 11
 12    final SecureSettings keystore = loadSecureSettings(initialEnv);//如果注册了安全模块则将相关配置加载进来
 13    final Environment environment = createEnvironment(foreground, pidFile, keystore, initialEnv.settings(), initialEnv.configFile());   //干之前干过的事情
 14    try {
 15        LogConfigurator.configure(environment);   //13、log 配置环境
 16    } catch (IOException e) {
 17        throw new BootstrapException(e);
 18    }
 19    if (environment.pidFile() != null) {
 20        try {
 21            PidFile.create(environment.pidFile(), true);
 22        } catch (IOException e) {
 23            throw new BootstrapException(e);
 24        }
 25    }
 26
 27    final boolean closeStandardStreams = (foreground == false) || quiet;
 28    try {
 29        if (closeStandardStreams) {
 30            final Logger rootLogger = ESLoggerFactory.getRootLogger();
 31            final Appender maybeConsoleAppender = Loggers.findAppender(rootLogger, ConsoleAppender.class);
 32            if (maybeConsoleAppender != null) {
 33                Loggers.removeAppender(rootLogger, maybeConsoleAppender);
 34            }
 35            closeSystOut();
 36        }
 37
 38        // fail if somebody replaced the lucene jars
 39        checkLucene();             //14、检查Lucene版本
 40
 41// install the default uncaught exception handler; must be done before security is initialized as we do not want to grant the runtime permission setDefaultUncaughtExceptionHandler
 42        Thread.setDefaultUncaughtExceptionHandler(
 43            new ElasticsearchUncaughtExceptionHandler(() -> Node.NODE_NAME_SETTING.get(environment.settings())));
 44
 45        INSTANCE.setup(true, environment);      //15、调用 setup 方法
 46
 47        try {
 48            // any secure settings must be read during node construction
 49            IOUtils.close(keystore);
 50        } catch (IOException e) {
 51            throw new BootstrapException(e);
 52        }
 53
 54        INSTANCE.start();         //26、调用 start 方法
 55
 56        if (closeStandardStreams) {
 57            closeSysError();
 58        }
 59    } catch (NodeValidationException | RuntimeException e) {
 60        // disable console logging, so user does not see the exception twice (jvm will show it already)
 61        final Logger rootLogger = ESLoggerFactory.getRootLogger();
 62        final Appender maybeConsoleAppender = Loggers.findAppender(rootLogger, ConsoleAppender.class);
 63        if (foreground && maybeConsoleAppender != null) {
 64            Loggers.removeAppender(rootLogger, maybeConsoleAppender);
 65        }
 66        Logger logger = Loggers.getLogger(Bootstrap.class);
 67        if (INSTANCE.node != null) {
 68            logger = Loggers.getLogger(Bootstrap.class, Node.NODE_NAME_SETTING.get(INSTANCE.node.settings()));
 69        }
 70        // HACK, it sucks to do this, but we will run users out of disk space otherwise
 71        if (e instanceof CreationException) {
 72            // guice: log the shortened exc to the log file
 73            ByteArrayOutputStream os = new ByteArrayOutputStream();
 74            PrintStream ps = null;
 75            try {
 76                ps = new PrintStream(os, false, "UTF-8");
 77            } catch (UnsupportedEncodingException uee) {
 78                assert false;
 79                e.addSuppressed(uee);
 80            }
 81            new StartupException(e).printStackTrace(ps);
 82            ps.flush();
 83            try {
 84                logger.error("Guice Exception: {}", os.toString("UTF-8"));
 85            } catch (UnsupportedEncodingException uee) {
 86                assert false;
 87                e.addSuppressed(uee);
 88            }
 89        } else if (e instanceof NodeValidationException) {
 90            logger.error("node validation exception\n{}", e.getMessage());
 91        } else {
 92            // full exception
 93            logger.error("Exception", e);
 94        }
 95        // re-enable it if appropriate, so they can see any logging during the shutdown process
 96        if (foreground && maybeConsoleAppender != null) {
 97            Loggers.addAppender(rootLogger, maybeConsoleAppender);
 98        }
 99        throw e;
100    }
101}

该方法主要有:

1、创建 Bootstrap 实例

2、如果注册了安全模块则将相关配置加载进来

3、创建 Elasticsearch 运行的必须环境以及相关配置, 如将 config、scripts、plugins、modules、logs、lib、bin 等配置目录加载到运行环境中

4、log 配置环境,创建日志上下文

5、检查是否存在 PID 文件,如果不存在,创建 PID 文件

6、检查 Lucene 版本

7、调用 setup 方法(用当前环境来创建一个节点)

setup 方法

 1private void setup(boolean addShutdownHook, Environment environment) throws BootstrapException {
 2    Settings settings = environment.settings();//根据环境得到配置
 3    try {
 4        spawner.spawnNativeControllers(environment);
 5    } catch (IOException e) {
 6        throw new BootstrapException(e);
 7    }
 8    initializeNatives(
 9        environment.tmpFile(),
10        BootstrapSettings.MEMORY_LOCK_SETTING.get(settings),
11        BootstrapSettings.SYSTEM_CALL_FILTER_SETTING.get(settings),
12        BootstrapSettings.CTRLHANDLER_SETTING.get(settings));
13    // initialize probes before the security manager is installed
14    initializeProbes();
15    if (addShutdownHook) {
16        Runtime.getRuntime().addShutdownHook(new Thread() {
17            @Override
18            public void run() {
19                try {
20                    IOUtils.close(node, spawner);
21                    LoggerContext context = (LoggerContext) LogManager.getContext(false);
22                    Configurator.shutdown(context);
23                } catch (IOException ex) {
24                    throw new ElasticsearchException("failed to stop node", ex);
25                }
26            }
27        });
28    }
29    try {
30        // look for jar hell
31        final Logger logger = ESLoggerFactory.getLogger(JarHell.class);
32        JarHell.checkJarHell(logger::debug);
33    } catch (IOException | URISyntaxException e) {
34        throw new BootstrapException(e);
35    }
36    // Log ifconfig output before SecurityManager is installed
37    IfConfig.logIfNecessary();
38    // install SM after natives, shutdown hooks, etc.
39    try {
40        Security.configure(environment, BootstrapSettings.SECURITY_FILTER_BAD_DEFAULTS_SETTING.get(settings));
41    } catch (IOException | NoSuchAlgorithmException e) {
42        throw new BootstrapException(e);
43    }
44    node = new Node(environment) {              //16、新建节点
45        @Override
46        protected void validateNodeBeforeAcceptingRequests(
47            final BootstrapContext context,
48            final BoundTransportAddress boundTransportAddress, List<BootstrapCheck> checks) throws NodeValidationException {
49            BootstrapChecks.check(context, boundTransportAddress, checks);
50        }
51    };
52}

上面代码最后就是 Node 节点的创建,这篇文章就不讲 Node 的创建了,下篇文章会好好讲一下 Node 节点的创建和正式启动 ES 节点的。

总结

这篇文章主要先把大概启动流程串通,因为篇幅较多所以拆开成两篇,先不扣细节了,后面流程启动文章写完后我们再单一的扣细节。

转载请务必注明原创地址为:http://www.54tianzhisheng.cn/2018/08/11/es-code02/

原文发布于微信公众号 - zhisheng(zhisheng_blog)

原文发表时间:2018-08-24

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏java思维导图

自己手写一个 SpringMVC 框架

前端框架很多,但没有一个框架称霸,后端框架现在Spring已经完成大一统.所以学习Spring是Java程序员的必修课. Spring 框架对于 Java 后端...

4166
来自专栏跟着阿笨一起玩NET

以读取博客园随笔备份为例 将xml 序列化成json,再序列化成对象

资源下载:http://files.cnblogs.com/codealone/ConsoleApplication2.zip

841
来自专栏屈定‘s Blog

造轮子--Excel报表工具

由于公司内部之前对于excel封装操作并不是很方便,而且对于特殊的需求不是很容易满足,这个月的任务是迁移部分业务小报表顺便重构下,因此这里造个轮子,便于导入和导...

1793
来自专栏拭心的安卓进阶之路

Android 进阶7:进程通信之 AIDL 的使用

记得 2015 年实习面试,笔试题里就有这道题:请介绍下 AIDL。 当时的我是懵逼的,只好老老实实空着。没想到后来面试时面试官大哥嘿嘿一笑说他也没用过这玩意,...

2068
来自专栏osc同步分享

springMVC

springmvc中有专用于页面跳转的controller,不会对请求做任何处理,直接跳转页面:     <!-- 对处理请求的controller进行映射 -...

2154
来自专栏菩提树下的杨过

spring mvc4:异常处理

前面学习过struts2的异常处理,今天来看下spring mvc4的异常处理: 一、Servlet配置文件修改 1 <bean id="excepti...

2115
来自专栏Java学习网

常见的 Java 错误及避免方法之第五集(每集10个错误后续持续发布)

当输入期间意外终止文件或流时,将抛出“EOFException”。 以下是抛出EOFException异常的一个示例,来自JavaBeat应用程序:

1423
来自专栏DT乱“码”

Memcached使用实例

package com.memcached.util; import java.io.BufferedWriter; import java.io.FileW...

2206
来自专栏技术墨客

Spring核心——IOC处理器扩展 原

Spring一直标注自己是一个非侵入式框架。非侵入式设计的概念并不新鲜,目标就是降低使用者和框架代码的耦合,毕竟框架的开发者和使用者几乎肯定不是同一个团队。Sp...

642
来自专栏菩提树下的杨过

dubbo/dubbox 增加原生thrift及avro支持

(facebook) thrift / (hadoop) avro / (google) probuf(grpc)是近几年来比较抢眼的高效序列化/rpc框架,d...

4038

扫码关注云+社区

领取腾讯云代金券