我们知道Elasticsearch从7开始开放了大量X-Pack的基础安全功能,默认情况下启动的时候就会配置开启安全功能,启用SSL,连接Elasticsearch需要采用HTTPS。
这种情况下Logstash应该如何连接上Elasticsearch呢?本文从头开始演示从logstash搭建到配置连接Elasticsearch,配置SSL。Elasticsearch和Logstash均以8.4.3版本为例,假设以成功搭建了一个Elasticsearch集群,HTTPS访问url为:https://192.168.56.11:9200
下载logstash-8.4.3-linux-x86_64.tar.gz,解压缩logstash-8.4.3.tar.gz,所有涉及配置的文件都在config目录,执行脚本在bin目录,进入logstash主目录,执行以下命令启动Logstash:
./bin/logstash -e 'input { stdin { } } output { stdout {} }'
这是一个最简单的信息采集,数据来源stdin代表标准输入,stdout代表标准输出。
输入"123456",可以看到在标准输出中输出以下信息:
{
"host" => {
"hostname" => "node1"
},
"message" => "12123456",
"@version" => "1",
"event" => {
"original" => "12123456"
},
"@timestamp" => 2023-02-02T05:09:39.011320047Z
}
如果要将信息存储到Elasticsearch,那么需要修改logstash的配置。在config目录下拷贝logstash-sample.conf并修改名字:
[elastic@node1 config]$ cp logstash-sample.conf logstash.conf
将logstash.conf的内容修改为如下所示:
input {
stdin {}
}
output {
stdout{
codec => rubydebug
}
elasticsearch {
hosts => ["https://192.168.56.11:9200"]
index => "stdin-%{+YYYY.MM.dd}"
user => "elastic"
password => "LYePogNEis=ogbMaUzmJ"
}
}
指定配置文件重新启动logstash
[elastic@node1 logstash-8.4.3]$ ./bin/logstash -f config/logstash.conf
提示以下错误:
[2023-02-02T14:11:41,073][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@192.168.56.11:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://192.168.56.11:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
这是因为Elasticsearch配置了SSL,所以需要在logstash中配置访问的证书,将logstash.conf修改如下:
input {
stdin {}
}
output {
stdout{
codec => rubydebug
}
elasticsearch {
hosts => ["https://192.168.56.11:9200"]
index => "stdin-%{+YYYY.MM.dd}"
user => "elastic"
password => "LYePogNEis=ogbMaUzmJ"
ssl_certificate_verification => true
truststore => "/home/elastic/elasticsearch-8.4.3/config/certs/http.p12"
truststore_password => "EDkicmcvTIaby_aFALRl3w"
}
}
其中ssl_certificate_verification => true
代表启用SSL,truststore配置的elasticsearch首次启动生成的证书,它是一个使用PKCS#12(公钥密码标准#12)加密的数字证书,存放在elasticsearch主目录下的config/certs目录,而truststore_password是truststore的密码,可以采用bin目录下的elasticsearch-keystore工具获取到。
[elastic@node1 elasticsearch-8.4.3]$ ./bin/elasticsearch-keystore list
warning: ignoring JAVA_HOME=/opt/jdk-17.0.5; using bundled JDK
keystore.seed
xpack.security.http.ssl.keystore.secure_password
xpack.security.transport.ssl.keystore.secure_password
xpack.security.transport.ssl.truststore.secure_password
[elastic@node1 elasticsearch-8.4.3]$ ./bin/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password
warning: ignoring JAVA_HOME=/opt/jdk-17.0.5; using bundled JDK
EDkicmcvTIaby_aFALRl3w
使用_cat API查看Elasticsearch中的索引,可以看到已经创建出来索引stdin-2023.02.02。
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open stdin-2023.02.02 aPtwNfIgRqihTqG8bmDgVg 1 1 1 0 6.1kb 6.1kb
在Kibana中查询,可以看到我们从标准输入中输入的"Hello World!"已经存储到了Elasticsearch中。