《Prometheus监控实战》第10章 探针监控
第10章 探针监控
- 探针监控可以在应用程序的外部进行探测。你可以查询应用程序的外部特征：它是否响应开放端口上的轮询请求并返回正确的数据或响应码。探针监控的一个示例是执行ICMP ping或echo检查并确认你已收到响应。这种类型的探针监控也称为黑盒监控，因为我们将内部应用程序视为黑盒
10.1 探针架构
- Prometheus通过运行Blackbox exporter来进行探测，该exporter会探测远程目标并暴露在本地端点上收集的任何时间序列，然后Prometheus作业将从这些端点中抓取指标
- 监控探针有三个约束：
- 它们需要能够访问到被探测的资源
- 探针需要放置在可以测试资源的正确位置上。例如，如果你正在测试对应用程序的外部访问，那么在防火墙后运行探针将不会验证此访问权限
- 探针exporter的位置能够被Prometheus服务器抓取
- 通常会将探针放置在企业网络之外分散的地理位置，以确保可以最大限度地覆盖故障检测和应用程序用户体验的数据惧
- 探针架构
10.2 Blackbox exporter
- Blackbox exporter（https://github.com/prometheus/blackbox_exporter）是一个在Apache 2.0许可证下的二进制Go语言应用程序。exporter允许通过HTTP、HTTPS、DNS、TCP和ICMP来探测端点。它的架构与其他exporter略有不同。在exporter内部，我们定义了一系列执行特定检查的模块，例如，检查Web服务器是否正在运行，或者DNS记录是否解析。在exporter运行时，它会在URL上暴露这些模块和API。Prometheus将目标和特定模块作为该URL的参数传递给这些目标。exporter执行检查并将生成的指标返回给Prometheus
10.3 安装exporter
- https://prometheus.io/download/#blackbox_exporter
- 代码清单：下载Blackbox exporter文件
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.darwin-amd64.tar.gz -O be.tar.gz
tar -zxf be.tar.gz
sudo cp -a blackbox_exporter-0.16.0.darwin-amd64/blackbox_exporter /usr/local/bin
- 代码清单：在Linux上检查Blackbox exporter版本
10.4 配置exporter
- 创建一个配置文件运行exporter
- 代码清单：文件prober.yml
sudo mkdir -p /etc/prober
sudo touch /etc/prober/prober.yml
- 代码清单：文件/etc/prober/prober.yml
modules:
http_2xx:
prober: http
timeout: 5s
http:
valid_status_codes: []
method: GET
icmp_check:
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: "ip4"
dns_examplecom_check:
prober: dns
dns:
preferred_ip_protocol: "ip4"
query_name: "www.example.com"
- 我们定义了三个检查：HTTP检查用于确保Web服务器在查询时返回2XX状态码；ICMP检查用于查看ping目标的结果；DNS检查用于查看DNS查询。让我们逐一研究每一个检查
- 提示：exporter示例配置对于帮助解释exporter的工作方式也很有用
modules:
http_2xx_example:
prober: http
timeout: 5s
http:
valid_http_versions: ["HTTP/1.1", "HTTP/2"]
valid_status_codes: [] # Defaults to 2xx
method: GET
headers:
Host: vhost.example.com
Accept-Language: en-US
Origin: example.com
no_follow_redirects: false
fail_if_ssl: false
fail_if_not_ssl: false
fail_if_body_matches_regexp:
- "Could not connect to database"
fail_if_body_not_matches_regexp:
- "Download the latest version here"
fail_if_header_matches: # Verifies that no cookies are set
- header: Set-Cookie
allow_missing: true
regexp: '.*'
fail_if_header_not_matches:
- header: Access-Control-Allow-Origin
regexp: '(\*|example\.com)'
tls_config:
insecure_skip_verify: false
preferred_ip_protocol: "ip4" # defaults to "ip6"
ip_protocol_fallback: false # no fallback to "ip6"
http_post_2xx:
prober: http
timeout: 5s
http:
method: POST
headers:
Content-Type: application/json
body: '{}'
http_basic_auth_example:
prober: http
timeout: 5s
http:
method: POST
headers:
Host: "login.example.com"
basic_auth:
username: "username"
password: "mysecret"
http_custom_ca_example:
prober: http
http:
method: GET
tls_config:
ca_file: "/certs/my_cert.crt"
tls_connect:
prober: tcp
timeout: 5s
tcp:
tls: true
tcp_connect_example:
prober: tcp
timeout: 5s
imap_starttls:
prober: tcp
timeout: 5s
tcp:
query_response:
- expect: "OK.*STARTTLS"
- send: ". STARTTLS"
- expect: "OK"
- starttls: true
- send: ". capability"
- expect: "CAPABILITY IMAP4rev1"
smtp_starttls:
prober: tcp
timeout: 5s
tcp:
query_response:
- expect: "^220 ([^ ]+) ESMTP (.+)$"
- send: "EHLO prober"
- expect: "^250-STARTTLS"
- send: "STARTTLS"
- expect: "^220"
- starttls: true
- send: "EHLO prober"
- expect: "^250-AUTH"
- send: "QUIT"
irc_banner_example:
prober: tcp
timeout: 5s
tcp:
query_response:
- send: "NICK prober"
- send: "USER prober prober prober :prober"
- expect: "PING :([^ ]+)"
send: "PONG ${1}"
- expect: "^:[^ ]+ 001"
icmp_example:
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: "ip4"
source_ip_address: "127.0.0.1"
dns_udp_example:
prober: dns
timeout: 5s
dns:
query_name: "www.prometheus.io"
query_type: "A"
valid_rcodes:
- NOERROR
validate_answer_rrs:
fail_if_matches_regexp:
- ".*127.0.0.1"
fail_if_all_match_regexp:
- ".*127.0.0.1"
fail_if_not_matches_regexp:
- "www.prometheus.io.\t300\tIN\tA\t127.0.0.1"
fail_if_none_matches_regexp:
- "127.0.0.1"
validate_authority_rrs:
fail_if_matches_regexp:
- ".*127.0.0.1"
validate_additional_rrs:
fail_if_matches_regexp:
- ".*127.0.0.1"
dns_soa:
prober: dns
dns:
query_name: "prometheus.io"
query_type: "SOA"
dns_tcp_example:
prober: dns
dns:
transport_protocol: "tcp" # defaults to "udp"
preferred_ip_protocol: "ip4" # defaults to "ip6"
query_name: "www.prometheus.io"
10.4.1 HTTP检查
- HTTP状态检查使用http探针。这个探针使用各种方法（如GET或POST）发出HTTP请求。我们为所有请求指定5秒的超时，然后将探针配置为发出GET请求。我们将valid_status_codes留空，它默认为任何2XX状态码。如果想验证是否返回了不同的状态码，则需要在此字段中指定数组形式的状态码
- http探针示例（http_probe）
# Accepted status codes for this probe. Defaults to 2xx.
[ valid_status_codes: <int>, ... | default = 2xx ]
# Accepted HTTP versions for this probe.
[ valid_http_versions: <string>, ... ]
# The HTTP method the probe will use.
[ method: <string> | default = "GET" ]
# The HTTP headers set for the probe.
headers:
[ <string>: <string> ... ]
# Whether or not the probe will follow any redirects.
[ no_follow_redirects: <boolean> | default = false ]
# Probe fails if SSL is present.
[ fail_if_ssl: <boolean> | default = false ]
# Probe fails if SSL is not present.
[ fail_if_not_ssl: <boolean> | default = false ]
# Probe fails if response body matches regex.
fail_if_body_matches_regexp:
[ - <regex>, ... ]
# Probe fails if response body does not match regex.
fail_if_body_not_matches_regexp:
[ - <regex>, ... ]
# Probe fails if response header matches regex. For headers with multiple values, fails if *at least one* matches.
fail_if_header_matches:
[ - <http_header_match_spec>, ... ]
# Probe fails if response header does not match regex. For headers with multiple values, fails if *none* match.
fail_if_header_not_matches:
[ - <http_header_match_spec>, ... ]
# Configuration for TLS protocol of HTTP probe.
tls_config:
[ <tls_config> ]
# The HTTP basic authentication credentials for the targets.
basic_auth:
[ username: <string> ]
[ password: <secret> ]
# The bearer token for the targets.
[ bearer_token: <secret> ]
# The bearer token file for the targets.
[ bearer_token_file: <filename> ]
# HTTP proxy server to use to connect to the targets.
[ proxy_url: <string> ]
# The IP protocol of the HTTP probe (ip4, ip6).
[ preferred_ip_protocol: <string> | default = "ip6" ]
[ ip_protocol_fallback: <boolean> | default = true ]
# The body of the HTTP request used in probe.
body: [ <string> ]
- 代码清单：验证状态码
valid_status_codes: ['200', '304']
10.4.2 ICMP检查
- 第二项检查是通过ICMP查看ping目标的结果。将探针设置为icmp并指定超时为5秒，并将协议配置为ip4
- icmp探针示例
# The IP protocol of the ICMP probe (ip4, ip6).
[ preferred_ip_protocol: <string> | default = "ip6" ]
[ ip_protocol_fallback: <boolean | default = true> ]
# The source IP address.
[ source_ip_address: <string> ]
# Set the DF-bit in the IP-header. Only works with ip4 and on *nix systems.
[ dont_fragment: <boolean> | default = false ]
# The size of the payload.
[ payload_size: <int> ]
10.4.3 DNS检查
- 最后一项检查使用dns探针确定DNS条目是否解析。在这种情况下，目标是我们想要解析的DNS服务器。同样要配置协议为ip4，然后指定一个查询
query_name: "www.example.com"
- 将检查指定域名的DNS是否会解析。接着向目标发送一个ANY查询类型的请求，DNS探针结果取决于查询返回的状态码。默认的成功条件是收到NOERROR响应代码
- dns探针示例
# The IP protocol of the DNS probe (ip4, ip6).
[ preferred_ip_protocol: <string> | default = "ip6" ]
[ ip_protocol_fallback: <boolean | default = true> ]
# The source IP address.
[ source_ip_address: <string> ]
[ transport_protocol: <string> | default = "udp" ] # udp, tcp
query_name: <string>
[ query_type: <string> | default = "ANY" ]
# List of valid response codes.
valid_rcodes:
[ - <string> ... | default = "NOERROR" ]
validate_answer_rrs:
fail_if_matches_regexp:
[ - <regex>, ... ]
fail_if_all_match_regexp:
[ - <regex>, ... ]
fail_if_not_matches_regexp:
[ - <regex>, ... ]
fail_if_none_matches_regexp:
[ - <regex>, ... ]
validate_authority_rrs:
fail_if_matches_regexp:
[ - <regex>, ... ]
fail_if_all_match_regexp:
[ - <regex>, ... ]
fail_if_not_matches_regexp:
[ - <regex>, ... ]
fail_if_none_matches_regexp:
[ - <regex>, ... ]
validate_additional_rrs:
fail_if_matches_regexp:
[ - <regex>, ... ]
fail_if_all_match_regexp:
[ - <regex>, ... ]
fail_if_not_matches_regexp:
[ - <regex>, ... ]
fail_if_none_matches_regexp:
[ - <regex>, ... ]
10.5 启动exporter
- 已经定义了三项检查，接下来启动exporter
- 代码清单：prober.yml
modules:
http_2xx:
prober: http
timeout: 5s
http:
valid_status_codes: []
method: GET
icmp_check:
prober: icmp
timeout: 5s
icmp:
preferred_ip_protocol: "ip4"
dns_examplecom_check:
prober: dns
dns:
preferred_ip_protocol: "ip4"
query_name: "www.example.com"
dns_tcp_example:
prober: dns
dns:
transport_protocol: "tcp" # defaults to "udp"
preferred_ip_protocol: "ip4" # defaults to "ip6"
query_name: "www.prometheus.io"
- 代码清单：启动exporter
sudo blackbox_exporter --config.file="/etc/prober/prober.yml"
- exporter在端口9115上运行，可以在http://localhost:9115上浏览控制台页面
- 控制台包含exporter自身的指标，以便同时监控exporter本身
10.6 创建Prometheus作业
- 现在创建Prometheus作业抓取exporter指标
- 代码清单：http_probes作业
- job_name: 'http_probe'
metrics_path: /probe
params:
module: [http_2xx_check]
file_sd_configs:
- files:
- 'targets/probes/http_probes.json'
refresh_interval: 5m
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: prober:9115
- 代码清单：http_probes目标
[{
"targets": [
"http://www.example.com",
"https://www.example.com",
""
]
}]
- 监控目标网站www.example.com的HTTP和HTTPS两种协议
- Prometheus如何找到exporter呢？我们使用relabel_configs覆盖目标的__address__标签以指定exporter的主机名。使用以下三个重新标记
- 重新标记通过将____address____标签（当前目标的地址）写入__param_target标签来创建参数
- 重新标记将__param_target标签写入instance标签
- 最后使用exporter的主机名（和端口）来重新标记____address__标签，在示例中为prober.example.com
- 重新标记会为抓取构造如下的URL
- 浏览此URL查看返回的指标
- 代码清单：http_2xx_check指标
- 重新加载或启动Prometheus，在控制台可看到这些作业的指标
本文参与 腾讯云自媒体同步曝光计划，分享自微信公众号。
原始发表：2019-12-16，如有侵权请联系 cloudcommunity@tencent.com 删除
评论
登录后参与评论
推荐阅读
腾讯云开发者
