## 前言

Prometheus 的规则分为两类：Recording Rules（预计算指标）和 Alerting Rules（告警规则）。写好规则是做监控告警的核心技能。本篇详解两类规则的语法、设计原则和常见模式。

---

## 一、Recording Rules：预计算指标

### 为什么需要 Recording Rules

```promql
# 复杂查询
sum(rate(http_requests_total{method="GET"}[5m])) by (service, status) 
  / 
sum(rate(http_requests_total[5m])) by (service) * 100

# 问题：
# 1. 每次查询都要重新计算
# 2. Dashboard 频繁查询 → Prometheus 负载高
# 3. 范围查询更慢（要逐点计算）
```

```promql
# Recording Rule 预计算
# http_request_error_rate:precomputed → 存入 TSDB
# 查询时直接读预计算结果，O(1)
```

### 语法

```yaml
# rules/recording.yml
groups:
  - name: http_metrics
    interval: 30s          # 每 30 秒评估一次
    rules:
      - record: job:http_requests:rate5m       # 指标名
        expr: sum by (job)(rate(http_requests_total[5m]))
        # 可选：添加标签
        labels:
          team: platform
      
      - record: job:http_requests:error_rate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job) 
          / 
          sum(rate(http_requests_total[5m])) by (job) * 100
      
      - record: instance:cpu_usage:ratio
        expr: |
          1 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance))
```

### 命名约定

```
规则名格式：
  <scope>:<metric>:<operation>

示例：
  job:http_requests:rate5m          # 范围:指标:操作
  instance:cpu_usage:ratio           # 实例级
  cluster:memory_usage:percentage   # 集群级
```

### 使用预计算指标

```promql
# 查询预计算结果
job:http_requests:rate5m

# Dashboard 查询更快
job:http_requests:error_rate > 5

# 告警规则用预计算指标
expr: job:http_requests:error_rate > 5
```

### 完整示例

```yaml
# rules/recording_rules.yml
groups:
  # ===== HTTP 指标 =====
  - name: http_recording
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job, method)(rate(http_requests_total[5m]))
      
      - record: job:http_requests:rate1m
        expr: sum by (job, method)(rate(http_requests_total[1m]))
      
      - record: job:http_requests:error_rate5m
        expr: |
          sum by (job)(rate(http_requests_total{status=~"5.."}[5m])) 
          / sum by (job)(rate(http_requests_total[5m])) * 100
      
      - record: job:http_request_duration:p99
        expr: |
          histogram_quantile(0.99, 
            sum by (job, le)(rate(http_request_duration_seconds_bucket[5m])))
      
      - record: job:http_request_duration:p95
        expr: |
          histogram_quantile(0.95,
            sum by (job, le)(rate(http_request_duration_seconds_bucket[5m])))
  
  # ===== 资源指标 =====
  - name: resource_recording
    interval: 30s
    rules:
      - record: instance:cpu_usage:ratio
        expr: 1 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])))
      
      - record: instance:memory_usage:ratio
        expr: 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
      
      - record: instance:disk_usage:ratio
        expr: 1 - (node_filesystem_avail_bytes / node_filesystem_size_bytes)
      
      - record: instance:network_rx:rate5m
        expr: rate(node_network_receive_bytes_total{device!~"lo|docker.*|veth.*"}[5m]) * 8
      
      - record: instance:network_tx:rate5m
        expr: rate(node_network_transmit_bytes_total{device!~"lo|docker.*|veth.*"}[5m]) * 8
  
  # ===== Kubernetes 指标 =====
  - name: k8s_recording
    interval: 30s
    rules:
      - record: namespace:pod_count
        expr: count by(namespace)(kube_pod_status_phase{phase="Running"})
      
      - record: namespace:cpu_request:total
        expr: sum by(namespace)(kube_pod_container_resource_requests{resource="cpu"})
      
      - record: namespace:memory_usage:total
        expr: sum by(namespace)(container_memory_working_set_bytes{container!=""})
```

---

## 二、Alerting Rules：告警规则

### 语法

```yaml
# rules/alerting.yml
groups:
  - name: alerting_rules
    interval: 15s
    rules:
      - alert: HighCPUUsage           # 告警名（唯一）
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m                        # 持续 5 分钟才告警
        labels:
          severity: warning           # 告警标签
        annotations:
          summary: "CPU usage > 80% on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | printf \"%.2f\" }}% (threshold: 80%)"
          runbook_url: "https://wiki.example.com/runbooks/high-cpu"
```

### 关键字段

| 字段 | 说明 |
|------|------|
| alert | 告警名称，唯一标识 |
| expr | PromQL 表达式，返回非空则触发 |
| for | 持续时间，防止瞬时波动误告警 |
| labels | 附加标签，用于路由 |
| annotations | 描述信息，支持模板 |

### for 的三阶段

```
告警生命周期：
  Pending（等待中）→ Firing（触发中）→ Resolved（已恢复）

  expr 成立 → Pending → for 时间后仍成立 → Firing → 发送告警
  expr 不成立 → Resolved → 发送恢复通知（send_resolved=true）
```

```
示例：for: 5m

  10:00  expr 成立 → Pending
  10:03  expr 仍成立 → 继续 Pending
  10:05  仍成立 → Firing → 发送告警
  10:08  expr 不成立 → Resolved → 发送恢复通知
```

---

## 三、告警规则最佳实践

### 1. 分级告警

```yaml
groups:
  - name: severity-based
    rules:
      # Critical：立即处理
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.job }} is down"
      
      # Warning：关注但不紧急
      - alert: HighCPU
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
      
      # Info：仅记录
      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 70
        for: 30m
        labels:
          severity: info
        annotations:
          summary: "Disk space low on {{ $labels.instance }}"
```

### 2. 使用预计算指标

```yaml
# ❌ 直接复杂查询（慢）
- alert: HighErrorRate
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
    / sum(rate(http_requests_total[5m])) by (service) * 100 > 5
  for: 5m

# ✅ 用 Recording Rule（快）
- alert: HighErrorRate
  expr: job:http_requests:error_rate5m > 5
  for: 5m
```

### 3. 注解模板

```yaml
- alert: HighMemory
  expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Memory usage > 90% on {{ $labels.instance }}"
    description: "Memory usage: {{ $value | printf \"%.1f\" }}%, available: {{ with printf \"node_memory_MemAvailable_bytes{instance=\\\"%s\\\"}\" $labels.instance | query }}{{ . | first | value | humanize1024 }}{{ end }}"
    dashboard: "https://grafana.example.com/d/node?var-instance={{ $labels.instance }}"
    runbook: "https://wiki.example.com/runbooks/high-memory"
```

### 4. 避免告警风暴

```yaml
# 使用 for 防止瞬时波动
- alert: PodCrashLooping
  expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
  for: 15m          # 持续 15 分钟才告警
  labels:
    severity: warning

# 使用 count 控制批量告警
- alert: ManyServicesDown
  expr: count(up == 0) > 3    # 超过 3 个服务宕机才告警
  for: 2m
  labels:
    severity: critical
```

---

## 四、完整告警规则集

```yaml
# rules/alerting_rules.yml
groups:
  # ===== 基础设施告警 =====
  - name: infrastructure
    rules:
      # 主机宕机
      - alert: NodeDown
        expr: up{job="node-exporter"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Node {{ $labels.instance }} is down"
      
      # CPU 高
      - alert: HighCPU
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "CPU > 80% on {{ $labels.instance }}"
      
      # 内存高
      - alert: HighMemory
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Memory > 90% on {{ $labels.instance }}"
      
      # 磁盘空间不足
      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk > 85% on {{ $labels.instance }} {{ $labels.mountpoint }}"
      
      # 磁盘满预测
      - alert: DiskWillFillIn4h
        expr: predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Disk will fill in 4h on {{ $labels.instance }}"
      
      # 系统负载高
      - alert: HighLoad
        expr: node_load1 > count by(instance)(node_cpu_seconds_total{mode="idle"}) * 1.5
        for: 10m
        labels:
          severity: warning
      
      # MTR（Mean Time to Recover）监控
      - alert: NetworkErrors
        expr: rate(node_network_receive_errs_total[5m]) > 0
        for: 5m
        labels:
          severity: warning
  
  # ===== Kubernetes 告警 =====
  - name: kubernetes
    rules:
      # Pod CrashLoopBackOff
      - alert: PodCrashLooping
        expr: increase(kube_pod_container_status_restarts_total[15m]) > 5
        for: 5m
        labels:
          severity: warning
      
      # Pod Not Ready
      - alert: PodNotReady
        expr: kube_pod_status_ready{condition="true"} == 0
        for: 10m
        labels:
          severity: warning
      
      # Deployment 副本不足
      - alert: DeploymentReplicasMismatch
        expr: kube_deployment_spec_replicas != kube_deployment_status_replicas
        for: 10m
        labels:
          severity: warning
      
      # Node Not Ready
      - alert: KubernetesNodeNotReady
        expr: kube_node_status_condition{condition="Ready",status!="true"} == 1
        for: 5m
        labels:
          severity: critical
  
  # ===== 应用告警 =====
  - name: application
    rules:
      # 高错误率
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          / sum(rate(http_requests_total[5m])) by (service) * 100 > 5
        for: 5m
        labels:
          severity: critical
      
      # 高延迟 P99
      - alert: HighLatency
        expr: |
          histogram_quantile(0.99,
            sum by (service, le)(rate(http_request_duration_seconds_bucket[5m]))) > 1
        for: 5m
        labels:
          severity: warning
      
      # 服务不可达
      - alert: ServiceUnavailable
        expr: up{job="app"} == 0
        for: 2m
        labels:
          severity: critical
  
  # ===== 数据库告警 =====
  - name: database
    rules:
      - alert: MySQLDown
        expr: mysql_up == 0
        for: 2m
        labels:
          severity: critical
      
      - alert: MySQLReplicationLag
        expr: mysql_slave_status_seconds_behind_master > 300
        for: 5m
        labels:
          severity: warning
      
      - alert: MySQLTooManyConnections
        expr: mysql_global_status_threads_connected / mysql_global_variables_max_connections * 100 > 80
        for: 5m
        labels:
          severity: warning
      
      - alert: RedisDown
        expr: redis_up == 0
        for: 2m
        labels:
          severity: critical
      
      - alert: RedisMemoryFull
        expr: redis_memory_used_bytes / redis_memory_max_bytes * 100 > 90
        for: 5m
        labels:
          severity: critical
```

---

## 五、配置与管理

### Prometheus 配置

```yaml
# prometheus.yml
rule_files:
  - "rules/recording/*.yml"
  - "rules/alerting/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']
```

### 验证规则

```bash
# 检查规则语法
promtool check rules rules/recording/*.yml rules/alerting/*.yml

# 检查配置
promtool check config prometheus.yml

# 查看已加载的规则
curl http://localhost:9090/api/v1/rules | jq

# 查看当前告警
curl http://localhost:9090/api/v1/alerts | jq
```

### 热重载

```bash
# 修改规则文件后热重载
curl -X POST http://localhost:9090/-/reload
```

> ⚠️ **踩坑提示**：
> - 告警名称必须全局唯一
> - `for` 时间不要太短，推荐至少 1-5 分钟
> - 避免 `for: 0`，瞬时告警容易误报
> - Recording Rule 指标名不要和原始指标名冲突
> - 每条规则都要写 annotations，方便接手人理解

---

## 要点回顾

| 规则类型 | 作用 | 使用场景 |
|---------|------|---------|
| Recording Rules | 预计算存储指标 | 复杂查询、频繁查询 |
| Alerting Rules | 触发告警 | 所有需要告警的场景 |

- Recording Rule 命名：`<scope>:<metric>:<operation>`
- Alerting Rule 三阶段：Pending → Firing → Resolved
- `for` 防瞬时波动，至少 1-5 分钟
- 告警分级：critical / warning / info
- annotations 写清 summary + description + runbook
- 验证用 promtool check rules

---

## 下一篇预告

告警规则搞定了，下一篇 **【Prometheus·可视化篇】Grafana 集成：数据源配置与 Dashboard 设计原则** 将讲解如何用 Grafana 做可视化。

Prometheus 的规则分为两类：Recording Rules（预计算指标）和 Alerting Rules（告警规则）。写好规则是做监控告警的核心技能。本篇详解两类规则的语法、设计原则和常见模式。

【Prometheus·告警篇】告警规则：Recording Rules 与 Alerting Rules 最佳实践

研发部经理

运维

云计算

Prometheus规则体系深度解析：Recording Rules预计算指标加速查询，Alerting Rules实现智能告警。涵盖完整语法、命名规范、告警三阶段生命周期及生产环境最佳实践，包含基础设施、K8s、应用、数据库全套告警配置模板，助力构建高效监控体系。

2026上云采购 | AI焕新·智启新局

tcmg

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

EdgeOne AI 安全实战专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

社区新版编辑器体验调研

【Prometheus·告警篇】告警规则：Recording Rules 与 Alerting Rules 最佳实践-腾讯云开发者社区-腾讯云

【Prometheus·告警篇】告警规则：Recording Rules 与 Alerting Rules 最佳实践

【Prometheus·告警篇】告警规则：Recording Rules 与 Alerting Rules 最佳实践

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐