vmalert

vmalert#

vmalert 是 VictoriaMetrics 生态中的 rule evaluator，用来执行：

recording rules:
    周期性计算 PromQL / MetricsQL，写回 VictoriaMetrics

alerting rules:
    周期性计算告警表达式，触发后发送到 Alertmanager

1. Architecture#

vmagent / Prometheus
    -> remote_write
    -> VictoriaMetrics
    <- query
    <- vmalert
    -> Alertmanager
    -> Slack / Email / Webhook / PagerDuty / etc.

2. Install#

Docker#

docker run -d \
  --name vmalert \
  -p 8880:8880 \
  -v ./rules:/rules \
  victoriametrics/vmalert:latest \
  -datasource.url=http://host.docker.internal:8428 \
  -remoteWrite.url=http://host.docker.internal:8428 \
  -notifier.url=http://host.docker.internal:9093 \
  -rule=/rules/*.yml

Binary#

vmalert \
  -datasource.url=http://localhost:8428 \
  -remoteWrite.url=http://localhost:8428/api/v1/write \
  -notifier.url=http://localhost:9093 \
  -rule=./rules/*.yml

3. vmalert Rules#

vmalert rule format is compatible with Prometheus rule files.

groups:
  - name: api-alerts
    interval: 30s
    rules:
      - alert: HighHttpErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))
          > 0.05
        for: 5m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "High HTTP error rate"
          description: "5xx error rate is higher than 5% for 5 minutes."

这条 rule 触发后，发送给 Alertmanager 的 alert labels 大致是：

ALERTS{
  alertname="HighHttpErrorRate",
  severity="warning",
  team="platform"
}

如果 PromQL 查询结果里保留了 service / env 等 labels，它们也会一起进入 Alertmanager。

4. Alertmanager Integration#

vmalert 通过 -notifier.url 把 alerting rules 触发的告警发送给 Alertmanager。Alertmanager 不关心告警来自 Prometheus 还是 vmalert，只根据 alert labels 做 route / group_by / inhibit_rules / receiver。

VictoriaMetrics:
    存储 metrics

vmalert:
    查询 VictoriaMetrics
    执行 alerting rules
    把 firing / resolved alerts 发送给 Alertmanager

Alertmanager:
    按 labels 分组、抑制、静默、路由
    发送到 Lark / Slack / Email / Webhook

vmalert#

vmalert \
  -datasource.url=http://victoriametrics:8428 \
  -remoteWrite.url=http://victoriametrics:8428/api/v1/write \
  -notifier.url=http://alertmanager:9093 \
  -rule=/etc/vmalert/rules/*.yml

Alertmanager#

route:
  receiver: lark-default
  group_by:
    - alertname
    - service
    - env
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # vmalert 发来的告警包含 team="platform" 时，进入 platform Lark 群
    # 匹配样例: ALERTS{alertname="HighHttpErrorRate", team="platform", service="order-api", env="prod", severity="warning"}
    - receiver: lark-platform
      matchers:
        - team="platform"

    # vmalert 发来的 critical 告警进入 critical Lark 群
    # 匹配样例: ALERTS{alertname="ApiDown", team="platform", service="order-api", env="prod", severity="critical"}
    - receiver: lark-critical
      matchers:
        - severity="critical"

receivers:
  - name: lark-default
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/default
        send_resolved: true

  - name: lark-platform
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/platform
        send_resolved: true

  - name: lark-critical
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/critical
        send_resolved: true

Rule Labels#

Alertmanager 的 matchers 依赖 vmalert rule 里写出的 labels，或者 PromQL 查询结果保留下来的 labels。

groups:
  - name: api-alerts
    interval: 30s
    rules:
      - alert: HighHttpErrorRate
        expr: |
          sum by (service, env) (
            rate(http_requests_total{status=~"5.."}[5m])
          )
          /
          sum by (service, env) (
            rate(http_requests_total[5m])
          )
          > 0.05
        for: 5m
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "High HTTP error rate"
          description: "5xx error rate is higher than 5% for 5 minutes."
          dashboard_url: "https://grafana.example.com/d/order-api"
          runbook_url: "https://wiki.example.com/runbooks/order-api-5xx"

这条 rule 进入 Alertmanager 后可以被这样路由：

team="platform":
    匹配 lark-platform route

severity="warning":
    可以匹配 warning route

service="order-api", env="prod":
    来自 PromQL 结果 label，可用于 group_by 或 route matchers

5. Recording Rule Example#

groups:
  - name: api-recording-rules
    interval: 30s
    rules:
      - record: service:http_requests:rate5m
        expr: |
          sum by (service, route, method, status) (
            rate(http_requests_total[5m])
          )

query after recording:

service:http_requests:rate5m{service="order-api"}

6. Best Practices#

Rule Files:
    rules 放进 Git 管理
    按 domain / service / team 拆分文件
    alerting rules 和 recording rules 可以分开目录

Alert Labels:
    必须有 severity
    推荐有 team / service / env
    不要把动态值放进 label

Annotations:
    summary 写一句话问题
    description 写影响范围和当前值
    runbook_url 指向排障文档
    dashboard_url 指向 Grafana 面板

Query:
    alert expression 要可读
    高成本查询先做 recording rule
    ratio 类告警要同时考虑分母太小的问题

Delivery:
    vmalert 只负责触发告警
    -notifier.url 指向 Alertmanager
    通知路由、静默、抑制交给 Alertmanager

7. Useful Alerts#

groups:
  - name: service-sli-alerts
    rules:
      - alert: HighP95Latency
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, service, route) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          ) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High p95 latency"
          description: "p95 latency is higher than 1s for 10 minutes."

8. References#

VMUI Forward