Alertmanager#

Alertmanager 负责接收 Prometheus / vmalert 发送过来的 alerts，然后完成分组、抑制、静默、路由和通知。

1. Core Concepts#

Concept	Meaning
`route`	告警路由树，决定不同 alert 发到哪里
`receiver`	通知接收方，比如 Slack、Email、Webhook、PagerDuty
`group_by`	按哪些 label 聚合告警
`group_wait`	第一条告警触发后，等待多久再发送
`group_interval`	同一个 group 新增告警后，多久再发送
`repeat_interval`	持续 firing 的告警多久重复通知
`inhibit_rules`	当高级别告警存在时，抑制低级别告警
`silence`	手工静默某些告警
`template`	通知内容模板

2. Basic Config#

global:
  resolve_timeout: 5m

route:
  receiver: lark-monitoring
  group_by:
    - alertname
    - environment
    - service
    - component
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    - receiver: lark-monitoring
      matchers:
        - service="ec2"
        - component="filesystem"
      group_by:
        - alertname
        - environment
        - service
        - component
        - instance_name
        - mountpoint

    - receiver: lark-monitoring
      matchers:
        - service="ec2"
        - component=~"cpu|memory|host"
      group_by:
        - alertname
        - environment
        - service
        - component
        - instance_name

    - receiver: lark-monitoring
      matchers:
        - service="alb"
        - component="targetgroup"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_LoadBalancer
        - dimension_TargetGroup

    - receiver: lark-monitoring
      matchers:
        - service="alb"
        - component="loadbalancer"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_LoadBalancer

    - receiver: lark-monitoring
      matchers:
        - service="ecs"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_ClusterName
        - dimension_ServiceName

    - receiver: lark-monitoring
      matchers:
        - service="sqs"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_QueueName

    - receiver: lark-monitoring
      matchers:
        - service="dynamodb"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_TableName
        - dimension_Operation

    - receiver: lark-monitoring
      matchers:
        - service=~"rds|aurora"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_DBClusterIdentifier

    - receiver: lark-monitoring
      matchers:
        - service=~"elasticache|valkey|redis"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_CacheClusterId

    - receiver: lark-monitoring
      matchers:
        - service="cloudfront"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_DistributionId

receivers:
  - name: lark-monitoring
    webhook_configs:
      - url: http://10.0.1.1:8080/alertmanager/lark
        send_resolved: true

3. EC2 Node Exporter Example#

# vmagent scrape node_exporter on AWS EC2.
# environment/service come from scrape labels.
scrape_configs:
  - job_name: node
    static_configs:
      - targets:
          - "ping-uat-exporters:9100"
        labels:
          environment: uat
          service: ec2
          instance_name: ping-uat-devops

---

# vmalert rule.
# alertname comes from alert; component comes from rule labels.
groups:
  - name: aws-ec2-filesystem.rules
    interval: 1m
    rules:
      - record: node:filesystem_used_percent
        expr: |
          100 * (
            1 -
            (
              node_filesystem_avail_bytes{
                mountpoint!~"/boot($|/.*)|/boot/efi($|/.*)|/run($|/.*)",
                fstype!~"tmpfs|devtmpfs|overlay|squashfs|proc|sysfs"
              }
              /
              node_filesystem_size_bytes{
                mountpoint!~"/boot($|/.*)|/boot/efi($|/.*)|/run($|/.*)",
                fstype!~"tmpfs|devtmpfs|overlay|squashfs|proc|sysfs"
              }
            )
          )

      - alert: AwsEc2DiskUsedHigh
        expr: node:filesystem_used_percent > 90
        for: 15m
        labels:
          severity: warning
          component: filesystem
        annotations:
          summary: "EC2 filesystem usage is high"
          description: "EC2 node {{ $labels.instance_name }} mount {{ $labels.mountpoint }} in {{ $labels.environment }} has filesystem used percent > 90% for 15m."

4. group_by Behavior#

matchers 决定 alert 命中哪条 route，group_by 决定命中 route 后怎么把 alerts 聚合成 notification group。

outer route group_by:
    默认分组策略
    也是整棵 route tree 的兜底分组
    通常应该是最粗、最稳定的一层

child route group_by:
    覆盖外层 group_by
    不是追加

group_by 越粗:
    一条通知包含更多 alerts
    通知数量少
    resolved 生命周期也更粗

group_by 越细:
    一条通知更接近一个故障对象
    通知数量多
    resolved 生命周期更清楚

关键点：Alertmanager 的 resolved 通知也是按 group 发的。如果 group 太粗，某个 alert 已经恢复，但同组里还有其他 alert firing，这个 group 整体仍然是 firing。

最外层 route.group_by 应该只放所有告警都适用的稳定 label，例如：

route:
  receiver: lark-monitoring
  group_by:
    - alertname
    - environment
    - service
    - component

其中 alertname 来自告警规则里的 alert 字段，例如 alert: AwsAlbElb5xxRateHigh 到 Alertmanager 后就是 alertname="AwsAlbElb5xxRateHigh"。它用来区分不同告警类型，避免同一个 environment/service/component 下的 5xx、latency、unhealthy host 被混在一条通知里。

不要在最外层放 mountpoint、dimension_QueueName、dimension_TargetGroup、pod 这类资源身份 label。它们应该放在对应子 route 里，否则默认分组会过细，很多不需要拆开的告警也会被拆成单条通知。

Example:

route:
  receiver: lark-monitoring
  group_by:
    - alertname
    - environment
    - service
    - component
  routes:
    - receiver: lark-monitoring
      matchers:
        - component="filesystem"
      group_by:
        - alertname
        - environment
        - service
        - component
        - instance_name
        - mountpoint

这个配置里，filesystem 告警命中子 route 后，使用子 route 的 group_by。同一台机器的 / 和 /data 会是两个 group：

group 1:
    alertname=AwsEc2DiskUsedHigh
    environment=uat
    service=ec2
    component=filesystem
    instance_name=ping-uat-app
    mountpoint=/

group 2:
    alertname=AwsEc2DiskUsedHigh
    environment=uat
    service=ec2
    component=filesystem
    instance_name=ping-uat-app
    mountpoint=/data

这样 / 和 /data 可以分别 firing、分别 resolved、分别 repeat。

如果 filesystem 只按 environment 分组：

group_by:
  - environment

那么所有 environment="uat" 的 filesystem alerts 都会进同一个 group：

alerts:
    instance_name=host-a, mountpoint=/
    instance_name=host-b, mountpoint=/data
    instance_name=host-c, mountpoint=/opt

这会减少通知数量，但恢复通知也会变粗。只要这个 group 里还有 alert firing，整个 group 就不会完全 resolved。

Rule of thumb:

service / SLO alerts:
    group_by 到 service 级别

resource alerts:
    group_by 到需要独立处理、独立恢复的资源级别

filesystem:
    通常包含 instance_name + mountpoint

ALB target group:
    通常包含 dimension_LoadBalancer + dimension_TargetGroup

SQS:
    通常包含 dimension_QueueName

5. Inhibition#

当同一个服务已经触发 critical 告警时，可以抑制 warning 告警，减少重复噪音。

inhibit_rules:
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal:
      - alertname
      - service
      - env

6. Alert Labels Standard#

required labels:
    alertname
    severity
    service
    env

recommended labels:
    team
    region
    cluster
    namespace

labels:
  severity: critical
  service: order-api
  env: prod
  team: platform

7. Alert Annotations Standard#

summary:
    一句话说明问题

description:
    说明影响范围、当前值、阈值、持续时间

runbook_url:
    指向排障文档

dashboard_url:
    指向 Grafana dashboard

annotations:
  summary: "High HTTP error rate"
  description: "order-api 5xx error rate is higher than 5% for 5 minutes."
  runbook_url: "https://wiki.example.com/runbooks/order-api-5xx"
  dashboard_url: "https://grafana.example.com/d/order-api"

8. Webhook Payload#

Alertmanager 调用 webhook_configs.url 时，会发送 Alertmanager 标准 JSON。为了排查 label、时间和 group 行为，adapter 调试日志建议打印完整 payload。

Firing notification:

{
  "receiver": "lark-monitoring",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertgroup": "aws-alb.alerting.rules",
        "alertname": "AwsAlbTargetGroupResponseTimeHigh",
        "component": "targetgroup",
        "dimension_LoadBalancer": "ping-uat-alb",
        "dimension_TargetGroup": "cpa-http1",
        "environment": "uat",
        "service": "alb",
        "severity": "warning"
      },
      "annotations": {
        "description": "Target group cpa-http1 on ping-uat-alb in uat has TargetResponseTime p95 > 1s.",
        "summary": "ALB target group p95 latency is high"
      },
      "startsAt": "2026-07-01T03:06:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://e93aa825e1ac:8880/vmalert/alert?group_id=18110391789748112193&alert_id=10464459587149070625",
      "fingerprint": "0c052cc9a45419d9"
    }
  ],
  "notification_reason": "first notification",
  "groupLabels": {
    "alertname": "AwsAlbTargetGroupResponseTimeHigh",
    "environment": "uat",
    "severity": "warning"
  },
  "commonLabels": {
    "alertgroup": "aws-alb.alerting.rules",
    "alertname": "AwsAlbTargetGroupResponseTimeHigh",
    "component": "targetgroup",
    "dimension_LoadBalancer": "ping-uat-alb",
    "dimension_TargetGroup": "cpa-http1",
    "environment": "uat",
    "service": "alb",
    "severity": "warning"
  },
  "commonAnnotations": {
    "description": "Target group cpa-http1 on ping-uat-alb in uat has TargetResponseTime p95 > 1s.",
    "summary": "ALB target group p95 latency is high"
  },
  "externalURL": "http://d06f02ab1181:9093",
  "version": "4",
  "groupKey": "{}/{severity=~\"P1|P2|critical|warning\"}:{alertname=\"AwsAlbTargetGroupResponseTimeHigh\", environment=\"uat\", severity=\"warning\"}",
  "truncatedAlerts": 0
}

Resolved notification:

{
  "receiver": "lark-monitoring",
  "status": "resolved",
  "alerts": [
    {
      "status": "resolved",
      "labels": {
        "alertgroup": "aws-alb.alerting.rules",
        "alertname": "AwsAlbTargetGroupResponseTimeHigh",
        "component": "targetgroup",
        "dimension_LoadBalancer": "ping-uat-alb",
        "dimension_TargetGroup": "cpa-http1",
        "environment": "uat",
        "service": "alb",
        "severity": "warning"
      },
      "annotations": {
        "description": "Target group cpa-http1 on ping-uat-alb in uat has TargetResponseTime p95 > 1s.",
        "summary": "ALB target group p95 latency is high"
      },
      "startsAt": "2026-07-01T03:06:00Z",
      "endsAt": "2026-07-01T03:11:00Z",
      "generatorURL": "http://e93aa825e1ac:8880/vmalert/alert?group_id=18110391789748112193&alert_id=10464459587149070625",
      "fingerprint": "0c052cc9a45419d9"
    }
  ],
  "notification_reason": "all alerts resolved",
  "groupLabels": {
    "alertname": "AwsAlbTargetGroupResponseTimeHigh",
    "environment": "uat",
    "severity": "warning"
  },
  "commonLabels": {
    "alertgroup": "aws-alb.alerting.rules",
    "alertname": "AwsAlbTargetGroupResponseTimeHigh",
    "component": "targetgroup",
    "dimension_LoadBalancer": "ping-uat-alb",
    "dimension_TargetGroup": "cpa-http1",
    "environment": "uat",
    "service": "alb",
    "severity": "warning"
  },
  "commonAnnotations": {
    "description": "Target group cpa-http1 on ping-uat-alb in uat has TargetResponseTime p95 > 1s.",
    "summary": "ALB target group p95 latency is high"
  },
  "externalURL": "http://d06f02ab1181:9093",
  "version": "4",
  "groupKey": "{}/{severity=~\"P1|P2|critical|warning\"}:{alertname=\"AwsAlbTargetGroupResponseTimeHigh\", environment=\"uat\", severity=\"warning\"}",
  "truncatedAlerts": 0
}

alerts 是数组，不是数字。adapter 日志里如果打印 payload.alerts.length，看到的 alerts: 1 表示本次 notification group 里包含 1 条 alert。这个数字会增加：当多个 alert 被 Alertmanager 按 group_by 聚合进同一个 group，并且一次通知一起发送时，就会变成 2、3 等。

Important fields:

status:
    group status，firing 或 resolved

groupLabels:
    当前 notification group 的 labels
    来自 route.group_by 指定的 label keys

commonLabels:
    本次 alerts 数组里所有 alert 共同拥有且值相同的 labels

alerts:
    本次通知包含的 alert list
    每条 alert 有自己的 labels / annotations / startsAt / endsAt

startsAt:
    这条 alert 第一次变成 active 的时间
    firing 和 resolved payload 中都会保留

endsAt:
    firing 时通常是 zero time 或未来 timeout
    resolved 时是恢复时间

annotations:
    给人看的文本，不参与 group_by

generatorURL:
    触发这个 alert 的 Prometheus / vmalert source link

adapter 如果要计算 duration，应该使用：

firing:
    now - alerts[i].startsAt

resolved:
    alerts[i].endsAt - alerts[i].startsAt

如果日志里看不到 startsAt，通常是因为 adapter 只打印了摘要，没有把 payload.alerts 数组完整打印出来。调试时应打印完整 payload，这样才能看到 alerts[] 里的 labels、annotations、startsAt、endsAt、generatorURL。

9. Common Notification Template#

这个模板适合 webhook / Slack / Feishu / DingTalk 二次改造时复用。

{{ define "alert.title" -}}
[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}
{{- end }}

{{ define "alert.text" -}}
{{ range .Alerts }}
Alert: {{ .Labels.alertname }}
Status: {{ .Status }}
Severity: {{ .Labels.severity }}
Service: {{ .Labels.service }}
Env: {{ .Labels.env }}
Summary: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
{{ if .Annotations.dashboard_url }}Dashboard: {{ .Annotations.dashboard_url }}{{ end }}
{{ if .Annotations.runbook_url }}Runbook: {{ .Annotations.runbook_url }}{{ end }}
StartsAt: {{ .StartsAt }}
{{ if eq .Status "resolved" }}EndsAt: {{ .EndsAt }}{{ end }}

{{ end }}
{{- end }}

Alertmanager config:

templates:
  - /etc/alertmanager/templates/*.tmpl

10. Best Practices#

Config:
    alertmanager.yml 放进 Git 管理
    receiver 名称用 team / channel / purpose 命名
    critical 和 warning 使用不同 route

Routing:
    默认 receiver 兜底
    按 severity / team / service 分发
    高优先级告警走电话 / PagerDuty / Opsgenie
    普通告警走 Slack / Email / IM

Noise Control:
    合理设置 group_by
    用 inhibition 减少重复通知
    用 silence 做维护窗口
    避免每个 instance 单独发一条消息

Template:
    title 简短
    body 包含 service / env / severity / summary / runbook / dashboard
    firing 和 resolved 都要可读

Security:
    webhook url / token 不要提交到 Git
    使用 Secret / env / external secret 管理敏感信息

11. References#

Alertmanager configuration: https://prometheus.io/docs/alerting/latest/configuration/
Notification templates: https://prometheus.io/docs/alerting/latest/notifications/
Prometheus alerting overview: https://prometheus.io/docs/alerting/latest/overview/