本文缩写（英文全称与中文含义）

缩写	英文全称／名称	中文含义
ALB	Application Load Balancer	应用负载均衡器
EC2	Elastic Compute Cloud	AWS 云服务器服务
IM	Instant Messaging	即时通信
JSON	JavaScript Object Notation	一种结构化数据格式
SLO	Service Level Objective	服务等级目标
SQS	Simple Queue Service	AWS 消息队列服务

Alertmanager#

Alertmanager 负责接收 Prometheus / vmalert 发送过来的 alerts，然后完成分组、抑制、静默、路由和通知。

1. Core Concepts#

Concept	Meaning
`route`	告警路由树，决定不同 alert 发到哪里
`receiver`	通知接收方，比如 Slack、Email、Webhook、PagerDuty
`group_by`	按哪些 label 聚合告警
`group_wait`	第一条告警触发后，等待多久再发送
`group_interval`	同一个 group 新增告警后，多久再发送
`repeat_interval`	持续 firing 的告警多久重复通知
`inhibit_rules`	当高级别告警存在时，抑制低级别告警
`silence`	手工静默某些告警
`template`	通知内容模板

2. Basic Config#

global:
  resolve_timeout: 5m

route:
  receiver: lark-monitoring
  group_by:
    - alertname
    - environment
    - service
    - component
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    - receiver: lark-monitoring
      matchers:
        - service="ec2"
        - component="filesystem"
      group_by:
        - alertname
        - environment
        - service
        - component
        - instance_name
        - mountpoint

    - receiver: lark-monitoring
      matchers:
        - service="ec2"
        - component=~"cpu|memory|host"
      group_by:
        - alertname
        - environment
        - service
        - component
        - instance_name

    - receiver: lark-monitoring
      matchers:
        - service="alb"
        - component="targetgroup"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_LoadBalancer
        - dimension_TargetGroup

    - receiver: lark-monitoring
      matchers:
        - service="alb"
        - component="loadbalancer"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_LoadBalancer

    - receiver: lark-monitoring
      matchers:
        - service="ecs"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_ClusterName
        - dimension_ServiceName

    - receiver: lark-monitoring
      matchers:
        - service="sqs"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_QueueName

    - receiver: lark-monitoring
      matchers:
        - service="dynamodb"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_TableName
        - dimension_Operation

    - receiver: lark-monitoring
      matchers:
        - service=~"rds|aurora"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_DBClusterIdentifier

    - receiver: lark-monitoring
      matchers:
        - service=~"elasticache|valkey|redis"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_CacheClusterId

    - receiver: lark-monitoring
      matchers:
        - service="cloudfront"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_DistributionId

receivers:
  - name: lark-monitoring
    webhook_configs:
      - url: http://10.0.1.1:8080/alertmanager/lark
        send_resolved: true

3. EC2 Node Exporter Example#

# vmagent scrape node_exporter on AWS EC2.
# environment/service come from scrape labels.
scrape_configs:
  - job_name: node
    static_configs:
      - targets:
          - "example-uat-exporters:9100"
        labels:
          environment: uat
          service: ec2
          instance_name: example-uat-devops

---

# vmalert rule.
# alertname comes from alert; component comes from rule labels.
groups:
  - name: aws-ec2-filesystem.rules
    interval: 1m
    rules:
      - record: node:filesystem_used_percent
        expr: |
          100 * (
            1 -
            (
              node_filesystem_avail_bytes{
                mountpoint!~"/boot($|/.*)|/boot/efi($|/.*)|/run($|/.*)",
                fstype!~"tmpfs|devtmpfs|overlay|squashfs|proc|sysfs"
              }
              /
              node_filesystem_size_bytes{
                mountpoint!~"/boot($|/.*)|/boot/efi($|/.*)|/run($|/.*)",
                fstype!~"tmpfs|devtmpfs|overlay|squashfs|proc|sysfs"
              }
            )
          )

      - alert: AwsEc2DiskUsedHigh
        expr: node:filesystem_used_percent > 90
        for: 15m
        labels:
          severity: warning
          component: filesystem
        annotations:
          summary: "EC2 filesystem usage is high"
          description: "EC2 node {{ $labels.instance_name }} mount {{ $labels.mountpoint }} in {{ $labels.environment }} has filesystem used percent > 90% for 15m."

4. group_by Behavior#

matchers 决定 alert 命中哪条 route，group_by 决定命中 route 后怎么把 alerts 聚合成 notification group。

outer route group_by:
    默认分组策略
    也是整棵 route tree 的兜底分组
    通常应该是最粗、最稳定的一层

child route group_by:
    覆盖外层 group_by
    不是追加

group_by 越粗:
    一条通知包含更多 alerts
    通知数量少
    resolved 生命周期也更粗

group_by 越细:
    一条通知更接近一个故障对象
    通知数量多
    resolved 生命周期更清楚

关键点：Alertmanager 的 resolved 通知也是按 group 发的。如果 group 太粗，某个 alert 已经恢复，但同组里还有其他 alert firing，这个 group 整体仍然是 firing。

最外层 route.group_by 应该只放所有告警都适用的稳定 label，例如：

route:
  receiver: lark-monitoring
  group_by:
    - alertname
    - environment
    - service
    - component

其中 alertname 来自告警规则里的 alert 字段，例如 alert: AwsAlbElb5xxRateHigh 到 Alertmanager 后就是 alertname="AwsAlbElb5xxRateHigh"。它用来区分不同告警类型，避免同一个 environment/service/component 下的 5xx、latency、unhealthy host 被混在一条通知里。

不要在最外层放 mountpoint、dimension_QueueName、dimension_TargetGroup、pod 这类资源身份 label。它们应该放在对应子 route 里，否则默认分组会过细，很多不需要拆开的告警也会被拆成单条通知。

Example:

route:
  receiver: lark-monitoring
  group_by:
    - alertname
    - environment
    - service
    - component
  routes:
    - receiver: lark-monitoring
      matchers:
        - component="filesystem"
      group_by:
        - alertname
        - environment
        - service
        - component
        - instance_name
        - mountpoint

这个配置里，filesystem 告警命中子 route 后，使用子 route 的 group_by。同一台机器的 / 和 /data 会是两个 group：

group 1:
    alertname=AwsEc2DiskUsedHigh
    environment=uat
    service=ec2
    component=filesystem
    instance_name=example-uat-app
    mountpoint=/

group 2:
    alertname=AwsEc2DiskUsedHigh
    environment=uat
    service=ec2
    component=filesystem
    instance_name=example-uat-app
    mountpoint=/data

这样 / 和 /data 可以分别 firing、分别 resolved、分别 repeat。

如果 filesystem 只按 environment 分组：

group_by:
  - environment

那么所有 environment="uat" 的 filesystem alerts 都会进同一个 group：

alerts:
    instance_name=host-a, mountpoint=/
    instance_name=host-b, mountpoint=/data
    instance_name=host-c, mountpoint=/opt

这会减少通知数量，但恢复通知也会变粗。只要这个 group 里还有 alert firing，整个 group 就不会完全 resolved。

Rule of thumb:

service / SLO alerts:
    group_by 到 service 级别

resource alerts:
    group_by 到需要独立处理、独立恢复的资源级别

filesystem:
    通常包含 instance_name + mountpoint

ALB target group:
    通常包含 dimension_LoadBalancer + dimension_TargetGroup

SQS:
    通常包含 dimension_QueueName

5. Inhibition#

当同一个服务已经触发 critical 告警时，可以抑制 warning 告警，减少重复噪音。

inhibit_rules:
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal:
      - alertname
      - service
      - env

6. Alert Labels Standard#

required labels:
    alertname
    severity
    service
    env

recommended labels:
    team
    region
    cluster
    namespace

labels:
  severity: critical
  service: order-api
  env: prod
  team: platform

7. Alert Annotations Standard#

summary:
    一句话说明问题

description:
    说明影响范围、当前值、阈值、持续时间

runbook_url:
    指向排障文档

dashboard_url:
    指向 Grafana dashboard

annotations:
  summary: "High HTTP error rate"
  description: "order-api 5xx error rate is higher than 5% for 5 minutes."
  runbook_url: "https://wiki.example.com/runbooks/order-api-5xx"
  dashboard_url: "https://grafana.example.com/d/order-api"

8. Webhook Payload#

Alertmanager 调用 webhook_configs.url 时，会发送 Alertmanager 标准 JSON。为了排查 label、时间和 group 行为，adapter 调试日志建议打印完整 payload。

Firing notification:

{
  "receiver": "lark-monitoring",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertgroup": "aws-alb.alerting.rules",
        "alertname": "AwsAlbTargetGroupResponseTimeHigh",
        "component": "targetgroup",
        "dimension_LoadBalancer": "example-uat-alb",
        "dimension_TargetGroup": "cpa-http1",
        "environment": "uat",
        "service": "alb",
        "severity": "warning"
      },
      "annotations": {
        "description": "Target group cpa-http1 on example-uat-alb in uat has TargetResponseTime p95 > 1s.",
        "summary": "ALB target group p95 latency is high"
      },
      "startsAt": "2026-07-01T03:06:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://e93aa825e1ac:8880/vmalert/alert?group_id=18110391789748112193&alert_id=10464459587149070625",
      "fingerprint": "0c052cc9a45419d9"
    }
  ],
  "notification_reason": "first notification",
  "groupLabels": {
    "alertname": "AwsAlbTargetGroupResponseTimeHigh",
    "environment": "uat",
    "severity": "warning"
  },
  "commonLabels": {
    "alertgroup": "aws-alb.alerting.rules",
    "alertname": "AwsAlbTargetGroupResponseTimeHigh",
    "component": "targetgroup",
    "dimension_LoadBalancer": "example-uat-alb",
    "dimension_TargetGroup": "cpa-http1",
    "environment": "uat",
    "service": "alb",
    "severity": "warning"
  },
  "commonAnnotations": {
    "description": "Target group cpa-http1 on example-uat-alb in uat has TargetResponseTime p95 > 1s.",
    "summary": "ALB target group p95 latency is high"
  },
  "externalURL": "http://d06f02ab1181:9093",
  "version": "4",
  "groupKey": "{}/{severity=~\"P1|P2|critical|warning\"}:{alertname=\"AwsAlbTargetGroupResponseTimeHigh\", environment=\"uat\", severity=\"warning\"}",
  "truncatedAlerts": 0
}

Resolved notification:

{
  "receiver": "lark-monitoring",
  "status": "resolved",
  "alerts": [
    {
      "status": "resolved",
      "labels": {
        "alertgroup": "aws-alb.alerting.rules",
        "alertname": "AwsAlbTargetGroupResponseTimeHigh",
        "component": "targetgroup",
        "dimension_LoadBalancer": "example-uat-alb",
        "dimension_TargetGroup": "cpa-http1",
        "environment": "uat",
        "service": "alb",
        "severity": "warning"
      },
      "annotations": {
        "description": "Target group cpa-http1 on example-uat-alb in uat has TargetResponseTime p95 > 1s.",
        "summary": "ALB target group p95 latency is high"
      },
      "startsAt": "2026-07-01T03:06:00Z",
      "endsAt": "2026-07-01T03:11:00Z",
      "generatorURL": "http://e93aa825e1ac:8880/vmalert/alert?group_id=18110391789748112193&alert_id=10464459587149070625",
      "fingerprint": "0c052cc9a45419d9"
    }
  ],
  "notification_reason": "all alerts resolved",
  "groupLabels": {
    "alertname": "AwsAlbTargetGroupResponseTimeHigh",
    "environment": "uat",
    "severity": "warning"
  },
  "commonLabels": {
    "alertgroup": "aws-alb.alerting.rules",
    "alertname": "AwsAlbTargetGroupResponseTimeHigh",
    "component": "targetgroup",
    "dimension_LoadBalancer": "example-uat-alb",
    "dimension_TargetGroup": "cpa-http1",
    "environment": "uat",
    "service": "alb",
    "severity": "warning"
  },
  "commonAnnotations": {
    "description": "Target group cpa-http1 on example-uat-alb in uat has TargetResponseTime p95 > 1s.",
    "summary": "ALB target group p95 latency is high"
  },
  "externalURL": "http://d06f02ab1181:9093",
  "version": "4",
  "groupKey": "{}/{severity=~\"P1|P2|critical|warning\"}:{alertname=\"AwsAlbTargetGroupResponseTimeHigh\", environment=\"uat\", severity=\"warning\"}",
  "truncatedAlerts": 0
}

alerts 是数组，不是数字。adapter 日志里如果打印 payload.alerts.length，看到的 alerts: 1 表示本次 notification group 里包含 1 条 alert。这个数字会增加：当多个 alert 被 Alertmanager 按 group_by 聚合进同一个 group，并且一次通知一起发送时，就会变成 2、3 等。

Important fields:

status:
    group status，firing 或 resolved

groupLabels:
    当前 notification group 的 labels
    来自 route.group_by 指定的 label keys

commonLabels:
    本次 alerts 数组里所有 alert 共同拥有且值相同的 labels

alerts:
    本次通知包含的 alert list
    每条 alert 有自己的 labels / annotations / startsAt / endsAt

startsAt:
    这条 alert 第一次变成 active 的时间
    firing 和 resolved payload 中都会保留

endsAt:
    firing 时通常是 zero time 或未来 timeout
    resolved 时是恢复时间

annotations:
    给人看的文本，不参与 group_by

generatorURL:
    触发这个 alert 的 Prometheus / vmalert source link

adapter 如果要计算 duration，应该使用：

firing:
    now - alerts[i].startsAt

resolved:
    alerts[i].endsAt - alerts[i].startsAt

如果日志里看不到 startsAt，通常是因为 adapter 只打印了摘要，没有把 payload.alerts 数组完整打印出来。调试时应打印完整 payload，这样才能看到 alerts[] 里的 labels、annotations、startsAt、endsAt、generatorURL。

9. Common Notification Template#

这个模板适合 webhook / Slack / Feishu / DingTalk 二次改造时复用。

{{ define "alert.title" -}}
[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}
{{- end }}

{{ define "alert.text" -}}
{{ range .Alerts }}
Alert: {{ .Labels.alertname }}
Status: {{ .Status }}
Severity: {{ .Labels.severity }}
Service: {{ .Labels.service }}
Env: {{ .Labels.env }}
Summary: {{ .Annotations.summary }}
Description: {{ .Annotations.description }}
{{ if .Annotations.dashboard_url }}Dashboard: {{ .Annotations.dashboard_url }}{{ end }}
{{ if .Annotations.runbook_url }}Runbook: {{ .Annotations.runbook_url }}{{ end }}
StartsAt: {{ .StartsAt }}
{{ if eq .Status "resolved" }}EndsAt: {{ .EndsAt }}{{ end }}

{{ end }}
{{- end }}

Alertmanager config:

templates:
  - /etc/alertmanager/templates/*.tmpl

10. Best Practices#

Config:
    alertmanager.yml 放进 Git 管理
    receiver 名称用 team / channel / purpose 命名
    critical 和 warning 使用不同 route

Routing:
    默认 receiver 兜底
    按 severity / team / service 分发
    高优先级告警走电话 / PagerDuty / Opsgenie
    普通告警走 Slack / Email / IM

Noise Control:
    合理设置 group_by
    用 inhibition 减少重复通知
    用 silence 做维护窗口
    避免每个 instance 单独发一条消息

Template:
    title 简短
    body 包含 service / env / severity / summary / runbook / dashboard
    firing 和 resolved 都要可读

Security:
    webhook url / token 不要提交到 Git
    使用 Secret / env / external secret 管理敏感信息

完整演练：维护静默、重复通知与外部心跳。

11. References#

Alertmanager configuration: https://prometheus.io/docs/alerting/latest/configuration/
Notification templates: https://prometheus.io/docs/alerting/latest/notifications/
Prometheus alerting overview: https://prometheus.io/docs/alerting/latest/overview/