GroupBy

Alertmanager group_by Best Practices#

group_by 不是简单地“越多越好”或“越少越好”。它定义的是一条通知代表什么，以及 firing / resolved / repeat 的生命周期边界。

1. Core Rule#

最外层 route.group_by 应该设计成整棵 route tree 的默认分组，也是兜底使用的最粗粒度告警 group。它不应该包含 mountpoint、queue name、targetgroup、instance_name 这类资源身份 label，否则所有没有命中子 route 的告警都会被默认打散。

推荐先用一个稳定的 root group：

route:
  receiver: lark-monitoring
  group_by:
    - alertname
    - environment
    - service
    - component
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

含义：

alertname:
    不同告警类型不要混在一起
    来自 Prometheus / vmalert rule 里的 alert 字段
    例如 alert: AwsAlbElb5xxRateHigh 最终会变成 alertname="AwsAlbElb5xxRateHigh"

environment:
    prod / uat / dev 分开

service:
    ec2 / alb / sqs / rds / elasticache / dynamodb / cloudfront / app service 分开

component:
    filesystem / cpu / memory / targetgroup / loadbalancer / queue / database / cache 分开

alertname 用来区分“是什么告警类型”。如果 root group_by 不包含 alertname，同一个 environment/service/component 下的延迟高、5xx 高、unhealthy host 可能会被合并到同一条通知里，排查时不直观。

这个 root group 适合大多数服务级、组件级、warning 类告警。然后对“需要独立处理、独立恢复”的资源类告警，在子 route 里覆盖 group_by。

设计最顶层 group_by 时按这个原则：

root group_by:
    放所有告警都应该有、且语义稳定的 label
    用来表达告警属于哪个类型、环境、服务、组件
    作为没有特殊 route 时的默认聚合方式

child route group_by:
    只在需要独立处理、独立恢复时加资源身份 label
    例如 instance_name / mountpoint / dimension_QueueName / dimension_TargetGroup

所以 root group_by 通常是“必须要的最粗粒度”，但不是最终唯一粒度。真正需要更细生命周期的告警，应该在子 route 里覆盖。

2. Why Not Too Coarse#

如果只按 environment 分组：

group_by:
  - environment

那么同一个环境里的很多不同告警会进入同一个 notification group：

environment=uat:
    EC2 filesystem /
    EC2 memory
    ALB targetgroup p95 latency
    SQS queue age

问题：

too coarse:
    一条通知里 alerts[] 太多
    不同处理动作混在一起
    resolved 也按 group 发，只要 group 里还有 alert firing，整个 group 就不会完全 resolved

3. Why Not Too Fine#

如果默认 group_by 放太多资源 label：

group_by:
  - alertname
  - environment
  - service
  - component
  - instance
  - pod
  - mountpoint
  - dimension_QueueName

问题：

too fine:
    所有告警都被打散
    warning 类告警容易刷屏
    服务级 SLO 告警被实例维度拆碎
    有些 label 在某些服务里不存在，分组策略变得难推理

所以 root route 保持通用，子 route 针对具体 service / component 覆盖。

4. Decision Guide#

业界常见做法不是固定“粗”或“细”，而是先判断这条告警代表的是服务现象，还是具体资源对象。

粗粒度分组适合这些场景：

Scenario	Recommended group_by	Why
服务级 SLO / 用户体验告警	`alertname`, `environment`, `service`	5xx rate、latency、error budget burn 代表服务整体退化，不应该按 instance / target / queue 拆成很多条
发布、版本、全局配置问题	`alertname`, `environment`, `service`, `version`	处理动作通常是回滚、暂停发布、调整配置，不是逐个资源修
区域级或依赖级故障	`alertname`, `environment`, `region`, `service`	比如某个 region 的 AWS 依赖异常，按单个 queue / db / targetgroup 拆开会放大噪音
大量 warning 趋势类告警	`alertname`, `environment`, `service`, `component`	用于提示趋势，不要求每个资源单独恢复通知

细粒度分组适合这些场景：

Scenario	Recommended group_by	Why
资源容量类告警	加上资源身份 label	磁盘、队列积压、DB 连接、cache 内存通常需要定位到具体资源
需要独立恢复通知	加上能区分恢复边界的 label	Alertmanager 的 resolved 也是按 group 发的；如果 `/` 恢复但 `/data` 还在 firing，粗分组不会给出“整组已恢复”
P1 / page 类告警	加上直接定位资源的 label	值班人员收到通知后应该能立刻知道处理对象
处理动作不同的维度	加上决定处理动作的 label	DynamoDB `GetItem` 慢和 `PutItem` 慢可能排查方向不同，适合保留 `dimension_Operation`

中等粒度是最常用的折中：

Alert Type	Example group_by	Why
EC2 CPU / memory	`alertname`, `environment`, `service`, `component`, `instance_name`	一台机器一条生命周期，避免被 CPU core、device 这类低层 label 打散
ALB loadbalancer	`alertname`, `environment`, `service`, `component`, `dimension_LoadBalancer`	ALB 级问题按 load balancer 处理
ALB targetgroup	`alertname`, `environment`, `service`, `component`, `dimension_LoadBalancer`, `dimension_TargetGroup`	target group 是实际处理对象
SQS queue	`alertname`, `environment`, `service`, `component`, `dimension_QueueName`	backlog / age 通常按队列处理

判断时可以按这几个问题走：

1. 这条告警代表服务整体症状，还是某个资源对象？
   service symptom -> 粗一点
   resource object -> 细一点

2. 多个 alert 的处理动作是否相同？
   相同 -> 可以粗一点
   不同 -> 应该细一点

3. 其中一个 alert 恢复时，是否需要独立 resolved 通知？
   需要 -> 把恢复边界对应的 label 放进 group_by
   不需要 -> 可以合并

4. 通知 card 是否能清楚展示 alerts[] 里的多条告警？
   能 -> 可以适当合并
   不能 -> 应该拆细

5. 这个 label 是否稳定、可读、低噪音？
   instance_name / mountpoint / queue name 通常可以
   pod uid / container id / request path 通常不要放进 group_by

group_by 只影响通知聚合，不会删除 alert 自身的 labels。即使 root route 只按 alertname、environment、service、component 分组，每条 alert 的 alerts[].labels 里仍然可以看到 instance_name、mountpoint、dimension_QueueName 等细节。真正需要关注的是：这些 alert 是否应该共享同一条 firing / repeat / resolved 生命周期。

5. Recommended Routes#

route:
  receiver: lark-monitoring
  group_by:
    - alertname
    - environment
    - service
    - component
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    # Filesystem sample alert labels:
    #   alert 1: alertname="AwsEc2DiskUsedHigh", environment="uat", service="ec2", component="filesystem", instance_name="ping-uat-app", mountpoint="/"
    #   alert 2: alertname="AwsEc2DiskUsedHigh", environment="uat", service="ec2", component="filesystem", instance_name="ping-uat-app", mountpoint="/data"
    #   alert 3: alertname="AwsEc2DiskUsedHigh", environment="uat", service="ec2", component="filesystem", instance_name="ping-uat-devops", mountpoint="/"
    # Label source: alertname from alert rule; environment/service/instance_name from node_exporter scrape labels; mountpoint from node metric; component from alert rule labels.
    # Result: 3 groups, because instance_name or mountpoint is different.
    - receiver: lark-monitoring
      matchers:
        - service="ec2"
        - component="filesystem"
      group_by:
        - alertname
        - environment
        - service
        - component
        - instance_name
        - mountpoint

    # EC2 host sample alert labels:
    #   alert 1: alertname="AwsEc2CpuUsedHigh", environment="uat", service="ec2", component="cpu", instance_name="ping-uat-app"
    #   alert 2: alertname="AwsEc2CpuUsedHigh", environment="uat", service="ec2", component="cpu", instance_name="ping-uat-app"
    #   alert 3: alertname="AwsEc2MemoryUsedHigh", environment="uat", service="ec2", component="memory", instance_name="ping-uat-app"
    # Label source: alertname from alert rule; environment/service/instance_name from node_exporter scrape labels; component from alert rule labels.
    # Result: alert 1 and alert 2 are in one group; alert 3 is another group because alertname/component is different.
    - receiver: lark-monitoring
      matchers:
        - service="ec2"
        - component=~"cpu|memory|host"
      group_by:
        - alertname
        - environment
        - service
        - component
        - instance_name

    # ALB target group sample alert labels:
    #   alert 1: alertname="AwsAlbTargetGroupResponseTimeHigh", environment="uat", service="alb", component="targetgroup", dimension_LoadBalancer="ping-uat-alb", dimension_TargetGroup="api-tg"
    #   alert 2: alertname="AwsAlbTargetGroupResponseTimeHigh", environment="uat", service="alb", component="targetgroup", dimension_LoadBalancer="ping-uat-alb", dimension_TargetGroup="admin-tg"
    #   alert 3: alertname="AwsAlbTargetGroupResponseTimeHigh", environment="uat", service="alb", component="targetgroup", dimension_LoadBalancer="ping-uat-docmost-alb", dimension_TargetGroup="api-tg"
    # Label source: alertname from alert rule; environment from YACE scrape/relabel; dimension_* from YACE metric; service/component from alert rule labels.
    # Result: 3 groups, because target group or parent load balancer is different.
    - receiver: lark-monitoring
      matchers:
        - service="alb"
        - component="targetgroup"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_LoadBalancer
        - dimension_TargetGroup

    # ALB load balancer sample alert labels:
    #   alert 1: alertname="AwsAlbElb5xxRateHigh", environment="uat", service="alb", component="loadbalancer", dimension_LoadBalancer="ping-uat-alb"
    #   alert 2: alertname="AwsAlbElb5xxRateHigh", environment="uat", service="alb", component="loadbalancer", dimension_LoadBalancer="ping-uat-alb"
    #   alert 3: alertname="AwsAlbElb5xxRateHigh", environment="uat", service="alb", component="loadbalancer", dimension_LoadBalancer="ping-uat-docmost-alb"
    # Label source: alertname from alert rule; environment from YACE scrape/relabel; dimension_LoadBalancer from YACE metric; service/component from alert rule labels.
    # Result: alert 1 and alert 2 are in one group; alert 3 is another group.
    - receiver: lark-monitoring
      matchers:
        - service="alb"
        - component="loadbalancer"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_LoadBalancer

    # ECS service sample alert labels, from aws:ecs_service:running_task_count:avg > 1:
    #   alert: alertname="AwsEcsServiceTaskScaledOut", environment="uat", service="ecs", component="service", dimension_ClusterName="ping-uat-cluster", dimension_ServiceName="group-data-center-backend-service"
    # Label source: alertname/service/component/severity from alert rule; environment/dimension_* kept by recording rule avg by (...) from YACE metric.
    # Result: one group per alertname/environment/service/component/dimension_ClusterName/dimension_ServiceName.
    - receiver: lark-monitoring
      matchers:
        - service="ecs"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_ClusterName
        - dimension_ServiceName

    # SQS sample alert labels:
    #   alert 1: alertname="AwsSqsOldestMessageAgeHigh", environment="prod", service="sqs", component="queue", dimension_QueueName="orders"
    #   alert 2: alertname="AwsSqsOldestMessageAgeHigh", environment="prod", service="sqs", component="queue", dimension_QueueName="payments"
    #   alert 3: alertname="AwsSqsMessagesVisibleHigh", environment="prod", service="sqs", component="queue", dimension_QueueName="orders"
    # Label source: alertname from alert rule; environment from YACE scrape/relabel; dimension_QueueName from YACE metric; service/component from alert rule labels.
    # Result: 3 groups, because queue name or alertname is different.
    - receiver: lark-monitoring
      matchers:
        - service="sqs"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_QueueName

    # DynamoDB sample alert labels:
    #   alert 1: alertname="AwsDynamoDbLatencyHigh", environment="prod", service="dynamodb", component="table", dimension_TableName="orders", dimension_Operation="GetItem"
    #   alert 2: alertname="AwsDynamoDbLatencyHigh", environment="prod", service="dynamodb", component="table", dimension_TableName="orders", dimension_Operation="PutItem"
    #   alert 3: alertname="AwsDynamoDbLatencyHigh", environment="prod", service="dynamodb", component="table", dimension_TableName="payments", dimension_Operation="GetItem"
    # Label source: alertname from alert rule; environment from YACE scrape/relabel; dimension_* from YACE metric; service/component from alert rule labels.
    # Result: 3 groups, because table or operation is different.
    - receiver: lark-monitoring
      matchers:
        - service="dynamodb"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_TableName
        - dimension_Operation

    # RDS / Aurora sample alert labels:
    #   alert 1: alertname="AwsRdsCpuUsedHigh", environment="prod", service="rds", component="database", dimension_DBClusterIdentifier="orders-db"
    #   alert 2: alertname="AwsRdsCpuUsedHigh", environment="prod", service="rds", component="database", dimension_DBClusterIdentifier="orders-db"
    #   alert 3: alertname="AwsRdsCpuUsedHigh", environment="prod", service="rds", component="database", dimension_DBClusterIdentifier="payments-db"
    # Label source: alertname from alert rule; environment from YACE scrape/relabel; dimension_DBClusterIdentifier from YACE metric; service/component from alert rule labels.
    # Result: alert 1 and alert 2 are in one group; alert 3 is another group.
    - receiver: lark-monitoring
      matchers:
        - service=~"rds|aurora"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_DBClusterIdentifier

    # ElastiCache / Valkey / Redis sample alert labels:
    #   alert 1: alertname="AwsElastiCacheMemoryUsedHigh", environment="prod", service="elasticache", component="cache", dimension_CacheClusterId="orders-cache-001"
    #   alert 2: alertname="AwsElastiCacheMemoryUsedHigh", environment="prod", service="elasticache", component="cache", dimension_CacheClusterId="orders-cache-001"
    #   alert 3: alertname="AwsElastiCacheSwapUsageDetected", environment="prod", service="elasticache", component="cache", dimension_CacheClusterId="orders-cache-001"
    # Label source: alertname from alert rule; environment from YACE scrape/relabel; dimension_CacheClusterId from YACE metric; service/component from alert rule labels.
    # Result: alert 1 and alert 2 are in one group; alert 3 is another group because alertname is different.
    - receiver: lark-monitoring
      matchers:
        - service=~"elasticache|valkey|redis"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_CacheClusterId

    # CloudFront sample alert labels:
    #   alert 1: alertname="AwsCloudFront5xxRateHigh", environment="prod", service="cloudfront", component="distribution", dimension_DistributionId="E123"
    #   alert 2: alertname="AwsCloudFront5xxRateHigh", environment="prod", service="cloudfront", component="distribution", dimension_DistributionId="E123"
    #   alert 3: alertname="AwsCloudFront5xxRateHigh", environment="prod", service="cloudfront", component="distribution", dimension_DistributionId="E456"
    # Label source: alertname from alert rule; environment from YACE scrape/relabel; dimension_DistributionId from YACE metric; service/component from alert rule labels.
    # Result: alert 1 and alert 2 are in one group; alert 3 is another group.
    - receiver: lark-monitoring
      matchers:
        - service="cloudfront"
      group_by:
        - alertname
        - environment
        - service
        - component
        - dimension_DistributionId

6. Service Decisions#

Service	Component	Recommended group_by resource	Why
`ec2`	`filesystem`	`instance_name`, `mountpoint`	不同 mountpoint 独立处理、独立恢复
`ec2`	`cpu` / `memory` / `host`	`instance_name`	一台机器一条生命周期
`alb`	`targetgroup`	`dimension_LoadBalancer`, `dimension_TargetGroup`	target group 是实际处理对象
`alb`	`loadbalancer`	`dimension_LoadBalancer`	ALB 级别问题按 LB 聚合
`ecs`	`service`	`dimension_ClusterName`, `dimension_ServiceName`	ECS service 是主要处理对象，cluster 用来区分同名 service
`sqs`	`queue`	`dimension_QueueName`	queue backlog / age 按队列处理
`dynamodb`	`table` / `operation`	`dimension_TableName`, `dimension_Operation`	table 和 operation 的处理动作不同
`rds` / `aurora`	`database`	`dimension_DBClusterIdentifier`	DB cluster 是主要处理对象
`elasticache` / `valkey` / `redis`	`cache`	`dimension_CacheClusterId`	cache cluster 是主要处理对象
`cloudfront`	`distribution`	`dimension_DistributionId`	distribution 是主要处理对象

7. Filesystem Example#

Labels:

alert 1:
    alertname=AwsEc2DiskUsedHigh
    environment=uat
    service=ec2
    component=filesystem
    instance_name=ping-uat-app
    mountpoint=/

alert 2:
    alertname=AwsEc2DiskUsedHigh
    environment=uat
    service=ec2
    component=filesystem
    instance_name=ping-uat-app
    mountpoint=/data

With:

group_by:
  - alertname
  - environment
  - service
  - component
  - instance_name
  - mountpoint

Result:

group 1:
    ping-uat-app /

group 2:
    ping-uat-app /data

这样 / 和 /data 可以分别 firing、分别 resolved、分别 repeat。适合你希望恢复通知清楚的资源类告警。

如果你希望一台机器的多个 mountpoint 合并到一条通知，去掉 mountpoint：

group_by:
  - alertname
  - environment
  - service
  - component
  - instance_name

这时一条通知里的 alerts[] 可能包含多个 mountpoint，adapter/card 必须能清楚列出每条 alert。

8. SLO Alerts#

服务级 SLO / 用户影响类告警不要按 instance / pod / target 维度打散。

Example:

route:
  routes:
    - receiver: lark-monitoring
      matchers:
        - component="slo"
      group_by:
        - alertname
        - environment
        - service

Reason:

SLO alert:
    代表服务整体用户体验
    值班人员先判断服务是否退化
    不应该被 instance / pod 维度拆成很多通知

9. Label Requirements#

group_by 只能使用最终到达 Alertmanager 的 alert labels。先确认 rule 没有把这些 label 聚合掉。

EC2 node exporter scrape labels:

scrape_configs:
  - job_name: node
    static_configs:
      - targets:
          - "ping-uat-exporters:9100"
        labels:
          service: ec2
          instance_name: ping-uat-devops
          environment: uat

CPU recording rule 如果使用 aggregation，要保留 instance_name：

100 - (
  avg by (environment, service, instance_name, instance) (
    rate(node_cpu_seconds_total{mode="idle"}[5m])
  ) * 100
)

Filesystem rule 没有 aggregation 时，通常会保留 scrape labels：

node:filesystem_used_percent{
  environment="uat",
  service="ec2",
  instance_name="ping-uat-devops",
  mountpoint="/"
}

如果 Alertmanager payload 里没有 instance_name，group_by 里写了也达不到预期。

有些 label 不是 exporter 原始 metric 自带的，而是你为了路由和分组在 alert rule 里补的。比如 YACE 采集的 ALB metric 里通常有 dimension_LoadBalancer、dimension_TargetGroup，但不一定有 service / component：

aws_applicationelb_request_count_sum{
  dimension_LoadBalancer="ping-uat-docmost-alb",
  dimension_TargetGroup="ping-uat-docmost-tg",
  environment="uat",
  job="yace"
}

如果 route 依赖：

matchers:
  - service="alb"
  - component="targetgroup"

就需要在 alert rule 里补上这些静态分类 label：

- alert: AwsAlbTargetGroupRequestCountLow
  expr: |
    sum by (environment, dimension_LoadBalancer, dimension_TargetGroup) (
      rate(aws_applicationelb_request_count_sum{dimension_TargetGroup!=""}[5m])
    ) < 1
  for: 5m
  labels:
    severity: warning
    service: alb
    component: targetgroup

这里 component 不是 YACE 自动识别出来的，而是 rule 作者定义的告警层级。动态资源 label 仍然应该来自 expr 结果，例如 dimension_LoadBalancer / dimension_TargetGroup。

10. Verify#

Check Alertmanager received labels:

curl -s http://alertmanager:9093/api/v2/alerts

Test filesystem route:

amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname=AwsEc2DiskUsedHigh \
  environment=uat \
  service=ec2 \
  component=filesystem \
  instance_name=ping-uat-devops \
  mountpoint=/

Test ALB target group route:

amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname=AwsAlbTargetGroupResponseTimeHigh \
  environment=uat \
  service=alb \
  component=targetgroup \
  dimension_LoadBalancer=ping-uat-alb \
  dimension_TargetGroup=cpa-http1

Checklist:

[ ] root group_by 是否是稳定默认值
[ ] 子 route 是否只覆盖需要特殊分组的资源类告警
[ ] group_by 里的 label 是否真的存在于 /api/v2/alerts
[ ] warning 是否会因为分组太细刷屏
[ ] resolved 是否需要按资源独立通知
[ ] card 是否能展示同一个 group 中的多条 alerts