Label Lifecycle

Alertmanager Label Lifecycle#

Alertmanager 的 group_by、route.matchers、inhibit_rules.equal 都只看最终 alert labels。要判断 group_by 是否能正确聚合，必须知道从 scrape metrics 到 Alertmanager 中间有哪些地方会新增、删除、改名或覆盖 label。

https://prometheus.io/docs/prometheus/latest/configuration/configuration/
https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
https://prometheus.io/docs/prometheus/latest/querying/operators/
https://prometheus.io/docs/alerting/latest/configuration/
https://docs.victoriametrics.com/victoriametrics/vmagent/
https://docs.victoriametrics.com/victoriametrics/vmalert/

1. Label Sources#

exporter / application:
    metric 自带 labels
    example: method / status / instance / dimension_TableName

service discovery:
    __address__ / __scheme__ / __metrics_path__ / __meta_* internal labels
    这些 label 主要给 relabel_configs 使用

scrape config:
    job_name 生成 job label
    static_configs.labels 可以给 target 添加 labels
    relabel_configs 可以 keep / drop / replace / labelmap

scrape result:
    target labels 会附加到每条 scraped sample
    默认会有 job 和 instance

metric_relabel_configs:
    scrape 后、写入存储前处理每条 sample
    可以删除 sample、删除 label、改 label value

remote write / vmagent:
    可以在发送到 VictoriaMetrics 前继续 relabel
    可以配置全局额外 label

PromQL / MetricsQL expression:
    selector 保留原 series labels
    aggregation by (...) 只保留 by 里列出的 labels
    without (...) 删除指定 labels，保留其他 labels
    binary operation 可能根据 on / ignoring 改变保留的 label set

vmalert / Prometheus alerting rule:
    alertname 自动变成 alert label
    expr 结果 labels 会进入 alert
    rule labels 会添加到 alert，并覆盖同名 label
    group labels / external labels 如果配置了，也可能添加到 alert

Alertmanager:
    接收最终 labels
    route / group_by / inhibition / silence 都基于这些 labels
    annotations / startsAt / endsAt / generatorURL 不是 labels

2. Order#

1. exporter exposes metric labels
2. service discovery creates __meta_* internal labels
3. scrape relabel_configs modifies target labels before scrape
4. scrape attaches target labels to samples
5. metric_relabel_configs modifies sample labels before ingestion
6. vmagent remote write relabel / extra labels modifies outgoing samples
7. VictoriaMetrics stores final metric labels
8. vmalert query expression returns series labels
9. aggregation keeps or drops labels
10. alerting rule adds alertname and rule labels
11. vmalert external labels may be added
12. Alertmanager receives final alert labels

最容易丢 label 的地方：

metric_relabel_configs:
    labeldrop / labelkeep / replace 写错

aggregation:
    sum by (...) 没有保留后续 group_by 需要的 label

rule labels:
    手动配置了同名 label，覆盖了 expr 结果里的 label

external labels:
    使用了和业务 label 同名的 key，造成理解混乱

3. Where To Add Labels#

核心原则：

越靠近数据源:
    越适合加目标身份 label

越靠近告警规则:
    越适合加告警语义 label

按阶段拆开看：

Stage Good Labels Avoid Labels Why
vmagent scrape / relabel environment, region, cluster, account_id, instance_name, role, team, service component=filesystem, severity, alert_type 这里描述的是 target / metric 从哪里来、属于谁、在哪个环境
recording rule 保留 environment, service, instance_name, mountpoint, dimension_LoadBalancer, dimension_TargetGroup, dimension_QueueName 不需要的高基数 label，比如 pod_uid, container_id, request_id aggregation 会决定后续 alert 还能看到哪些 label
alert rule labels severity, component, team, alertgroup, runbook_class, service 动态资源值，比如固定写死 dimension_TargetGroup 这里描述的是这条告警的性质、路由、归属和处理方式

Node exporter example:

vmagent scrape labels:
    environment="uat"
    service="ec2"
    instance_name="ping-uat-devops"

alert rule labels:
    severity="warning"
    component="filesystem"

final alert labels:
    alertname="AwsEc2DiskUsedHigh"
    environment="uat"
    service="ec2"
    instance_name="ping-uat-devops"
    component="filesystem"
    severity="warning"

不要在 node exporter scrape 阶段统一加 component="filesystem"。同一个 node exporter target 会产生 CPU、memory、filesystem、network 等多类 metric；如果太早加 component=filesystem，CPU / memory 告警也可能继承错误的 component，并命中错误的 Alertmanager route。

4. vmagent Scrape Sample#

Scrape config:

global:
  external_labels:
    monitor: vmagent-uat

scrape_configs:
  - job_name: yace
    scrape_interval: 60s
    static_configs:
      - targets:
          - 127.0.0.1:5000
        labels:
          env: uat
          cluster: ping-uat
          exporter: yace

    relabel_configs:
      - target_label: region
        replacement: ap-east-1

    metric_relabel_configs:
      - regex: dimension_AvailabilityZone
        action: labeldrop
        # 示例目的：删除 AZ label，真实配置要按实际需要决定

Exporter 原始 metric：

aws_applicationelb_target_response_time_p95{
  dimension_LoadBalancer="ping-uat-alb",
  dimension_TargetGroup="sub2api-targets",
  dimension_AvailabilityZone=""
} 1.25

Scrape target labels:

job:
    yace

instance:
    127.0.0.1:5000

env / cluster / exporter:
    来自 static_configs.labels

region:
    来自 relabel_configs

After scrape and metric relabel:

aws_applicationelb_target_response_time_p95{
  job="yace",
  instance="127.0.0.1:5000",
  env="uat",
  cluster="ping-uat",
  exporter="yace",
  region="ap-east-1",
  dimension_LoadBalancer="ping-uat-alb",
  dimension_TargetGroup="sub2api-targets"
} 1.25

Notes:

__meta_*:
    只在 relabel 阶段使用，默认不会进入最终 metric

dimension_AvailabilityZone:
    这里被 metric_relabel_configs 删除

job / instance:
    常见默认 target labels

5. Label Conflict#

如果 scraped metric 自己带了 job / instance / env 这类 label，可能和 scrape target labels 冲突。

默认行为通常应该理解为：

target label:
    由 scrape config / service discovery / relabel_configs 产生

scraped label:
    exporter 暴露出来的原始 label

conflict:
    默认 target label 优先
    scraped 里的冲突 label 会被重命名成 exported_<label>

Example:

exporter sample:
    http_requests_total{job="app-self", env="inside-app"} 100

scrape target:
    job="order-api"
    env="prod"

stored sample:
    http_requests_total{
      job="order-api",
      env="prod",
      exported_job="app-self",
      exported_env="inside-app"
    } 100

如果设置 honor_labels: true，冲突时会保留 exporter 自己的 label。生产里要谨慎使用，因为它会让 scrape config 中预期的 job / env 被 exporter 覆盖。

6. vmalert Rule Sample#

vmalert 查询 VictoriaMetrics 里的 series。表达式返回哪些 labels，取决于 PromQL/MetricsQL。

groups:
  - name: alb-p1.rules
    interval: 1m
    labels:
      team: platform
    rules:
      - alert: AlbTargetResponseTimeHigh
        expr: |
          avg by (
            env,
            cluster,
            region,
            dimension_LoadBalancer,
            dimension_TargetGroup
          ) (
            aws_applicationelb_target_response_time_p95{
              env="uat",
              dimension_TargetGroup!="",
              dimension_LoadBalancer!="ping-uat-alb/canary"
            }
          ) > 1
        for: 5m
        labels:
          severity: P1
          service: alb
          component: targetgroup
          load_balancer: ping-uat-alb
        annotations:
          summary: "ALB target group p95 latency is high"
          description: "Target group {{ $labels.dimension_TargetGroup }} on {{ $labels.dimension_LoadBalancer }} has p95 latency > 1s."

Expression output labels:

{env="uat", cluster="ping-uat", region="ap-east-1", dimension_LoadBalancer="ping-uat-alb", dimension_TargetGroup="sub2api-targets"} 1.25

Why:

avg by (...):
    只保留 by 中列出的 labels
    job / instance / exporter 被聚合掉

dimension_TargetGroup:
    被保留，因为 by (...) 里写了它

dimension_AvailabilityZone:
    之前已经被 metric_relabel_configs 删除，后面无法恢复

Final alert labels sent to Alertmanager:

{
  alertname="AlbTargetResponseTimeHigh",
  env="uat",
  cluster="ping-uat",
  region="ap-east-1",
  dimension_LoadBalancer="ping-uat-alb",
  dimension_TargetGroup="sub2api-targets",
  team="platform",
  severity="P1",
  service="alb",
  component="targetgroup",
  load_balancer="ping-uat-alb"
}

Annotations are not labels:

summary:
    ALB target group p95 latency is high

description:
    Target group sub2api-targets on ping-uat-alb has p95 latency > 1s.

Alertmanager group_by 不能使用 summary / description，只能使用 labels 里的 key。

7. Add Routing Labels In Rule#

YACE 采集的 AWS metric 通常只带 AWS dimension，不会自动生成你想用于 Alertmanager route 的业务分类 label。

Raw metric:

aws_applicationelb_request_count_sum{
  dimension_LoadBalancer="ping-uat-docmost-alb",
  dimension_TargetGroup="ping-uat-docmost-tg",
  environment="uat",
  job="yace"
}

如果 Alertmanager route 需要按 service="alb"、component="targetgroup" 匹配，就在 alert rule 的 labels 里补：

- alert: AwsAlbTargetGroupRequestCountLow
  expr: |
    sum by (environment, dimension_LoadBalancer, dimension_TargetGroup) (
      rate(aws_applicationelb_request_count_sum{dimension_TargetGroup!=""}[5m])
    ) < 1
  for: 5m
  labels:
    severity: warning
    service: alb
    component: targetgroup

结果：

dimension_LoadBalancer / dimension_TargetGroup:
    来自 expr 结果，是动态资源 label

service / component / severity:
    来自 rule labels，是静态分类 label

不要把动态资源值写死在 rule labels 里，例如不要固定写 dimension_TargetGroup: ping-uat-docmost-tg。这会让所有 alert 都变成同一个 target group。

8. group_by Design#

Alertmanager example:

route:
  receiver: lark-monitoring
  group_by:
    - alertname
    - env
    - service
    - component
    - dimension_LoadBalancer
    - dimension_TargetGroup
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

这个 group_by 能正常工作，因为最终 alert labels 中确实存在：

alertname
env
service
component
dimension_LoadBalancer
dimension_TargetGroup

如果 rule 写成：

avg by (env, cluster, region, dimension_LoadBalancer) (...)

最终 alert labels 里就没有 dimension_TargetGroup。这时 Alertmanager 不能按 TargetGroup 分组，同一个 ALB 下多个 target group 的告警可能被合并到同一个 group。

Rule labels 可以补静态 label：

labels:
  severity: P1
  service: alb
  component: targetgroup

Rule labels 不适合补动态资源 label：

bad:
    labels:
      dimension_TargetGroup: sub2api-targets

reason:
    每条 alert 的 target group 应该来自 expr 结果 label
    静态写死会让所有 alert 都变成同一个 target group

9. Complete Label Timeline#

exporter:
    dimension_LoadBalancer="ping-uat-alb"
    dimension_TargetGroup="sub2api-targets"
    dimension_AvailabilityZone=""

scrape target:
    job="yace"
    instance="127.0.0.1:5000"
    env="uat"
    cluster="ping-uat"
    exporter="yace"

relabel_configs:
    region="ap-east-1"

metric_relabel_configs:
    drop dimension_AvailabilityZone

stored metric:
    job="yace"
    instance="127.0.0.1:5000"
    env="uat"
    cluster="ping-uat"
    exporter="yace"
    region="ap-east-1"
    dimension_LoadBalancer="ping-uat-alb"
    dimension_TargetGroup="sub2api-targets"

vmalert expr avg by:
    keep env / cluster / region / dimension_LoadBalancer / dimension_TargetGroup
    drop job / instance / exporter

vmalert group labels:
    team="platform"

vmalert rule labels:
    severity="P1"
    service="alb"
    component="targetgroup"
    load_balancer="ping-uat-alb"

auto alert label:
    alertname="AlbTargetResponseTimeHigh"

Alertmanager receives:
    alertname
    env
    cluster
    region
    dimension_LoadBalancer
    dimension_TargetGroup
    team
    severity
    service
    component
    load_balancer

10. How To Verify#

Check stored metric labels:

aws_applicationelb_target_response_time_p95{env="uat", dimension_TargetGroup!=""}

Check vmalert active alerts:

curl -s http://vmalert:8880/api/v1/alerts
curl -s http://vmalert:8880/api/v1/rules

Check Alertmanager received labels:

curl -s http://alertmanager:9093/api/v2/alerts

Test route with the same labels:

amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname=AlbTargetResponseTimeHigh \
  env=uat \
  service=alb \
  component=targetgroup \
  dimension_LoadBalancer=ping-uat-alb \
  dimension_TargetGroup=sub2api-targets \
  severity=P1 \
  team=platform

Rule of thumb:

If a label is missing in stored metric:
    check scrape config / relabel_configs / metric_relabel_configs

If a label is present in stored metric but missing in alert:
    check PromQL aggregation by (...) / without (...)

If a label is present in alert but value is unexpected:
    check rule labels / group labels / external labels for overwrite

If Alertmanager group_by does not work:
    inspect /api/v2/alerts and confirm that every group_by key exists in final labels

11. Checklist#

Before choosing group_by:
    [ ] inspect raw metric labels in Grafana Explore / VMUI
    [ ] confirm scrape target labels: job / instance / env / cluster / region
    [ ] review relabel_configs and metric_relabel_configs
    [ ] check aggregation keeps resource labels needed by group_by
    [ ] avoid high-cardinality labels such as pod_uid / request_id
    [ ] keep stable labels: alertname / env / service / component / resource
    [ ] verify final Alertmanager labels via /api/v2/alerts

Good group_by examples:
    alertname / env / service
    alertname / env / service / component
    alertname / env / service / dimension_QueueName
    alertname / env / service / dimension_TableName / dimension_Operation

Risky group_by examples:
    instance
    pod
    container_id
    request_id
    dynamically generated rule labels
Backward Lark Inheritance
GroupBy Forward