Prometheus And vmagent Relabeling 


https://prometheus.io/docs/prometheus/latest/configuration/configuration/
https://docs.victoriametrics.com/relabeling/

1. What Relabeling Solves#

rename labels
copy labels
drop labels
drop samples
keep only samples that match a condition
normalize tag names
reduce cardinality
prepare labels for dashboards and alerts

最常见场景：

custom_tag_environment -> environment
tag_Environment -> environment
drop exporter self-metrics such as go_* / process_*
drop noisy labels not used by alert or dashboard
route scrape target metadata into instance / job / __param_target

2. Three Stages#

relabel_configs#

作用阶段：

target scrape 之前
处理的是 target labels，不是样本本身
常用于改 __address__、__param_target、instance、job

典型用途：

relabel_configs:
  - source_labels: [__address__]
    target_label: __param_target

  - source_labels: [__param_target]
    target_label: instance

  - target_label: __address__
    replacement: blackbox-exporter:9115

metric_relabel_configs#

作用阶段：

target 已经抓到样本之后
写入 TSDB / remote write 之前
处理的是每一个 metric sample 的 labels 和样本是否保留

典型用途：

metric_relabel_configs:
  - source_labels: [custom_tag_environment]
    regex: "(.+)"
    target_label: environment
    replacement: "$1"
    action: replace

  - regex: "custom_tag_environment"
    action: labeldrop

write_relabel_configs#

作用阶段：

只在 remote write 前生效
常用于只对外发出的数据做过滤和改名
本地 TSDB 里的数据不受影响

3. Core Fields#

下面这些字段不是每个 action 都必须用，但它们构成了 relabeling 的核心语法。

source_labels#

写法：

source_labels: [label_a, label_b]

作用：

从哪些现有 labels 取值
多个 label 会先拼接，再交给 regex / action 处理

例子：

source_labels: [custom_tag_environment]

separator#

默认：

;

作用：

多个 source_labels 拼接时，中间用什么字符连接

例子：

source_labels: [namespace, pod]
separator: "/"
target_label: workload
replacement: "$1"

如果 namespace=prod、pod=api-0，拼接串会是：

prod/api-0

regex#

默认：

(.*)

作用：

决定匹配条件
也决定 replacement 里的 $1 $2 ... 能取到哪些分组

例子：

regex: "(.+)"

target_label#

作用：

把结果写到哪个 label

例子：

target_label: environment

replacement#

默认：

$1

作用：

regex 匹配成功后，最终写入 target_label 的值

例子：

replacement: "$1"
replacement: "prod-$1"
replacement: "static-value"

modulus#

作用：

只在 hashmod action 中使用
对 source_labels 拼接结果做 hash，再取模

例子：

source_labels: [instance]
modulus: 16
target_label: shard
action: hashmod

action#

作用：

决定这条 relabel rule 到底做什么

if#

这是 VictoriaMetrics / vmagent 常见的扩展写法，用于先按 PromQL label selector 形式限制规则生效范围。

写法：

if: '{job="yace"}'

作用：

只有满足 selector 的样本或 target，才继续应用这条 relabel rule
适合在同一个 scrape_config 里只改某一类 metrics

4. Action Full Table#

下面这张表覆盖 Prometheus 标准 relabel actions，也是 vmagent 常用的兼容动作集合。

Action What It Does Typical Use
replace 匹配 regex 后，把 replacement 写入 target_label rename / copy label
keep 匹配 regex 的样本保留，其余样本丢弃 keep only wanted metrics
drop 匹配 regex 的样本丢弃，其余保留 drop noisy metrics
hashmod hash 后取模，把结果写入 target_label scrape sharding
labelmap 按 label name 匹配并重命名 label names bulk rename labels
labeldrop 按 label name 删除 labels remove high-cardinality labels
labelkeep 只保留匹配的 label names keep only a controlled label set
lowercase 把拼接结果转小写写入 target_label normalize values
uppercase 把拼接结果转大写写入 target_label normalize values
keepequal 只有 source value 等于 target_label 当前值时才保留样本 equality gate
dropequal 当 source value 等于 target_label 当前值时丢弃样本 equality-based drop

5. Every Action In Detail#

replace#

最常用。

- source_labels: [custom_tag_environment]
  regex: "(.+)"
  target_label: environment
  replacement: "$1"
  action: replace

适合：

复制 label
改 label 名
做简单拼接
写固定值

keep#

保留匹配样本，其他全部丢弃。

- source_labels: [__name__]
  regex: "aws_cloudfront_.*"
  action: keep

适合：

只保留某一类 metrics
remote write 前裁剪数据量

drop#

丢掉匹配样本。

- source_labels: [__name__]
  regex: "go_.*|process_.*|promhttp_.*"
  action: drop

适合：

删除 exporter 自身指标
删除已知没价值的样本

注意：

drop 删除的是整个 sample
不是只删某个 label

hashmod#

- source_labels: [instance]
  modulus: 8
  target_label: shard
  action: hashmod

适合：

做水平分片
把 target 或 sample 分到固定桶

labelmap#

按 label name 匹配，不是按 label value 匹配。

- regex: "tag_(.+)"
  replacement: "$1"
  action: labelmap

效果示意：

tag_Environment -> Environment
tag_Project -> Project

适合：

批量改 label 名
不想一条条写 replace

labeldrop#

按 label name 删除。

- regex: "custom_tag_environment"
  action: labeldrop

适合：

保留 sample，只删除某个 label
清理高基数或临时 label

labelkeep#

只保留匹配到的 label names，其他 label 都删掉。

- regex: "__name__|job|instance|environment|dimension_.*"
  action: labelkeep

适合：

强制控制 label 集合
remote write 前瘦身

风险：

写错 regex 会把有用 labels 一起删掉

lowercase#

- source_labels: [Environment]
  target_label: environment
  action: lowercase

适合：

把 Env / ENV / prod / PROD 统一成小写值

uppercase#

- source_labels: [region]
  target_label: region_upper
  action: uppercase

适合：

需要固定大写格式时使用

keepequal#

只有 source value 和 target_label 当前值完全相等时才保留。

- source_labels: [namespace]
  target_label: environment
  action: keepequal

适合：

当你已经有 target_label，想做严格相等校验

限制：

这个动作是“相等比较”
不是“regex 匹配 + replacement”
实际使用时应保持规则极简
通常只需要:
    source_labels
    target_label
    action

dropequal#

当 source value 和 target_label 当前值完全相等时丢弃。

- source_labels: [namespace]
  target_label: environment
  action: dropequal

适合：

排除某类完全相等的样本

限制：

和 keepequal 一样
它是相等比较动作，不是正则替换动作
通常只需要:
    source_labels
    target_label
    action

6. Which Fields Matter For Which Action#

Action source_labels regex target_label replacement modulus
replace yes yes yes yes no
keep yes yes no no no
drop yes yes no no no
hashmod yes no yes no yes
labelmap no yes no yes no
labeldrop no yes no no no
labelkeep no yes no no no
lowercase yes no yes no no
uppercase yes no yes no no
keepequal yes no yes no no
dropequal yes no yes no no

注意：

没有写 regex 时，不代表 action 不做匹配
而是使用默认值或 action 自己的比较逻辑

`keepequal` / `dropequal`:
    应按“source_labels 与 target_label 是否完全相等”来理解
    不要把它们当成 replace 的变种

7. Common Patterns#

rename one label#

metric_relabel_configs:
  - source_labels: [custom_tag_environment]
    regex: "(.+)"
    target_label: environment
    replacement: "$1"
    action: replace

copy then drop original label#

metric_relabel_configs:
  - source_labels: [tag_Environment]
    regex: "(.+)"
    target_label: environment
    replacement: "$1"
    action: replace

  - regex: "tag_Environment"
    action: labeldrop

drop exporter self-metrics#

metric_relabel_configs:
  - source_labels: [__name__]
    regex: "go_.*|process_.*|promhttp_.*"
    action: drop

keep only one metric family#

metric_relabel_configs:
  - source_labels: [__name__]
    regex: "aws_cloudfront_.*"
    action: keep

combine multiple labels into one#

metric_relabel_configs:
  - source_labels: [environment, dimension_ServiceName]
    separator: "/"
    target_label: service_key
    replacement: "$1"
    action: replace

remove noisy labels#

metric_relabel_configs:
  - regex: "pod_uid|container_id|image_id"
    action: labeldrop

8. YACE Examples#

custom_tag_environment -> environment#

metric_relabel_configs:
  - source_labels: [custom_tag_environment]
    regex: "(.+)"
    target_label: environment
    replacement: "$1"
    action: replace

  - regex: "custom_tag_environment"
    action: labeldrop

tag_Environment -> environment#

metric_relabel_configs:
  - source_labels: [tag_Environment]
    regex: "(.+)"
    target_label: environment
    replacement: "$1"
    action: replace

when not to use target-level static labels#

bad fit:
    one YACE target scrapes uat and prod together
    one YACE target scrapes multiple AWS accounts

why:
    static target labels apply to every sample from that scrape target
    they cannot distinguish per-job or per-resource context

9. How To Choose The Right Action#

want to rename or copy a label:
    replace

want to remove a label only:
    labeldrop

want to remove the whole sample:
    drop

want to keep only matching samples:
    keep

want to keep only some label names:
    labelkeep

want to rename many label names at once:
    labelmap

want to shard:
    hashmod

want to normalize case:
    lowercase / uppercase

10. Common Mistakes#

mistake:
    use drop when you only want to remove a label
result:
    the whole metric sample disappears

mistake:
    forget that regex matches the concatenated source_labels string
result:
    replacement or keep/drop condition does not work as expected

mistake:
    assume tag_environment and tag_Environment are the same
result:
    relabel rule never matches

mistake:
    use labelkeep with too narrow regex
result:
    useful labels vanish and dashboards break

mistake:
    use target-level static labels for multi-environment YACE scrape
result:
    all samples get the same fake environment

11. Debug Checklist#

  1. Confirm the label exists before relabeling. Example: query the exporter endpoint or raw metric sample first.

  2. Confirm the exact label name and case. Example: tag_Environment vs tag_environment.

  3. Confirm whether the rule is in relabel_configs or metric_relabel_configs. Target relabel and sample relabel are different stages.

  4. Confirm whether you intended to drop a label or a whole sample. labeldrop and drop are not interchangeable.

  5. Confirm the final stored sample in VMUI. Query the exact metric and inspect its labels.

12. Minimal Practical Example#

目标：

YACE 暴露:
    tag_Environment="uat"

dashboard 统一要求:
    environment="uat"

配置：

scrape_configs:
  - job_name: yace
    static_configs:
      - targets:
          - yace:5001

    metric_relabel_configs:
      - source_labels: [custom_tag_environment]
        regex: "(.+)"
        target_label: environment
        replacement: "$1"
        action: replace

      - source_labels: [tag_Environment]
        regex: "(.+)"
        target_label: environment
        replacement: "$1"
        action: replace

      - regex: "custom_tag_environment"
        action: labeldrop

结果：

before:
    tag_Environment="uat"
    custom_tag_environment="uat"

after:
    tag_Environment="uat"
    environment="uat"

如果你还想删掉原来的 tag_Environment：

      - regex: "tag_Environment"
        action: labeldrop
Backward Exporters
Prometheus Alerting Forward