Lark Inheritance

Lark Inheritance#

Alertmanager 的子 route 会继承父 route 中没有显式覆盖的配置。对 Lark / Feishu 告警来说，可以把默认 receiver、分组策略、重复通知间隔放在父路由，子路由只维护 matchers，这样不同 service / team 的告警可以复用同一个 Lark 发送入口。

1. Use Case#

Problem:
    多个业务都发 Lark 告警
    每个子路由都重复写 receiver / group_by / interval
    后续调整通知频率时容易漏改

Solution:
    父 route 定义 Lark receiver 和通用策略
    子 route 只定义匹配条件
    需要特殊升级时，子 route 再覆盖 receiver

2. Parent Route#

父路由设置默认 Lark receiver。没有设置 receiver 的子路由会继承这个 receiver。下面是一个多层 route 例子，覆盖 team、env、service、severity、region 几类常见分发场景。

group_by 和 matchers 都是基于 Alertmanager 收到的 alert labels，不是直接基于原始 metric name。通常这些 label 来自 Prometheus rule 的 labels:，也可能来自 PromQL 结果里保留下来的 metric labels。

groups:
  - name: application.rules
    rules:
      - alert: HighHttpErrorRate
        expr: |
          sum by (service, env, team) (
            rate(http_requests_total{status=~"5.."}[5m])
          )
          /
          sum by (service, env, team) (
            rate(http_requests_total[5m])
          ) > 0.05
        labels:
          severity: warning
          team: platform
        annotations:
          summary: "High HTTP 5xx error rate"

这条 rule 触发后，Alertmanager 看到的告警可以理解成：

ALERTS{
  alertname="HighHttpErrorRate",
  severity="warning",
  team="platform",
  service="checkout-api",
  env="prod"
}

所以：

matchers:
    team="platform" 匹配 alert label team
    severity="warning" 匹配 alert label severity

group_by:
    alertname / service / env 表示把这三个 label 值相同的 alerts 合并成一组通知

route:
  receiver: lark-default
  group_by:
    - alertname
    - service
    - env
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Platform team: receiver 使用 lark-platform
    # 继承 root route 的 group_by / group_wait / group_interval
    # repeat_interval 在这里覆盖为 2h
    # 匹配样例: ALERTS{alertname="HighHttpErrorRate", team="platform", service="checkout-api", env="prod", severity="warning"}
    - matchers:
        - team="platform"
      receiver: lark-platform
      repeat_interval: 2h
      routes:
        # warning: receiver 继承 lark-platform
        # repeat_interval 继承 platform route 的 2h
        # 匹配样例: ALERTS{alertname="HighHttpErrorRate", team="platform", service="checkout-api", env="prod", severity="warning"}
        - matchers:
            - severity="warning"

        # critical: receiver 继承 lark-platform
        # repeat_interval 覆盖为 30m
        # 匹配样例: ALERTS{alertname="ApiDown", team="platform", service="gateway-api", env="prod", severity="critical"}
        - matchers:
            - severity="critical"
          repeat_interval: 30m

        # database service: receiver 覆盖为 lark-dba
        # group_by 覆盖为 alertname / service / instance
        # 匹配样例: ALERTS{alertname="RedisMemoryHigh", team="platform", service="redis", instance="redis-01", env="prod", severity="warning"}
        - receiver: lark-dba
          matchers:
            - service=~"mysql|postgres|redis"
          group_by:
            - alertname
            - service
            - instance

    # Payment team: receiver 使用 lark-payment
    # repeat_interval 继承 root route 的 4h
    # 匹配样例: ALERTS{alertname="PaymentErrorRateHigh", team="payment", service="payment-api", env="prod", severity="warning"}
    - receiver: lark-payment
      matchers:
        - team="payment"
      routes:
        # prod critical: receiver 继承 lark-payment
        # repeat_interval 覆盖为 15m
        # 匹配样例: ALERTS{alertname="PaymentApiDown", team="payment", service="payment-api", env="prod", severity="critical"}
        - matchers:
            - env="prod"
            - severity="critical"
          repeat_interval: 15m

        # staging: receiver 继承 lark-payment
        # repeat_interval 覆盖为 12h，降低重复通知频率
        # 匹配样例: ALERTS{alertname="PaymentErrorRateHigh", team="payment", service="payment-api", env="staging", severity="warning"}
        - matchers:
            - env="staging"
          repeat_interval: 12h

    # EU region: receiver 使用 lark-eu-oncall
    # repeat_interval 继承 root route 的 4h
    # 匹配样例: ALERTS{alertname="HighLatency", team="platform", service="checkout-api", env="prod", region="eu-west-1", severity="warning"}
    - receiver: lark-eu-oncall
      matchers:
        - region="eu-west-1"
        - severity=~"critical|warning"

    # Low priority: receiver 使用 lark-low-priority
    # repeat_interval 覆盖为 24h
    # 匹配样例: ALERTS{alertname="DiskWillFillIn7Days", team="platform", service="node", env="prod", severity="info"}
    - receiver: lark-low-priority
      matchers:
        - severity="info"
      repeat_interval: 24h

3. Lark Receiver#

Alertmanager 没有内置 Lark receiver，通常使用 webhook adapter 把 Alertmanager webhook payload 转成 Lark bot message。

receivers:
  - name: lark-platform
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/platform
        send_resolved: true

4. Override Receiver#

如果不同业务、环境、严重级别要发到不同 Lark 群，可以只在需要改变发送目标的位置覆盖 receiver。下面是一个更完整的 receiver 覆盖样例。

route:
  receiver: lark-default
  group_by:
    - alertname
    - service
    - env
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Platform team 默认发 platform Lark 群
    # 匹配样例: ALERTS{alertname="HighHttpErrorRate", team="platform", service="checkout-api", env="prod", severity="warning"}
    - receiver: lark-platform
      matchers:
        - team="platform"
      routes:
        # 继承 lark-platform
        # 匹配样例: ALERTS{alertname="HighHttpErrorRate", team="platform", service="checkout-api", env="prod", severity="warning"}
        - matchers:
            - severity="warning"

        # 覆盖 receiver，发到 platform critical 群
        # 匹配样例: ALERTS{alertname="ApiDown", team="platform", service="gateway-api", env="prod", severity="critical"}
        - receiver: lark-platform-critical
          matchers:
            - severity="critical"
          repeat_interval: 30m

        # 覆盖 receiver，DB 告警发 DBA 群
        # 匹配样例: ALERTS{alertname="RedisMemoryHigh", team="platform", service="redis", instance="redis-01", env="prod", severity="warning"}
        - receiver: lark-dba
          matchers:
            - service=~"mysql|postgres|redis"

    # Payment team 默认发 payment Lark 群
    # 匹配样例: ALERTS{alertname="PaymentErrorRateHigh", team="payment", service="payment-api", env="prod", severity="warning"}
    - receiver: lark-payment
      matchers:
        - team="payment"
      routes:
        # payment prod critical 单独升级
        # 匹配样例: ALERTS{alertname="PaymentApiDown", team="payment", service="payment-api", env="prod", severity="critical"}
        - receiver: lark-payment-critical
          matchers:
            - env="prod"
            - severity="critical"
          repeat_interval: 15m

        # payment job 类告警发批处理群
        # 匹配样例: ALERTS{alertname="SettlementJobFailed", team="payment", service="settlement-job", job="settlement-daily", env="prod", severity="warning"}
        - receiver: lark-payment-batch
          matchers:
            - service=~"settlement-job|reconcile-job"
          group_by:
            - alertname
            - service
            - job

    # Security team 默认发 security Lark 群
    # 匹配样例: ALERTS{alertname="SuspiciousLoginSpike", team="security", service="auth-api", env="prod", severity="warning"}
    - receiver: lark-security
      matchers:
        - team="security"
      routes:
        # 安全 critical 永远走 security critical 群
        # 匹配样例: ALERTS{alertname="WafBlockedTrafficSpike", team="security", service="edge-waf", env="prod", severity="critical"}
        - receiver: lark-security-critical
          matchers:
            - severity="critical"
          repeat_interval: 10m

receivers:
  - name: lark-default
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/default
        send_resolved: true

  - name: lark-platform
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/platform
        send_resolved: true

  - name: lark-platform-critical
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/platform-critical
        send_resolved: true

  - name: lark-dba
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/dba
        send_resolved: true

  - name: lark-payment
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/payment
        send_resolved: true

  - name: lark-payment-critical
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/payment-critical
        send_resolved: true

  - name: lark-payment-batch
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/payment-batch
        send_resolved: true

  - name: lark-security
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/security
        send_resolved: true

  - name: lark-security-critical
    webhook_configs:
      - url: http://alertmanager-feishu:8080/webhook/security-critical
        send_resolved: true

命中关系：

team="platform", severity="warning":
    lark-platform

team="platform", severity="critical":
    lark-platform-critical

team="platform", service="redis":
    lark-dba

team="payment", env="prod", severity="critical":
    lark-payment-critical

team="payment", service="settlement-job":
    lark-payment-batch

team="security", severity="critical":
    lark-security-critical

5. Continue#

默认情况下，一个 alert 命中某个子路由后，不会继续匹配后面的 sibling route。需要同时发到多个 Lark 群时，使用 continue: true。

route:
  receiver: lark-default
  routes:
    # Payment team 告警先发 team 群，并继续匹配后面的 critical route
    # 匹配样例: ALERTS{alertname="PaymentApiDown", team="payment", service="payment-api", env="prod", severity="critical"}
    - receiver: lark-team
      matchers:
        - team="payment"
      continue: true

    # 所有 critical 告警发 critical 群
    # 匹配样例: ALERTS{alertname="PaymentApiDown", team="payment", service="payment-api", env="prod", severity="critical"}
    - receiver: lark-critical
      matchers:
        - severity="critical"

这个配置里，team="payment" 且 severity="critical" 的告警会同时发送到 lark-team 和 lark-critical。

6. Checklist#

Route:
    root route 必须配置 receiver
    通用 group_by / group_wait / group_interval / repeat_interval 放父 route
    子 route 只写 matchers 和真正需要覆盖的字段
    critical 告警可以覆盖 repeat_interval 或 receiver

Lark:
    Lark bot webhook 使用 Secret 管理
    adapter endpoint 按 team / purpose 命名
    firing 和 resolved 都要发送
    message body 保留 alertname / severity / service / env / summary / runbook_url

Noise Control:
    不要按 instance 作为默认 group_by
    用 inhibition 抑制同一故障下的 warning
    维护窗口使用 silence，不要临时删除 route