Feishu

Feishu / Lark#

Alertmanager 没有内置 Feishu / Lark receiver。生产环境通常不要让 Alertmanager 直接调用 Lark custom bot webhook，而是放一个很小的 webhook adapter：

vmalert / Prometheus
  -> Alertmanager
  -> Lark adapter
  -> Lark custom bot webhook

原因很简单：Alertmanager webhook payload 和 Lark bot message payload 不是同一种 JSON。adapter 负责格式转换、签名、重试、脱敏和统一消息模板。

1. Links#

https://prometheus.io/docs/alerting/latest/configuration/#webhook_config
https://prometheus.io/docs/alerting/latest/notifications/
https://open.feishu.cn/document/client-docs/bot-v3/add-custom-bot
https://open.larksuite.com/document/client-docs/bot-v3/add-custom-bot

2. Important Points#

Alertmanager:
    只负责 route / group / silence / inhibition
    webhook_configs 发送 Alertmanager 标准 JSON

Lark adapter:
    接收 Alertmanager JSON
    转成 Lark text 或 card message
    使用 Lark custom bot webhook URL
    如果机器人开启签名校验，再额外使用 signing secret
    失败时快速记录日志，返回明确状态

Lark custom bot:
    webhook URL 本身带有 hook token，属于 secret
    URL 里的 UUID / token 不是 signing secret
    signing secret 是机器人安全设置里开启签名校验后单独生成的值
    建议开启签名校验
    不要把 webhook url 写入 Git / Docmost / dashboard

如果所有 P1 / P2 告警都发同一个群，Alertmanager 不需要按 service / component 分 route。保留这些 labels 只是为了消息内容、分组、silence 和排障。

3. Alertmanager Config#

统一通知到一个 Lark receiver：

global:
  resolve_timeout: 5m

route:
  receiver: lark-monitoring
  group_by:
    - alertname
    - environment
    - severity
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: lark-monitoring
    webhook_configs:
      - url: http://127.0.0.1:19090/alertmanager/lark
        send_resolved: true
        max_alerts: 20

如果 adapter 和 Alertmanager 不在同一台机器，用内网地址：

receivers:
  - name: lark-monitoring
    webhook_configs:
      - url: http://lark-alertmanager-adapter:19090/alertmanager/lark
        send_resolved: true

不要把 Lark bot webhook 直接放到 Alertmanager：

bad:
    Alertmanager -> https://open.feishu.cn/open-apis/bot/v2/hook/xxx

reason:
    Alertmanager JSON payload is not Lark bot message payload.

4. Adapter Environment#

export PORT=19090
export LARK_WEBHOOK_URL='https://open.larksuite.com/open-apis/bot/v2/hook/526a9671-82da-4cc4-bb48-a7125e52f526'
# 如果 Lark bot 没有开启签名校验，可以不设置这个变量或留空。
export LARK_SIGNING_SECRET=''
export LOG_LEVEL=info

说明：

LARK_WEBHOOK_URL:
    Lark / Feishu 群机器人 webhook URL
    URL 自身已经包含 hook token，例如:
        https://open.larksuite.com/open-apis/bot/v2/hook/<uuid>
        https://open.feishu.cn/open-apis/bot/v2/hook/<uuid>
    这个 URL 可以直接用于发送 Lark bot message

LARK_SIGNING_SECRET:
    机器人安全设置里的签名 secret
    它不是 webhook URL 里的 UUID / token
    如果没有开启签名，可以留空，但生产建议开启

PORT:
    adapter 监听端口

先直接测试 Lark bot webhook，确认机器人本身可用：

curl -sS -X POST "$LARK_WEBHOOK_URL" \
  -H 'content-type: application/json' \
  -d '{
    "msg_type": "text",
    "content": {
      "text": "manual lark bot test"
    }
  }'

如果这个测试失败，先检查 Lark bot 的关键词校验、签名校验、IP 白名单和 webhook URL，不要先排查 Alertmanager。

5. Node.js Adapter#

package.json:

{
  "name": "lark-alertmanager-adapter",
  "version": "1.0.0",
  "type": "module",
  "private": true,
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.19.2"
  }
}

server.js:

import crypto from 'node:crypto';
import express from 'express';

const app = express();
app.use(express.json({ limit: '1mb' }));

const port = Number(process.env.PORT || 19090);
const listenAddr = process.env.LISTEN_ADDR || '127.0.0.1';
const webhookUrl = process.env.LARK_WEBHOOK_URL;
// Optional. Only set this when Lark custom bot security uses signature verification.
const signingSecret = process.env.LARK_SIGNING_SECRET || '';

if (!webhookUrl) {
  throw new Error('LARK_WEBHOOK_URL is required');
}

function larkSign(timestamp, secret) {
  if (!secret) return undefined;

  const stringToSign = `${timestamp}\n${secret}`;
  return crypto
    .createHmac('sha256', stringToSign)
    .update('')
    .digest('base64');
}

function value(obj, key, fallback = '-') {
  const v = obj && obj[key];
  return v === undefined || v === null || v === '' ? fallback : v;
}

function firstValue(...values) {
  for (const v of values) {
    if (v !== undefined && v !== null && v !== '') return v;
  }
  return '-';
}

function severityColor(status, severity) {
  if (String(status || '').toLowerCase() === 'resolved') return 'green';

  const normalized = String(severity || '').toLowerCase();
  if (['critical', 'p0', 'p1'].includes(normalized)) return 'red';
  if (['warning', 'p2'].includes(normalized)) return 'orange';
  if (['info', 'p3'].includes(normalized)) return 'blue';
  return 'yellow';
}

function markdown(content) {
  return {
    tag: 'markdown',
    content,
  };
}

function field(label, fieldValue) {
  return markdown(`**${label}:** ${value({ v: fieldValue }, 'v')}`);
}

function compactLine(items) {
  return items
    .filter((item) => item && item.value !== undefined && item.value !== null && item.value !== '')
    .map((item) => `**${item.label}:** ${item.value}`)
    .join('    ');
}

function parseTime(ts) {
  if (!ts || ts.startsWith('0001-01-01')) return undefined;
  const d = new Date(ts);
  return Number.isNaN(d.getTime()) ? undefined : d;
}

function formatDuration(start, end) {
  const s = parseTime(start);
  const e = parseTime(end);
  if (!s || !e || e < s) return '-';

  let seconds = Math.round((e.getTime() - s.getTime()) / 1000);
  const days = Math.floor(seconds / 86400);
  seconds %= 86400;
  const hours = Math.floor(seconds / 3600);
  seconds %= 3600;
  const minutes = Math.floor(seconds / 60);
  seconds %= 60;

  const parts = [];
  if (days) parts.push(`${days}d`);
  if (hours) parts.push(`${hours}h`);
  if (minutes) parts.push(`${minutes}m`);
  if (!parts.length || seconds) parts.push(`${seconds}s`);
  return parts.slice(0, 3).join(' ');
}

function envOf(labels) {
  return firstValue(labels.env, labels.environment);
}

const BODY_LABEL_ORDER = [
  'region',
  'cluster',
  'namespace',
  'instance_name',
  'mountpoint',
  'resource',
  'dimension_ClusterName',
  'dimension_ServiceName',
  'dimension_LoadBalancer',
  'dimension_TargetGroup',
  'dimension_QueueName',
  'dimension_TableName',
  'dimension_Operation',
  'dimension_DBClusterIdentifier',
  'dimension_CacheClusterId',
  'dimension_DistributionId',
];

const HIDDEN_BODY_LABELS = new Set([
    'alertname',
    'severity',
    'env',
    'environment',
    'service',
    'component',
    'instance',
    'job',
    'team',
    'alertgroup',
    'account_id',
    'name',
    'prometheus',
    'replica',
]);

function isBodyLabel(key) {
  return (
    key === 'resource' ||
    key === 'region' ||
    key === 'cluster' ||
    key === 'namespace' ||
    key === 'instance_name' ||
    key === 'mountpoint' ||
    key.startsWith('dimension_')
  );
}

function bodyLabelEntries(labels) {
  return Object.entries(labels)
    .filter(([key, v]) => !HIDDEN_BODY_LABELS.has(key) && isBodyLabel(key) && v !== undefined && v !== null && v !== '')
    .sort(([a], [b]) => {
      const ai = BODY_LABEL_ORDER.includes(a) ? BODY_LABEL_ORDER.indexOf(a) : 100;
      const bi = BODY_LABEL_ORDER.includes(b) ? BODY_LABEL_ORDER.indexOf(b) : 100;
      if (ai !== bi) return ai - bi;
      return a.localeCompare(b);
    })
    .slice(0, 10);
}

function buildAlertElements(alert) {
  const labels = alert.labels || {};
  const annotations = alert.annotations || {};
  const duration = alert.status === 'resolved' ? formatDuration(alert.startsAt, alert.endsAt) : formatDuration(alert.startsAt, new Date().toISOString());

  const elements = [];

  if (labels.component) {
    elements.push(field('Component', labels.component));
  }

  const labelLines = bodyLabelEntries(labels)
    .map(([key, v]) => `- ${key}: ${v}`)
    .join('\n');
  if (labelLines) {
    elements.push(markdown(`**Resource Labels:**\n${labelLines}`));
  }

  if (annotations.description) {
    elements.push(markdown(`**Description:**\n${annotations.description}`));
  }

  // const links = [
  //   annotations.dashboard_url ? `[Dashboard](${annotations.dashboard_url})` : '',
  //   annotations.runbook_url ? `[Runbook](${annotations.runbook_url})` : '',
  //   alert.generatorURL ? `[Source](${alert.generatorURL})` : '',
  // ].filter(Boolean);
  // if (links.length) {
  //   elements.push(markdown(`**Links:** ${links.join(' | ')}`));
  // }

  elements.push(markdown(compactLine([
    { label: 'StartsAt', value: alert.startsAt },
    { label: 'EndsAt', value: alert.status === 'resolved' ? alert.endsAt : '' },
    { label: 'Duration', value: duration },
  ])));

  return elements;
}

function cardTitle(status, commonLabels, alerts) {
  const firstAlert = alerts[0] || {};
  const labels = { ...(commonLabels || {}), ...(firstAlert.labels || {}) };
  const severity = value(labels, 'severity', 'unknown').toUpperCase();
  const env = envOf(labels);
  const service = value(labels, 'service', 'service');
  const summary = firstAlert.annotations?.summary || value(labels, 'alertname', 'Alertmanager');
  return `[${String(status || 'unknown').toUpperCase()}] ${severity} | ${env} | ${service} | ${summary}`;
}

function buildLarkCard(payload) {
  const status = (payload.status || 'unknown').toUpperCase();
  const commonLabels = payload.commonLabels || {};
  const alerts = Array.isArray(payload.alerts) ? payload.alerts : [];
  const severity = value(commonLabels, 'severity', '');
  const title = cardTitle(status, commonLabels, alerts);

  const elements = [];
  if (alerts.length > 1) {
    elements.push(field('Alerts', alerts.length));
  }

  for (const [index, alert] of alerts.entries()) {
    if (index > 0) elements.push({ tag: 'hr' });
    elements.push(...buildAlertElements(alert));
  }

  return {
    msg_type: 'interactive',
    card: {
      config: {
        wide_screen_mode: true,
      },
      header: {
        template: severityColor(payload.status, severity),
        title: {
          tag: 'plain_text',
          content: title,
        },
      },
      elements,
    },
  };
}

async function sendToLark(message) {
  const timestamp = Math.floor(Date.now() / 1000).toString();
  const body = { ...message };

  const sign = larkSign(timestamp, signingSecret);
  if (sign) {
    body.timestamp = timestamp;
    body.sign = sign;
  }

  const response = await fetch(webhookUrl, {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify(body),
  });

  const responseText = await response.text();
  if (!response.ok) {
    throw new Error(`Lark webhook failed: ${response.status} ${responseText}`);
  }

  return responseText;
}

app.get('/healthz', (req, res) => {
  res.json({ ok: true });
});

app.post('/alertmanager/lark', async (req, res) => {
  console.log('received alertmanager webhook payload:');
  console.log(JSON.stringify(req.body || {}, null, 2));
    
  try {
    const message = buildLarkCard(req.body || {});
    await sendToLark(message);
    res.json({ ok: true });
  } catch (error) {
    console.error(error);
    res.status(502).json({ ok: false, error: error.message });
  }
});

app.listen(port, listenAddr, () => {
  console.log(`lark alertmanager adapter listening on ${listenAddr}:${port}`);
});

关键点：

listen address:
    默认绑定 127.0.0.1，适合同机 Alertmanager 调用
    如果 Docker / remote host 需要访问，再改成 0.0.0.0 并加防火墙限制

response:
    Lark webhook 调用成功才返回 2xx
    失败返回 502，让 Alertmanager 有机会重试

message:
    使用 Lark interactive card
    firing critical / P1 使用 red header
    warning / P2 使用 orange header
    resolved 使用 green header
    title 使用 status + severity + env + service + summary
    body 分为 component / resource labels / description / links / timeline
    不在 body 重复展示 title 已经包含的 status / severity / env / service
    不在 body 重复展示 title 已经包含的 summary
    不展示 Alertmanager group labels 和 receiver

resource labels:
    不按 service 写特殊分支
    统一展示稳定的资源定位 labels
    例如 instance_name / mountpoint / region / cluster / dimension_*
    隐藏 alertname / severity / environment / service / job / instance / account_id 等噪音 label
    如果需要展示触发值，在 rule annotations 里显式添加 value / threshold

alerts count:
    Alerts 表示同一个 Alertmanager group 里这次通知包含多少条 alert
    单条 alert 时不显示
    多条 alert 被 group_by 聚合到一起时才显示 Alerts: N

timeline:
    firing 显示 StartsAt 和当前持续时间
    resolved 显示 StartsAt / EndsAt / Duration
    resolved card 一眼能看到这次告警持续了多久

payload size:
    max_alerts 控制 Alertmanager 一次推送的 alerts 数量
    express json limit 防止异常大 payload

debug log:
    adapter 会打印完整 Alertmanager webhook payload
    可以看到 alerts[] / startsAt / endsAt / labels / annotations
    生产长期运行时建议关闭完整 payload 日志或做脱敏

当前 card 标题示例：

[FIRING] CRITICAL | prod | sqs | SQS oldest message age is high
[RESOLVED] CRITICAL | uat | alb | ALB ELB 5xx rate is high
[FIRING] WARNING | prod | ec2 | EC2 filesystem usage is high

card body 示例：

Component: filesystem

Resource Labels:
- instance_name: ping-dev-exporters
- mountpoint: /

Description:
EC2 node ping-dev-exporters:9100 mount / in dev has filesystem used percent > 90% for 15m.

Links: Source

StartsAt: 2026-06-30T11:07:00Z    Duration: 12h 5m 12s

6. Docker Compose#

文件结构：

lark-alertmanager-adapter-compose/
    compose.yml
    .env
    lark-alertmanager-adapter/
        package.json
        package-lock.json
        server.js

compose.yml：

services:
  lark-alertmanager-adapter:
    image: node:20-alpine
    working_dir: /app
    command: sh -c "npm ci --omit=dev && node server.js"
    restart: unless-stopped
    environment:
      PORT: "19090"
      LISTEN_ADDR: "0.0.0.0"
      LARK_WEBHOOK_URL: ${LARK_WEBHOOK_URL}
      LARK_SIGNING_SECRET: ${LARK_SIGNING_SECRET}
    volumes:
      - ./lark-alertmanager-adapter:/app
      - lark-alertmanager-adapter-node-modules:/app/node_modules
    ports:
      - "127.0.0.1:19090:19090"

volumes:
  lark-alertmanager-adapter-node-modules:

这里的 top-level volumes 是 Docker Compose named volume 定义：

volumes:
  lark-alertmanager-adapter-node-modules:

它和 service 里的 /app/node_modules 挂载配合使用：

services:
  lark-alertmanager-adapter:
    volumes:
      - ./lark-alertmanager-adapter:/app
      - lark-alertmanager-adapter-node-modules:/app/node_modules

含义：

/app/package.json:
    来自宿主机 ./lark-alertmanager-adapter/package.json

/app/server.js:
    来自宿主机 ./lark-alertmanager-adapter/server.js

/app/node_modules:
    来自 Docker named volume
    不直接写到宿主机源码目录

为什么需要：

npm ci:
    会写入 /app/node_modules

source mount:
    ./lark-alertmanager-adapter:/app 用来把源码放进容器

named volume:
    单独接管 /app/node_modules
    避免 node_modules 写入宿主机源码目录
    避免只读挂载或权限导致 npm ci 创建 node_modules 失败

查看 volume：

docker volume ls | grep lark-alertmanager
docker volume inspect "$(docker volume ls --format '{{.Name}}' | grep lark-alertmanager-adapter-node-modules | head -1)"

清理并重新安装依赖：

这些命令需要在 compose.yml 所在目录执行。docker compose down 只会停止并删除当前 compose project 里的容器和默认网络，不会关闭其他目录、其他 project 的 Docker Compose 容器。

# Run in the directory that contains compose.yml.
docker compose down
docker volume rm "$(docker volume ls --format '{{.Name}}' | grep lark-alertmanager-adapter-node-modules | head -1)"
docker compose up -d lark-alertmanager-adapter

.env 不要提交到 Git：

LARK_WEBHOOK_URL=https://open.larksuite.com/open-apis/bot/v2/hook/xxxxxxxxxx
# Optional. Leave empty if Lark bot signature verification is disabled.
LARK_SIGNING_SECRET=

lark-alertmanager-adapter/package.json 和 lark-alertmanager-adapter/server.js 使用上面 Node.js Adapter 小节里的内容。如果没有 package-lock.json，第一次可以先在 lark-alertmanager-adapter/ 目录里生成：

cd lark-alertmanager-adapter
npm install --package-lock-only
cd ..

启动命令在 compose.yml 所在目录执行。lark-alertmanager-adapter 是 compose service 名称，-d 表示后台运行：

docker compose up -d lark-alertmanager-adapter

查看 compose service 状态：

docker compose ps
docker compose ps lark-alertmanager-adapter

检查 health endpoint：

curl -s http://127.0.0.1:19090/healthz

预期返回：

{"ok":true}

Alertmanager 同机调用 adapter：

receivers:
  - name: lark-monitoring
    webhook_configs:
      - url: http://127.0.0.1:19090/alertmanager/lark
        send_resolved: true

说明：

container listen:
    LISTEN_ADDR=0.0.0.0 是为了让 Docker 端口映射能访问容器内服务

node_modules:
    npm ci 会写入 /app/node_modules
    所以 ./lark-alertmanager-adapter 不能挂成只读 :ro
    使用 named volume 保存 node_modules，避免写入宿主机源码目录

host bind:
    ports 使用 127.0.0.1:19090:19090，表示只暴露给宿主机本机
    EC2 外部不能直接访问这个 adapter

7. systemd#

Directory layout:

/opt/lark-alertmanager-adapter/
    package.json
    package-lock.json
    server.js
/etc/lark-alertmanager-adapter.env
/etc/systemd/system/lark-alertmanager-adapter.service

Environment file:

sudo install -d -m 0755 /opt/lark-alertmanager-adapter
sudo tee /etc/lark-alertmanager-adapter.env >/dev/null <<'EOF'
PORT=19090
LARK_WEBHOOK_URL=https://open.larksuite.com/open-apis/bot/v2/hook/526a9671-82da-4cc4-bb48-a7125e52f526
LARK_SIGNING_SECRET=
EOF
sudo chmod 0600 /etc/lark-alertmanager-adapter.env

systemd unit:

[Unit]
Description=Lark Alertmanager Adapter
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
WorkingDirectory=/opt/lark-alertmanager-adapter
EnvironmentFile=/etc/lark-alertmanager-adapter.env
ExecStart=/usr/bin/node /opt/lark-alertmanager-adapter/server.js
Restart=on-failure
RestartSec=5s
User=lark-adapter
Group=lark-adapter
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/lark-alertmanager-adapter

[Install]
WantedBy=multi-user.target

Install and start:

sudo useradd --system --home /opt/lark-alertmanager-adapter --shell /usr/sbin/nologin lark-adapter
cd /opt/lark-alertmanager-adapter
sudo npm ci --omit=dev
sudo chown -R lark-adapter:lark-adapter /opt/lark-alertmanager-adapter
sudo systemctl daemon-reload
sudo systemctl enable --now lark-alertmanager-adapter
sudo systemctl status lark-alertmanager-adapter --no-pager

8. Manual Test#

Test adapter without Alertmanager. 这会生成 Lark card，不是纯 text 消息：

curl -sS -X POST http://127.0.0.1:19090/alertmanager/lark \
  -H 'content-type: application/json' \
  -d '{
    "status": "firing",
    "groupLabels": {
      "alertname": "ServiceErrorRateHigh",
      "env": "uat",
      "severity": "P1"
    },
    "commonLabels": {
      "alertname": "ServiceErrorRateHigh",
      "severity": "P1",
      "env": "uat",
      "service": "payment",
      "component": "api"
    },
    "alerts": [
      {
        "status": "firing",
        "labels": {
          "alertname": "ServiceErrorRateHigh",
          "severity": "P1",
          "env": "uat",
          "service": "payment",
          "component": "api",
          "resource": "payment-api",
          "team": "platform",
          "region": "ap-east-1",
          "cluster": "ping-uat"
        },
        "annotations": {
          "summary": "Service error rate is high",
          "description": "payment-api has error rate > 5% for 5 minutes.",
          "dashboard_url": "https://grafana.example.com/d/payment",
          "runbook_url": "https://wiki.example.com/runbooks/payment-api"
        },
        "startsAt": "2026-06-29T00:00:00Z"
      }
    ]
  }'

Card structure:

header:
    [FIRING] TestAlert
    color from status / severity

body:
    Component
    Resource Labels
    Description
    Dashboard
    Runbook
    StartsAt

Check logs:

journalctl -u lark-alertmanager-adapter -n 100 --no-pager

Expected log prefix:

received alertmanager webhook payload:

Test from Alertmanager host:

amtool check-config /etc/alertmanager/alertmanager.yml
curl -s http://127.0.0.1:9093/api/v2/status

如果 Alertmanager 已经 reload：

curl -X POST http://127.0.0.1:9093/-/reload

9. amtool Route Test#

amtool config routes test 用来验证一组 alert labels 会匹配到哪个 Alertmanager receiver。它适合排查：

route:
    matchers 是否写对
    receiver 名称是否能匹配
    多层 routes 继承后最终会走哪个 receiver

not check:
    alert 是否已经进入 Alertmanager
    alert 是否被 silence / inhibit
    group_wait 是否还在等待
    Alertmanager 是否能连到 webhook adapter
    Lark adapter 是否能发到 Lark

Basic#

amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname=ServiceErrorRateHigh \
  severity=P1 \
  env=uat \
  service=payment \
  component=api

如果输出是：

lark-monitoring

表示这组 labels 会匹配到 lark-monitoring receiver。

ALB Example#

这个例子和 vmalert 的 ALB P1 rule labels 对齐：

amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname=ping-uat-alb-targetgroup-TargetResponseTimeHigh-P1 \
  severity=P1 \
  env=uat \
  service=alb \
  component=targetgroup

Expected:

lark-monitoring

如果 route 里依赖更多 labels，也要一起传进去：

amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname=ping-uat-alb-targetgroup-TargetResponseTimeHigh-P1 \
  severity=P1 \
  env=uat \
  service=alb \
  component=targetgroup \
  load_balancer=ping-uat-alb \
  dimension_LoadBalancer=ping-uat-alb \
  dimension_TargetGroup=sub2api-targets

Compare With Live Alert#

先从 Alertmanager API 看真实 alert labels：

curl -s http://127.0.0.1:9093/api/v2/alerts

把这条 alert 的 labels 拿出来，原样传给 amtool config routes test。不要凭记忆手写 labels，否则容易漏掉 route 实际依赖的字段。

Common Results#

output is lark-monitoring:
    route 匹配正确
    如果 Lark 没收到，继续查 silence / inhibit / group_wait / webhook URL / adapter logs

output is default receiver:
    子 route 没匹配上
    检查 matchers 需要的 label 是否存在、大小写是否一致

output is unexpected receiver:
    routes 顺序或 continue 行为可能不符合预期
    检查同级 routes 的匹配顺序

Next Checks#

route 测试通过但 Lark 没收到时，继续看当前生效配置和 alert 状态：

curl -s http://127.0.0.1:9093/api/v2/status
curl -s http://127.0.0.1:9093/api/v2/alerts

重点确认：

configYAML:
    lark-monitoring receiver 的 webhook_configs.url 是当前 adapter 地址

alert status:
    silencedBy 为空
    inhibitedBy 为空

adapter logs:
    能看到 received alertmanager webhook payload
    payload 里包含 alerts[] / startsAt / endsAt

10. Security#

Secrets:
    Lark webhook URL must be outside Git because it contains the hook token
    signing secret must also be outside Git if signature verification is enabled
    use env file / Docker secret / systemd credential / cloud secret manager

Network:
    prefer 127.0.0.1 when Alertmanager and adapter are on the same host
    if adapter listens on 0.0.0.0, restrict security group / firewall

Logs:
    do not log webhook url
    do not log signing secret
    full payload logging is useful for debugging
    disable or redact full payload logs for long-running production use when labels / annotations may contain sensitive values

Operations:
    send_resolved should be true for incident close-loop
    group_by should avoid one message per instance unless needed
    max_alerts prevents huge message bursts

11. Common Problems#

Symptom	Check
Alertmanager shows webhook 400/500	adapter logs and Lark response body
Lark direct webhook test fails	bot webhook URL, keyword check, signature check, IP whitelist
Lark says sign invalid	`LARK_SIGNING_SECRET`; it is not the UUID in `LARK_WEBHOOK_URL`; also check timestamp and server clock
No message in group	bot removed, webhook changed, network egress blocked
Too many messages	`group_by`, `group_interval`, `repeat_interval`, `max_alerts`
Adapter cannot be reached	bind address, firewall, Docker port mapping, systemd status