VictoriaLogs Alerting

验证：未记录

适用版本: 未记录
验证范围: 尚未登记验证记录。

本文缩写（英文全称与中文含义）

缩写	英文全称／名称	中文含义
ALB	Application Load Balancer	应用负载均衡器
SLO	Service Level Objective	服务等级目标
VMUI	VictoriaMetrics User Interface	VictoriaMetrics 查询界面

1. Important Points#

VictoriaLogs 可以直接做日志告警，不必先把每条日志转成 metrics。标准链路是：

ALB / application logs
    -> VictoriaLogs
        -> vmalert runs LogsQL periodically
            -> Alertmanager
                -> Feishu / Slack / PagerDuty

核心点：

vmalert rule group:
    type: vlogs

expr:
    LogsQL query

alert route:
    vmalert -> Alertmanager -> receiver

适合直接用日志告警：

业务错误码
关键异常文本
特定接口连续 5xx
支付回调失败
订单创建失败
安全审计关键事件

更适合用 metrics 告警：

整体 5xx 比例
请求量
延迟
资源饱和度
SLO burn rate

2. Rule Example#

假设日志已经有结构化字段：service、route、status。

groups:
  - name: alb-business-alerts
    type: vlogs
    interval: 1m
    rules:
      - alert: Checkout5xx
        expr: |
          service:checkout route:/api/checkout status:~"5.."
          | stats count() as errors
          | filter errors >= 5
        for: 2m
        labels:
          severity: critical
          service: checkout
          route: /api/checkout
        annotations:
          summary: "checkout 在近窗口持续出现 5xx"
          description: "checkout /api/checkout 5xx errors >= 5"

这个 rule 的意思：

每 1 分钟跑一次 LogsQL
筛选 checkout /api/checkout 的 5xx 日志
统计命中数量为 errors
errors >= 5 时进入 pending
持续 2 分钟后 firing
告警发送给 Alertmanager

3. vmalert Run#

vmalert datasource 指向 VictoriaLogs 查询地址，notifier 指向 Alertmanager。

vmalert -datasource.url=http://victorialogs:9428 -notifier.url=http://alertmanager:9093 -rule=/etc/vmalert/vlogs/*.yml

如果 VictoriaLogs 前面有 vmauth，就让 vmalert 访问 vmauth：

vmalert -datasource.url=http://vmauth:8427 -notifier.url=http://alertmanager:9093 -rule=/etc/vmalert/vlogs/*.yml

4. Rule Design#

不要按所有 route 自动告警，否则很快会告警风暴。只筛选少数关键业务 route，并设置“次数 + 持续时间”。

bad:
    all route 5xx > 0 alert immediately

better:
    critical route only
    errors >= 10 in 5m
    for: 2m

实用组合：

CloudWatch / application metrics:
    发现服务整体异常

VictoriaLogs rules:
    关键业务失败直接告警

Alert payload:
    service
    normalized route
    status
    error count
    Grafana Explore / VMUI query link

5. Log Structure Requirement#

前提是日志要尽量结构化。不要只存 ALB 原始整行文本；入库时最好解析出稳定字段。

推荐字段：

service:
    application or business service name

route:
    normalized route
    remove id and query string
    example: /api/order/:id instead of /api/order/123?debug=true

elb_status_code:
    ALB response status

target_status_code:
    target response status

target_group:
    ALB target group

error_code:
    business error code if available

trace_id:
    used for investigation, but avoid using it as stream field

6. More Examples#

payment callback failed#

groups:
  - name: payment-log-alerts
    type: vlogs
    interval: 1m
    rules:
      - alert: PaymentCallbackFailed
        expr: |
          service:payment event:payment_callback_failed
          | stats count() as errors
          | filter errors >= 3
        for: 2m
        labels:
          severity: critical
          service: payment
        annotations:
          summary: "payment callback failed"

critical exception text#

groups:
  - name: app-exception-alerts
    type: vlogs
    interval: 1m
    rules:
      - alert: OrderCreateException
        expr: |
          service:order-api "OrderCreateFailed"
          | stats count() as errors
          | filter errors >= 5
        for: 3m
        labels:
          severity: warning
          service: order-api
        annotations:
          summary: "order create exception increased"

7. Common Pitfalls#

using logs for all SLO alerts:
    use metrics for ratio, latency, request volume, burn rate

alerting every route:
    creates alert storm
    whitelist critical business routes

raw unparsed ALB logs:
    rules become slow and fragile
    parse route / status / target_group at ingestion time

high-cardinality stream fields:
    do not use request_id / trace_id / user_id as stream fields

missing for duration:
    single bad log line should not page people
    use count threshold + for duration

8. References#

VictoriaLogs alerting: https://docs.victoriametrics.com/victorialogs/vmalert/
VictoriaLogs querying: https://docs.victoriametrics.com/victorialogs/querying/
vmalert: https://docs.victoriametrics.com/victoriametrics/vmalert/