YACE Dashboard


1. YACE Config#

apiVersion: v1alpha1
sts-region: ap-northeast-1
discovery:
  jobs:
    - type: AWS/ApplicationELB
      regions: [ap-northeast-1]
      customTags:
        - key: environment
          value: uat
      period: 300
      length: 600
      nilToZero: true
      metrics:
        - name: RequestCount
          statistics: [Sum]
        - name: HealthyHostCount
          statistics: [Minimum]
        - name: UnHealthyHostCount
          statistics: [Maximum]
        - name: HTTPCode_Target_2XX_Count
          statistics: [Sum]
        - name: HTTPCode_Target_4XX_Count
          statistics: [Sum]
        - name: HTTPCode_Target_5XX_Count
          statistics: [Sum]
        - name: HTTPCode_ELB_5XX_Count
          statistics: [Sum]
        - name: TargetResponseTime
          statistics: [Average, p95, p99]
        - name: TargetConnectionErrorCount
          statistics: [Sum]
        - name: RejectedConnectionCount
          statistics: [Sum]
        - name: ActiveConnectionCount
          statistics: [Sum]
        - name: NewConnectionCount
          statistics: [Sum]
        - name: ProcessedBytes
          statistics: [Sum]

2. Dashboard Rows#

Row Panels
Overview request count, healthy hosts, target 5xx, ALB 5xx, target latency by target group
Drilldown status code split, target group health, connection errors
Alert Signals target 5xx rate in 5m, target 5xx count in 5m, request count in 5m, target response time p95 by target group
Capacity request trend, active/new connections, processed bytes
Debug rejected connections

Trend note:

带 Trend 的 panel 不是原始单点值。

当前 ALB dashboard 里:
    Request Trend
        使用最近 1 小时 rolling sum

period and length#

period: 300
length: 600

这里的含义不是：

YACE 每 300 秒才拉一次数据

而是：

period:
    CloudWatch 每个 datapoint 的统计窗口
    period: 300 表示每个 datapoint 是 5 分钟窗口

length:
    每次查询 CloudWatch 时，向前回看的时间范围
    length: 600 表示每次查询最近 10 分钟的数据

对 ALB 这类 Sum 指标，period: 300 表示：

这一条 datapoint 是“过去 300 秒内的总数”
不是“YACE 每 300 秒只取一条以后就不管了”

例如：

RequestCount + Statistic: Sum + period: 300
    表示 5 分钟窗口内的请求总数

HTTPCode_Target_5XX_Count + Statistic: Sum + period: 300
    表示 5 分钟窗口内 target 返回的 5xx 总数

为什么 length 通常比 period 更长：

CloudWatch datapoint 可能有轻微延迟
length 比 period 长，可以让 YACE 查询最近多个窗口
降低短时漏点概率

常见组合：

period: 300
length: 600
    5 分钟 datapoint
    每次查询最近 10 分钟

period: 60
length: 300
    1 分钟 datapoint
    每次查询最近 5 分钟

metrics needed for the 2 alert-signal panels#

如果你的目标不是做完整 dashboard，而只是支持你现在关心的这 2 类告警信号：

1. Target 5xx rate high
2. TargetResponseTimeHigh

那么最小采集集只需要围绕这两个信号：

Target 5xx rate high:
    RequestCount Sum
    HTTPCode_Target_5XX_Count Sum

TargetResponseTimeHigh:
    TargetResponseTime p95

对应的最小 YACE config 可以写成：

apiVersion: v1alpha1
sts-region: ap-northeast-1
discovery:
  jobs:
    - type: AWS/ApplicationELB
      regions: [ap-northeast-1]
      customTags:
        - key: environment
          value: uat
      period: 300
      length: 600
      nilToZero: true
      metrics:
        - name: RequestCount
          statistics: [Sum]
        - name: HTTPCode_Target_5XX_Count
          statistics: [Sum]
        - name: TargetResponseTime
          statistics: [p95]

这份最小配置只够支持：

Target 5xx Rate (5m) By Target Group
Target 5xx Count (5m) By Target Group
Request Count (5m) By Target Group
TargetResponseTime p95 By Target Group

如果你还要保留当前这份 ALB dashboard 里的完整视图，而不只是这 2 类告警信号：

还需要继续采:
    HealthyHostCount
    UnHealthyHostCount
    HTTPCode_Target_2XX_Count
    HTTPCode_Target_4XX_Count
    HTTPCode_ELB_5XX_Count
    TargetResponseTime Average / p99
    TargetConnectionErrorCount
    RejectedConnectionCount
    ActiveConnectionCount
    NewConnectionCount
    ProcessedBytes

3. Verify#

{__name__=~"aws_applicationelb_.*"}

告警辅助面板里，TargetGroup 相关总量查询统一过滤：

dimension_AvailabilityZone=""

这样只保留 LoadBalancer + TargetGroup 这一层聚合结果，不把 AvailabilityZone + LoadBalancer + TargetGroup 的明细重复加进去。

当前 Target Latency By Target Group 也使用同样的过滤思路：

avg by (dimension_LoadBalancer, dimension_TargetGroup) (
  aws_applicationelb_target_response_time_p95{
    environment=~"$environment",
    dimension_LoadBalancer=~"$dimension_LoadBalancer",
    dimension_TargetGroup=~".+",
    dimension_AvailabilityZone=""
  }
)

这样展示的是 target group 级延迟，而不是把所有 target group 汇总成单个 ALB 级延迟。

这里额外加：

dimension_TargetGroup=~".+"

是为了只保留真正带 target group 维度的时序，避免把只有 ALB 级别的时序混进来。
=~ / !~ 这类 PromQL matcher 的通用说明放在： content/Grafana/DashboardPromQLReference.md

alert signals and rule alignment#

如果你的 rule 是：

Target 5xx rate > 5%
and request count in 5m >= 100
and target 5xx count in 5m >= 10

那么 dashboard 里新增的 Alert Signals row 就是用来解释这条 rule 为什么触发的。

Target 5xx Rate (5m) By Target Group:

100 *
sum by (dimension_LoadBalancer, dimension_TargetGroup) (
  sum_over_time(
    aws_applicationelb_httpcode_target_5_xx_count_sum{
      environment=~"$environment",
      dimension_LoadBalancer=~"$dimension_LoadBalancer",
      dimension_AvailabilityZone=""
    }[5m]
  )
)
/
clamp_min(
  sum by (dimension_LoadBalancer, dimension_TargetGroup) (
    sum_over_time(
      aws_applicationelb_request_count_sum{
        environment=~"$environment",
        dimension_LoadBalancer=~"$dimension_LoadBalancer",
        dimension_AvailabilityZone=""
      }[5m]
    )
  ),
  1
)

Target 5xx Count (5m) By Target Group:

sum by (dimension_LoadBalancer, dimension_TargetGroup) (
  sum_over_time(
    aws_applicationelb_httpcode_target_5_xx_count_sum{
      environment=~"$environment",
      dimension_LoadBalancer=~"$dimension_LoadBalancer",
      dimension_AvailabilityZone=""
    }[5m]
  )
)

Request Count (5m) By Target Group:

sum by (dimension_LoadBalancer, dimension_TargetGroup) (
  sum_over_time(
    aws_applicationelb_request_count_sum{
      environment=~"$environment",
      dimension_LoadBalancer=~"$dimension_LoadBalancer",
      dimension_AvailabilityZone=""
    }[5m]
  )
)

TargetResponseTime p95 By Target Group:

avg by (dimension_LoadBalancer, dimension_TargetGroup) (
  aws_applicationelb_target_response_time_p95{
    environment=~"$environment",
    dimension_LoadBalancer=~"$dimension_LoadBalancer",
    dimension_TargetGroup=~".+",
    dimension_AvailabilityZone=""
  }
)

这 4 个 panel 的目的不是替代告警，而是：

rule 触发后
你能立刻看到是哪一个 target group 触发
以及它的 rate / request / target 5xx 三个量分别是多少

4. Optional: Shorten LoadBalancer and TargetGroup Labels#

如果你不想在 Grafana 里看到这种完整值：

dimension_LoadBalancer="app/ping-uat-alb/d12dac00d9c29937"
dimension_TargetGroup="targetgroup/ping-ninedata/9f7333c03055bfe4"

可以在 Prometheus / vmagent scrape yace 时做 metric_relabel_configs，只保留中间那段业务名：

scrape_configs:
  - job_name: yace
    static_configs:
      - targets:
          - yace:5001

    metric_relabel_configs:
      - source_labels: [dimension_LoadBalancer]
        regex: '[^/]+/([^/]+)/[^/]+'
        target_label: dimension_LoadBalancer
        replacement: '$1'
        action: replace

      - source_labels: [dimension_TargetGroup]
        regex: '[^/]+/([^/]+)/[^/]+'
        target_label: dimension_TargetGroup
        replacement: '$1'
        action: replace

效果：

app/ping-uat-alb/d12dac00d9c29937
    -> ping-uat-alb

targetgroup/ping-ninedata/9f7333c03055bfe4
    -> ping-ninedata

这个改法只影响 label value，不会改 metric name。

适合：

想让 Grafana legend 更短
想让 dashboard 更容易读
不需要在面板里保留 AWS 生成的后缀 id

注意：

如果两个不同 ALB 或 target group 的中间名字相同
relabel 后它们会看起来一样
CloudWatch Dashboard Forward