Grafana ECS Service Dashboards

本文缩写（英文全称与中文含义）

缩写	英文全称／名称	中文含义
ALB	Application Load Balancer	应用负载均衡器
AWS	Amazon Web Services	亚马逊云服务
CPU	Central Processing Unit	中央处理器
ECS	Elastic Container Service	AWS 容器编排服务
GC	Garbage Collection	垃圾回收
HTTP	Hypertext Transfer Protocol	超文本传输协议
OOM	Out Of Memory	内存不足
YACE	Yet Another CloudWatch Exporter	把 CloudWatch 指标转换为 Prometheus 指标的采集器

Links#

1. Dashboard Rule#

Folder: Compute
Dashboard: ECS Service - Overview

Rows:
    Service Health:
        desired 是否满足，CPU / memory 是否接近瓶颈，container 是否频繁 restart

    Scaling Signals:
        running / desired / pending 的关系，CPU / memory 长期趋势，deployment / capacity gap 症状

为什么不保留四层:

ECS service 原生 CloudWatch metrics 不多。
如果没有接入 ALB metrics 和应用 Prometheus metrics，拆成 Overview / Drilldown / Capacity / Debug 很容易重复。

所以 ECS service 这类 dashboard 更适合压成 1 到 2 个 row：
    第一行回答服务当前是否健康
    第二行回答是否需要扩缩容，或者 deployment / capacity provider 是否有问题

2. Overview#

Panel	Metrics	Why
Service running tasks	`RunningTaskCount`, `DesiredTaskCount`	running 小于 desired 是服务容量不足或部署异常
CPU usage	`CPUUtilization`	ECS service 的 compute saturation
Memory usage	`MemoryUtilization`	ECS service 最常见 scaling / OOM signal
Request rate	app `http_requests_total` or ALB `RequestCount`	没有流量上下文时 CPU / memory 很难判断
Error rate	app `5xx` or ALB `HTTPCode_Target_5XX_Count`	用户侧失败信号
Latency	app latency p95/p99 or ALB `TargetResponseTime`	用户侧慢请求信号

request / error / latency 来源:

ECS namespace:
    不提供完整 request / error / latency。
    ECS metrics 主要回答 service / task 资源和运行状态。

ALB, if service is HTTP behind ALB:
    RequestCount
    HTTPCode_Target_5XX_Count
    HTTPCode_ELB_5XX_Count
    TargetResponseTime

App metrics, recommended:
    http_requests_total
    http_request_duration_seconds
    error counter by route / status / exception

3. Dashboard Panels#

Row	Panel	Metrics	Why
Service Health	Running vs desired tasks	`RunningTaskCount`, `DesiredTaskCount`	第一优先级先看服务有没有满足 desired
Service Health	CPU utilization	`CPUUtilization`	判断 CPU 是否接近 target tracking 或 service saturation
Service Health	Memory utilization	`MemoryUtilization`	最常见的 ECS autoscaling / OOM 风险信号
Service Health	Pending task count	`PendingTaskCount`	发现部署卡住、容量不足、任务无法及时拉起
Scaling Signals	Task count by service	`RunningTaskCount`, `DesiredTaskCount`, `PendingTaskCount`	区分扩容中、部署卡住、capacity 不足
Scaling Signals	CPU / memory trend	`CPUUtilization`, `MemoryUtilization`	判断现有 scaling target 是否合适
Scaling Signals	Scaling / deployment symptoms	`DesiredTaskCount - RunningTaskCount`, `PendingTaskCount`	快速识别 desired-running gap 和 pending 压力

4. YACE Metrics#

如果 ECS dashboard 使用 aws_ecs_* metrics，需要 YACE 采集 AWS/ECS。应用请求量、错误率、延迟仍然建议由应用 /metrics 或 ALB metrics 提供。

YACE config:

apiVersion: v1alpha1
sts-region: ap-northeast-1
discovery:
  jobs:
    - type: AWS/ECS
      regions:
        - ap-northeast-1
      customTags:
        - key: environment
          value: uat
      period: 300
      length: 600
      nilToZero: true
      metrics:
        - name: CPUUtilization
          statistics: [Average]
        - name: MemoryUtilization
          statistics: [Average]
    - type: ECS/ContainerInsights
      regions:
        - ap-northeast-1
      customTags:
        - key: environment
          value: uat
      period: 300
      length: 600
      nilToZero: true
      metrics:
        - name: RunningTaskCount
          statistics: [Average]
        - name: DesiredTaskCount
          statistics: [Average]
        - name: PendingTaskCount
          statistics: [Average]

Container Insights note:

下面这些指标不属于标准 AWS/ECS 这段 YACE 配置:
    RunningTaskCount
    DesiredTaskCount
    PendingTaskCount

它们来自:
    ECS/ContainerInsights

如果 dashboard 要展示 task count / pending，
需要同时采:
    AWS/ECS
    ECS/ContainerInsights

如果 ECS service behind ALB，并且 dashboard 要显示 request / error / latency，YACE 还需要采集 AWS/ApplicationELB。ALB 的详细 YACE 配置见 ALB YACE Dashboard。

确认 metric name:

{__name__=~"aws_ecs_.*"}

5. Do Not Put In Overview#

不要放:
    每个 container 的所有 cgroup counters
    runtime GC / heap / goroutine / event loop 全量明细
    每个 deployment event 的长文本

原因:
    ECS overview 应该优先回答 desired 是否满足、资源是否饱和、用户体验是否变差