AWS ALB Monitoring

验证：未记录
适用版本
未记录
验证范围
尚未登记验证记录。

本文缩写（英文全称与中文含义）
缩写 英文全称／名称 中文含义
ALB Application Load Balancer 应用负载均衡器
API Application Programming Interface 应用程序编程接口
AWS Amazon Web Services 亚马逊云服务
AZ Availability Zone 可用区
CPU Central Processing Unit 中央处理器
DNS Domain Name System 域名系统
EC2 Elastic Compute Cloud AWS 云服务器服务
ECS Elastic Container Service AWS 容器编排服务
HTTP Hypertext Transfer Protocol 超文本传输协议
IP Internet Protocol 网际协议；IP 地址是网络寻址标识
LCU Load Balancer Capacity Unit 负载均衡器容量单位
P0 Priority 0 优先级编号；具体严重程度按团队规范定义
P1 Priority 1 优先级编号；具体严重程度按团队规范定义
P2 Priority 2 优先级编号；具体严重程度按团队规范定义
P3 Priority 3 优先级编号；具体严重程度按团队规范定义
S3 Simple Storage Service AWS 对象存储服务
SG Security Group 安全组
SLO Service Level Objective 服务等级目标
SNS Simple Notification Service AWS 消息通知服务
SSM AWS Systems Manager AWS 系统管理服务；SSM 是沿用的简称
TLS Transport Layer Security 传输层安全协议
WAF Web Application Firewall Web 应用防火墙

1. Important Points#

ALB 告警要区分三类问题:

availability:
    no healthy target
    high target 5xx rate
    ALB 5xx
    rejected connections

performance:
    high target response time
    high target connection errors
    unusual request drop

security / traffic:
    WAF blocked request spike
    fixed-response 4xx spike
    HTTPCode_ELB_4XX_Count spike
    suspicious source IP from access logs

告警原则:

use rate for errors:
    target 5xx count alone is noisy
    target 5xx / request count is better

use absolute count for infrastructure failure:
    HealthyHostCount == 0
    RejectedConnectionCount > 0

use p95/p99 for latency:
    average hides tail latency

low traffic service:
    combine rate and minimum request count
    otherwise 1 failed request can produce 100% error rate

2. Must-have Alarms#

ALB 自身的 5xx 与 target 返回的 5xx 分开看；错误率还要结合请求量，不能用一次低流量失败推断持续故障。

Severity Alarm Meaning Why Monitor Metric / Expression Threshold Duration
P0 No healthy target target group 没有健康 target ALB 无法把流量转发到可用后端 HealthyHostCount Minimum < 1 2m
P1 Target 5xx rate high target 返回 5xx 的比例过高 发现应用、依赖或发布导致的用户请求失败 100 * Target5xx / RequestCount >= 5% 5m
P1 ALB 5xx exists ALB 自身生成 5xx 发现无健康 target、连接失败或协议问题 HTTPCode_ELB_5XX_Count Sum >= 10 5m
P1 Target p95 latency high target 响应 p95 延迟高 发现后端性能退化和用户体验风险 TargetResponseTime p95 >= 1s 10m
P2 Rejected connections ALB 拒绝连接 发现连接/LCU 压力或突发流量超限 RejectedConnectionCount Sum > 0 5m
P2 Target connection errors ALB 连接 target 失败 发现安全组、端口、target 崩溃或网络路径问题 TargetConnectionErrorCount Sum > 0 5m
P2 Unhealthy target exists 至少存在不健康 target 提前发现部分实例故障，避免容量继续下降 UnHealthyHostCount Maximum > 0 5m
P3 Traffic abnormal drop 请求量相对历史基线异常下降 发现 DNS、路由、证书、发布或上游流量异常 anomaly detection / baseline env-specific 15m

Recommended dimensions:

LoadBalancer:
    app/prod-public-alb/abc123

TargetGroup:
    targetgroup/prod-order-api/def456

AvailabilityZone:
    use only when debugging AZ-specific issue
    most service alarms should not include AZ dimension

3. SNS Topic#

Create one topic per alert route:

aws sns create-topic --name prod-critical-alerts --region ap-east-1

aws sns subscribe --topic-arn arn:aws:sns:ap-east-1:111122223333:prod-critical-alerts --protocol email --notification-endpoint sre@example.com --region ap-east-1

For ChatOps, route SNS to:

common options:
    AWS Chatbot to Slack
    Lambda webhook relay
    PagerDuty / Opsgenie integration
    EventBridge rule to incident workflow

4. No Healthy Target Alarm#

This is the highest value ALB alarm. If no healthy target exists, ALB cannot send traffic to the service.

aws cloudwatch put-metric-alarm --alarm-name "P0-ALB-prod-order-api-no-healthy-target" --alarm-description "ALB target group has no healthy target" --namespace AWS/ApplicationELB --metric-name HealthyHostCount --dimensions Name=TargetGroup,Value=targetgroup/prod-order-api/def456 Name=LoadBalancer,Value=app/prod-public-alb/abc123 --statistic Minimum --period 60 --evaluation-periods 2 --datapoints-to-alarm 2 --threshold 1 --comparison-operator LessThanThreshold --treat-missing-data breaching --alarm-actions arn:aws:sns:ap-east-1:111122223333:prod-critical-alerts --ok-actions arn:aws:sns:ap-east-1:111122223333:prod-critical-alerts --region ap-east-1

Verify:

aws cloudwatch describe-alarms --alarm-names "P0-ALB-prod-order-api-no-healthy-target" --region ap-east-1

aws elbv2 describe-target-health --target-group-arn arn:aws:elasticloadbalancing:ap-east-1:111122223333:targetgroup/prod-order-api/def456 --region ap-east-1

5. Target 5xx Rate Alarm#

Target 5xx means backend returned 5xx. This usually points to application, dependency, deploy, or capacity issue.

aws cloudwatch put-metric-alarm --alarm-name "P1-ALB-prod-order-api-target-5xx-rate" --alarm-description "Target 5xx rate is >= 5% for 5 minutes" --evaluation-periods 1 --datapoints-to-alarm 1 --threshold 5 --comparison-operator GreaterThanOrEqualToThreshold --treat-missing-data notBreaching --metrics '[
    {
      "Id": "req",
      "MetricStat": {
        "Metric": {
          "Namespace": "AWS/ApplicationELB",
          "MetricName": "RequestCount",
          "Dimensions": [
            {"Name": "LoadBalancer", "Value": "app/prod-public-alb/abc123"},
            {"Name": "TargetGroup", "Value": "targetgroup/prod-order-api/def456"}
          ]
        },
        "Period": 300,
        "Stat": "Sum"
      },
      "ReturnData": false
    },
    {
      "Id": "e5xx",
      "MetricStat": {
        "Metric": {
          "Namespace": "AWS/ApplicationELB",
          "MetricName": "HTTPCode_Target_5XX_Count",
          "Dimensions": [
            {"Name": "LoadBalancer", "Value": "app/prod-public-alb/abc123"},
            {"Name": "TargetGroup", "Value": "targetgroup/prod-order-api/def456"}
          ]
        },
        "Period": 300,
        "Stat": "Sum"
      },
      "ReturnData": false
    },
    {
      "Id": "target_5xx_rate",
      "Expression": "IF(req>=100,100*e5xx/req,0)",
      "Label": "Target 5xx percent",
      "ReturnData": true
    }
  ]' --alarm-actions arn:aws:sns:ap-east-1:111122223333:prod-critical-alerts --ok-actions arn:aws:sns:ap-east-1:111122223333:prod-critical-alerts --region ap-east-1

Why req>=100:

low traffic service:
    1 request and 1 failure = 100%
    this is often not P1

high enough sample:
    only alert when enough requests happened in the period
    tune 100 based on service traffic

6. ALB 5xx Alarm#

ALB 5xx means the load balancer itself could not successfully handle or forward the request. Common causes are no healthy target, target connection problem, TLS/HTTP protocol problem, or internal ALB error.

aws cloudwatch put-metric-alarm --alarm-name "P1-ALB-prod-public-alb-elb-5xx" --alarm-description "ALB generated 5xx responses" --namespace AWS/ApplicationELB --metric-name HTTPCode_ELB_5XX_Count --dimensions Name=LoadBalancer,Value=app/prod-public-alb/abc123 --statistic Sum --period 300 --evaluation-periods 1 --datapoints-to-alarm 1 --threshold 10 --comparison-operator GreaterThanOrEqualToThreshold --treat-missing-data notBreaching --alarm-actions arn:aws:sns:ap-east-1:111122223333:prod-critical-alerts --region ap-east-1

7. Latency Alarm#

Use TargetResponseTime p95 for backend response latency. This is not full browser latency; it measures the time from when ALB sends request to target until target starts sending response.

aws cloudwatch put-metric-alarm --alarm-name "P1-ALB-prod-order-api-target-p95-latency" --alarm-description "Target response time p95 is high" --namespace AWS/ApplicationELB --metric-name TargetResponseTime --dimensions Name=TargetGroup,Value=targetgroup/prod-order-api/def456 Name=LoadBalancer,Value=app/prod-public-alb/abc123 --extended-statistic p95 --period 60 --evaluation-periods 10 --datapoints-to-alarm 8 --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold --treat-missing-data notBreaching --alarm-actions arn:aws:sns:ap-east-1:111122223333:prod-critical-alerts --region ap-east-1

Tune by service:

public API:
    p95 >= 1s for 10m often P1/P2

internal API:
    p95 threshold should follow SLO

worker callback / webhook:
    threshold may be higher, but should still be explicit

8. WAF Blocked Request Alarm#

If WAF is attached to ALB, alert on blocked request spike. WAF metrics are in AWS/WAFV2.

aws cloudwatch put-metric-alarm --alarm-name "P2-WAF-prod-public-alb-blocked-spike" --alarm-description "WAF blocked requests increased" --namespace AWS/WAFV2 --metric-name BlockedRequests --dimensions Name=WebACL,Value=prod-public-alb-web-acl Name=Rule,Value=ALL Name=Region,Value=ap-east-1 --statistic Sum --period 300 --evaluation-periods 2 --datapoints-to-alarm 2 --threshold 1000 --comparison-operator GreaterThanOrEqualToThreshold --treat-missing-data notBreaching --alarm-actions arn:aws:sns:ap-east-1:111122223333:prod-security-alerts --region ap-east-1

Interpretation:

blocked spike can mean:
    real attack
    crawler / scanner
    false positive after rule change
    broken client behavior

next check:
    WAF sampled requests
    WAF logs
    ALB access logs by source IP / path / user-agent

9. EventBridge For Alarm State#

Route CloudWatch alarm state changes to automation:

{
  "source": ["aws.cloudwatch"],
  "detail-type": ["CloudWatch Alarm State Change"],
  "detail": {
    "state": {
      "value": ["ALARM"]
    },
    "alarmName": [{
      "prefix": "P"
    }]
  }
}

Create rule:

aws events put-rule --name cloudwatch-alarm-to-incident --event-pattern file://cloudwatch-alarm-event-pattern.json --region ap-east-1

Common targets:

target:
    SNS topic
    Lambda incident router
    Step Functions runbook
    SSM Automation

10. Access Log Investigation#

Access logs are delivered to S3. Use Athena for incident queries.

Typical questions:

what changed:
    5xx by target
    4xx by path
    top source IP
    top user-agent
    WAF blocked IP and path correlation
    latency by target group

Example Athena query after creating the ALB access log table:

SELECT
  request_url,
  target_status_code,
  elb_status_code,
  COUNT(*) AS requests
FROM alb_access_logs
WHERE day = '2026/06/03'
  AND elb = 'app/prod-public-alb/abc123'
  AND (elb_status_code LIKE '5%' OR target_status_code LIKE '5%')
GROUP BY request_url, target_status_code, elb_status_code
ORDER BY requests DESC
LIMIT 50;

Top source IP:

SELECT
  client_ip,
  COUNT(*) AS requests,
  SUM(CASE WHEN elb_status_code LIKE '4%' THEN 1 ELSE 0 END) AS elb_4xx,
  SUM(CASE WHEN elb_status_code LIKE '5%' THEN 1 ELSE 0 END) AS elb_5xx
FROM alb_access_logs
WHERE day = '2026/06/03'
GROUP BY client_ip
ORDER BY requests DESC
LIMIT 50;

11. Dashboard#

Dashboard widgets:

traffic:
    RequestCount
    ActiveConnectionCount
    NewConnectionCount

errors:
    HTTPCode_ELB_4XX_Count
    HTTPCode_ELB_5XX_Count
    HTTPCode_Target_4XX_Count
    HTTPCode_Target_5XX_Count
    target 5xx rate metric math

latency:
    TargetResponseTime p50 / p95 / p99

health:
    HealthyHostCount
    UnHealthyHostCount
    TargetConnectionErrorCount

security:
    WAF AllowedRequests
    WAF BlockedRequests
    WAF CountedRequests

12. Incident Checklist#

no healthy target:
    describe-target-health
    check ECS/EC2 deployment
    check app readiness endpoint
    check target SG allows ALB SG
    check container/instance logs

target 5xx:
    compare deployment time
    inspect app logs by trace/request id
    check dependency errors
    check target CPU/memory/database

ALB 5xx:
    check target connection errors
    check healthy host count
    check TLS/protocol mismatch
    inspect ALB access logs

latency:
    check target p95/p99
    check app dependency latency
    check target saturation
    check database and downstream services

WAF blocked spike:
    inspect WAF logs and sampled requests
    compare recent WAF rule change
    check source IP / country / path
    decide block, allowlist, or rule tuning
ALB Auth Forward

搜索文档

输入关键词，搜索所有文档。

↑ ↓ 选择 · Enter 打开 · Esc 关闭