AWS EC2 Monitoring

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html
https://docs.aws.amazon.com/ebs/latest/userguide/using_cloudwatch_ebs.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-common-scenarios.html

AWS/EC2 default metrics:
    status check
    CPU
    network
    EBS IO at instance level

CloudWatch Agent metrics:
    memory
    filesystem
    swap
    process / agent status if configured

AWS/EBS metrics:
    volume queue
    volume throughput / IOPS
    burst balance for burstable volumes

Period:
    每个 datapoint 的统计窗口，不是告警持续时间。

Evaluation:
    M/N datapoints means datapoints_to_alarm / evaluation_periods.
    例如 10/10 datapoints with 60s period means 10 consecutive 60s datapoints.

Alarm After:
    roughly Period * EvaluationPeriods.
    例如 10 * 60s = about 10m.

status check:
    P0/P1 用 2/2 datapoints，快速确认但避免单点抖动。

resource saturation:
    CPU / memory / disk / EBS queue 默认连续 datapoints，避免一次采样误报。

5m Sum metric:
    cost / network / discrete count 用 300s period，减少噪音。

env-specific:
    network throughput and agent heartbeat must be based on instance type, workload baseline, and collection method.

{
  "metrics": {
    "namespace": "CWAgent",
    "append_dimensions": {
      "InstanceId": "${aws:InstanceId}",
      "InstanceType": "${aws:InstanceType}",
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}"
    },
    "metrics_collected": {
      "mem": {
        "measurement": [
          "mem_used_percent"
        ],
        "metrics_collection_interval": 60
      },
      "disk": {
        "measurement": [
          "used_percent"
        ],
        "metrics_collection_interval": 60,
        "resources": [
          "/"
        ],
        "ignore_file_system_types": [
          "sysfs",
          "devtmpfs",
          "tmpfs",
          "overlay"
        ]
      },
      "swap": {
        "measurement": [
          "swap_used_percent"
        ],
        "metrics_collection_interval": 60
      }
    }
  }
}

sudo yum install -y amazon-cloudwatch-agent

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config \
  -m ec2 \
  -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json \
  -s

aws cloudwatch list-metrics \
  --namespace CWAgent \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --region ap-east-1

aws cloudwatch put-metric-alarm \
  --alarm-name "P0-EC2-i-0123456789abcdef0-status-check-failed" \
  --namespace AWS/EC2 \
  --metric-name StatusCheckFailed \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --statistic Maximum \
  --period 60 \
  --evaluation-periods 2 \
  --datapoints-to-alarm 2 \
  --threshold 0 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data missing \
  --alarm-actions arn:aws:sns:ap-east-1:111122223333:prod-critical-alerts \
  --region ap-east-1

aws cloudwatch put-metric-alarm \
  --alarm-name "P1-EC2-i-0123456789abcdef0-root-disk-used-high" \
  --namespace CWAgent \
  --metric-name disk_used_percent \
  --dimensions \
      Name=InstanceId,Value=i-0123456789abcdef0 \
      Name=path,Value=/ \
      Name=fstype,Value=xfs \
  --statistic Average \
  --period 60 \
  --evaluation-periods 10 \
  --datapoints-to-alarm 10 \
  --threshold 90 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data missing \
  --alarm-actions arn:aws:sns:ap-east-1:111122223333:prod-critical-alerts \
  --region ap-east-1

aws cloudwatch put-metric-alarm \
  --alarm-name "P2-EBS-vol-0123456789abcdef0-queue-high" \
  --namespace AWS/EBS \
  --metric-name VolumeQueueLength \
  --dimensions Name=VolumeId,Value=vol-0123456789abcdef0 \
  --statistic Average \
  --period 60 \
  --evaluation-periods 10 \
  --datapoints-to-alarm 10 \
  --threshold 10 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data missing \
  --alarm-actions arn:aws:sns:ap-east-1:111122223333:prod-warning-alerts \
  --region ap-east-1

health:
    StatusCheckFailed
    StatusCheckFailed_Instance
    StatusCheckFailed_System

compute:
    CPUUtilization
    CPUCreditBalance / CPUSurplusCreditsCharged for T family

memory / disk:
    mem_used_percent
    disk_used_percent by path
    swap_used_percent

storage:
    VolumeQueueLength
    VolumeReadOps / VolumeWriteOps
    VolumeReadBytes / VolumeWriteBytes
    BurstBalance for burstable volumes

network:
    NetworkIn / NetworkOut
    NetworkPacketsIn / NetworkPacketsOut

agent:
    CloudWatch Agent metric freshness

process monitoring:
    critical process alive
    systemd unit failed
    CloudWatch Agent procstat or SSM inventory

log monitoring:
    kernel OOM
    disk filesystem errors
    application fatal logs
    auth failure / sudo / ssh logs when required

patch / security:
    SSM patch compliance
    Inspector findings
    EDR / antivirus health if used

Auto Scaling:
    ASG InService instance count
    launch failure
    lifecycle hook stuck
    capacity rebalance / spot interruption if used

backup / recovery:
    EBS snapshot success
    AWS Backup job status
    restore test result

network detail:
    security group / NACL change events
    VPC Flow Logs for rejected traffic if needed

enough for:
    EC2 health
    CPU / memory / disk
    EBS queue / burst risk
    basic network trend

not enough for:
    application availability
    OS security compliance
    backup assurance
    ASG fleet health
    detailed network troubleshooting