Java Auto Scaling

Links#

1. Scaling Strategy#

Java on ECS 的主扩缩容指标通常选 request rate 或 CPU。JVM memory 更适合做 guardrail，因为 heap 预留、GC 行为和 actual demand 不一定线性对应。

Workload	Primary Scaling Metric	Target
HTTP API behind ALB	`ALBRequestCountPerTarget`	load test safe RPS per task 的 60-70%
CPU-bound service	`ECSServiceAverageCPUUtilization`	55-65%
worker / consumer	custom `backlog_per_task`	backlog 在业务 SLO 内清空
memory-bound batch	custom business metric + memory guard	不把 JVM heap usage 单独作为默认主指标

Java guardrails:

p95 latency:
    <= service SLO

CPU:
    average <= 70% at target load
    sustained > 80% means scale-out or optimize

GC:
    p95 pause stays inside latency budget
    GC time ratio should not keep rising under stable RPS

heap:
    old gen after full GC should stay stable
    container memory headroom >= 20-25%

threads:
    servlet / reactive worker pool not saturated
    DB connection pool wait time near zero at target load

Decision:

Spring Boot / HTTP service:
    use ALBRequestCountPerTarget when traffic drives load
    use CPU when per-request cost varies heavily

async worker:
    use queue backlog per task
    alert on oldest message age

avoid:
    memory-only scaling for normal JVM services
    high maxCapacity without checking DB pool and downstream quotas

2. ECS Policy#

aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --scalable-dimension ecs:service:DesiredCount \
  --resource-id service/prod-apps/order-api \
  --min-capacity 2 \
  --max-capacity 30 \
  --region ap-east-1

CPU target tracking:

aws application-autoscaling put-scaling-policy \
  --service-namespace ecs \
  --scalable-dimension ecs:service:DesiredCount \
  --resource-id service/prod-apps/order-api \
  --policy-name order-api-java-cpu \
  --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration '{
    "TargetValue": 60,
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ECSServiceAverageCPUUtilization"
    },
    "ScaleOutCooldown": 90,
    "ScaleInCooldown": 300
  }' \
  --region ap-east-1

ALB request count target tracking:

aws application-autoscaling put-scaling-policy \
  --service-namespace ecs \
  --scalable-dimension ecs:service:DesiredCount \
  --resource-id service/prod-apps/order-api \
  --policy-name order-api-java-rps-per-task \
  --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration '{
    "TargetValue": 180,
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ALBRequestCountPerTarget",
      "ResourceLabel": "app/prod-public-alb/abc123/targetgroup/prod-order-api/def456"
    },
    "ScaleOutCooldown": 90,
    "ScaleInCooldown": 300
  }' \
  --region ap-east-1

3. Load Test For Threshold#

Goal: 找到 JVM warmup 后的 steady-state capacity。

test setup:
    desiredCount = 1
    same task CPU/memory as production
    same JAVA_TOOL_OPTIONS as production
    enable GC logs or export JVM metrics
    run warmup traffic before measuring

increase load:
    warmup 10-20m
    50 RPS -> 100 -> 150 -> 200 -> 250
    each measured step >= 15m

stop point:
    p95 latency > SLO
    5xx >= 1%
    CPU avg >= 75%
    GC pause breaks latency budget
    connection pool wait appears
    old gen after GC keeps growing

Example result:

task size:
    1 vCPU / 2GB

observed:
    250 RPS: p95 220ms, CPU 64%, GC p95 pause 35ms
    320 RPS: p95 530ms, CPU 82%, DB pool wait appears

safe RPS per task:
    250

autoscaling target:
    250 * 0.70 = 175
    choose ALBRequestCountPerTarget target = 175-180

Useful commands:

hey -z 15m -c 150 https://api.example.com/orders

aws logs tail /ecs/order-api \
  --since 30m \
  --filter-pattern '"gc" "pause"' \
  --region ap-east-1

4. Readiness Checklist#

scaling:
    minCapacity >= 2
    maxCapacity checked against DB pool, Redis, downstream API quota
    request-count and CPU policies do not fight each other

jvm:
    heap limit is container-aware
    MaxRAMPercentage / InitialRAMPercentage are explicit
    graceful shutdown waits for in-flight requests

observability:
    ALB RPS / latency / 5xx
    ECS CPU / memory
    JVM heap / non-heap
    GC pause / GC time ratio
    thread pool and DB pool metrics