ECS Logs

Links#

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/firelens-taskdef.html
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html
https://docs.fluentbit.io/manual/pipeline/outputs/http
https://docs.victoriametrics.com/victorialogs/data-ingestion/fluentbit/
https://docs.victoriametrics.com/victorialogs/data-ingestion/
https://docs.victoriametrics.com/victorialogs/querying/

1. Important Points#

ECS 日志进 VictoriaLogs，优先推荐 FireLens / Fluent Bit 直连。

recommended:
    app stdout/stderr -> FireLens / Fluent Bit -> VictoriaLogs /insert/jsonline

transition:
    awslogs -> CloudWatch Logs -> subscription -> VictoriaLogs

当前假设：

app logs:
    普通文本日志
    不改业务代码
    app 继续只写 stdout/stderr

VictoriaLogs:
    使用 /insert/jsonline
    使用 _msg_field=log
    使用稳定低基数字段作为 _stream_fields

2. Plain Text Logs#

Format=json_lines 说的是 Fluent Bit 发给 VictoriaLogs 的 HTTP body 格式，不要求 app 自己打印 JSON。

app stdout:
    order submitted request_id=req-123

FireLens / Fluent Bit record:
    {
      "date": "2026-06-10T08:15:30.000000000Z",
      "log": "order submitted request_id=req-123",
      "ecs_cluster": "ecs-uat-ap-east-1",
      "ecs_task_arn": "arn:aws:ecs:ap-east-1:123456789012:task/...",
      "ecs_task_definition": "order-api:42"
    }

普通文本日志要用：

_msg_field=log
Format=json_lines

不要用：

_msg_field=log.message

log.message 只适合已经被解析成嵌套 JSON 的日志。

3. Recommended Task Definition#

这个示例适合普通文本日志，并且不修改业务代码。service、env、image_version 由 log_router 注入。

{
  "family": "order-api",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "512",
  "memory": "1024",
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecs-task-execution-role",
  "containerDefinitions": [
    {
      "name": "log_router",
      "image": "123456789012.dkr.ecr.ap-east-1.amazonaws.com/aws-for-fluent-bit-custom:2026-06-10",
      "essential": true,
      "environment": [
        { "name": "SERVICE_NAME", "value": "order-api" },
        { "name": "ENV_NAME", "value": "uat" },
        { "name": "IMAGE_VERSION", "value": "2026-06-02" }
      ],
      "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
          "enable-ecs-log-metadata": "true",
          "config-file-type": "file",
          "config-file-value": "/fluent-bit/custom/extra.conf"
        }
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/firelens/log-router",
          "awslogs-region": "ap-east-1",
          "awslogs-stream-prefix": "firelens",
          "awslogs-create-group": "true",
          "mode": "non-blocking",
          "max-buffer-size": "25m"
        }
      }
    },
    {
      "name": "app",
      "image": "123456789012.dkr.ecr.ap-east-1.amazonaws.com/order-api:2026-06-02",
      "essential": true,
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "http",
          "Host": "192.168.0.1",
          "Port": "9428",
          "URI": "/insert/jsonline?_stream_fields=service,env,ecs_cluster,ecs_task_definition,image_version&_msg_field=log&_time_field=date",
          "Format": "json_lines",
          "json_date_format": "iso8601",
          "json_date_key": "date",
          "tls": "On",
          "tls.verify": "On"
        },
        "secretOptions": []
      }
    }
  ]
}

关键点：

log_router:
    运行 Fluent Bit
    加载 /fluent-bit/custom/extra.conf
    给每条日志补 service/env/image_version
    自己的 stdout/stderr 使用 awslogs 进入 CloudWatch Logs

app:
    不需要改代码
    只需要 logDriver=awsfirelens
    stdout/stderr 会交给 log_router

4. Custom Fluent Bit Config#

/fluent-bit/custom/extra.conf 是 log_router 容器内的文件路径，不是 app 容器里的路径，也不是宿主机路径。

提供这个文件有几种方式：

Method	Works On	Recommendation
custom `aws-for-fluent-bit` image	Fargate / EC2	simplest and recommended
mounted volume path	Fargate / EC2	possible, but more moving parts
`config-file-type=s3`	EC2 direct; Fargate needs init container pattern	useful for central config, more operational complexity

推荐先用自定义镜像。原因是配置跟随镜像版本发布，task 启动时文件一定在 log_router 容器内。

extra.conf:

[FILTER]
    Name record_modifier
    Match *
    Record service ${SERVICE_NAME}
    Record environment ${ENV_NAME}
    Record platform ecs
    Record source cloudwatch   # 如果值里有空格，才建议加引号，例如：Record service_name "order api"
    Record version ${IMAGE_VERSION}

[FILTER]
    Name modify
    Match *
    Copy ecs_cluster cluster
    Copy ecs_task_definition task_definition

这样最终 record 里会同时有原始字段和短字段：

ecs_cluster
cluster
ecs_task_definition
task_definition

如果想只保留短字段，可以用 Rename，但不建议一开始就删原始 metadata。保留原始字段更方便排查 FireLens 行为。

自定义 log_router 镜像：

FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:stable

COPY extra.conf /fluent-bit/custom/extra.conf

本地文件：

aws-for-fluent-bit-custom/
├── Dockerfile
└── extra.conf

build image：

cd aws-for-fluent-bit-custom

docker build --platform linux/amd64 -t aws-for-fluent-bit-custom:2026-06-10 .

push to ECR：

AWS_ACCOUNT_ID="123456789012"
AWS_REGION="ap-east-1"
ECR_REPO="aws-for-fluent-bit-custom"
IMAGE_TAG="2026-06-10"

aws ecr create-repository --repository-name "$ECR_REPO" --region "$AWS_REGION"
aws ecr get-login-password --region "$AWS_REGION" | docker login --username AWS --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com"
docker tag "aws-for-fluent-bit-custom:$IMAGE_TAG" "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO:$IMAGE_TAG"
docker push "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO:$IMAGE_TAG"

然后 task definition 里使用容器内路径：

"firelensConfiguration": {
  "type": "fluentbit",
  "options": {
    "enable-ecs-log-metadata": "true",
    "config-file-type": "file",
    "config-file-value": "/fluent-bit/custom/extra.conf"
  }
}

如果用 volume，config-file-value 仍然写容器内挂载后的路径：

"mountPoints": [
  {
    "sourceVolume": "fluent-bit-config",
    "containerPath": "/fluent-bit/custom",
    "readOnly": true
  }
]

"config-file-value": "/fluent-bit/custom/extra.conf"

如果用 S3：

"firelensConfiguration": {
  "type": "fluentbit",
  "options": {
    "enable-ecs-log-metadata": "true",
    "config-file-type": "s3",
    "config-file-value": "arn:aws:s3:::company-ecs-config/fluent-bit/order-api-extra.conf"
  }
}

注意：

Fargate:
    FireLens custom config 直接只支持 config-file-type=file
    如果要从 S3 来，需要 aws-for-fluent-bit init container / multi-config pattern

EC2 launch type:
    可以直接用 config-file-type=s3

IAM:
    S3 config 需要 task execution role 有 s3:GetObject 权限

如果不想依赖 Fluent Bit 环境变量展开，也可以由 CI/CD 发布时直接生成固定值配置：

[FILTER]
    Name record_modifier
    Match *
    Record service order-api
    Record env uat
    Record image_version 2026-06-02

5. Stream Fields#

VictoriaLogs 的 _stream_fields 是从最终 JSON record 里取字段值。字段不会因为写在 URI 里就自动出现。

6. ECS Metadata#

当 enable-ecs-log-metadata=true 时，AWS FireLens 默认加入的 ECS metadata 字段是：

Field	Example	Stream Field
`ecs_cluster`	`ecs-uat-ap-east-1`	yes
`ecs_task_arn`	`arn:aws:ecs:ap-east-1:123456789012:task/...`	no
`ecs_task_definition`	`order-api:42`	yes
`ec2_instance_id`	`i-0123456789abcdef0`	EC2 launch type only

默认 metadata 不包含：

container name
container image
image tag
service name
environment name
ECS service name
launch type

所以 service、env、image_version 需要自己补。不能改业务代码时，就在 log_router 的 Fluent Bit filter 里补。

7. Image Version#

ECS / FireLens 不会自动把 task definition 里的 app image tag 写进日志 record。

推荐做法：

CI/CD release knows:
    app image = 123456789012.dkr.ecr.ap-east-1.amazonaws.com/order-api:2026-06-02

CI/CD injects:
    IMAGE_VERSION=2026-06-02

Fluent Bit adds:
    image_version=2026-06-02

VictoriaLogs stream:
    image_version="2026-06-02"

不要在 app 里硬编码版本。版本应该来自发布流程，和 task definition 里的 image tag 保持一致。

8. How FireLens Containers Work Together#

task definition 里有两个 container：

log_router:
    FireLens sidecar
    运行 aws-for-fluent-bit
    接收 app container 的 stdout/stderr
    发送到 VictoriaLogs

app:
    业务容器
    使用 logDriver=awsfirelens

两者的关联不是靠 dependsOn 或显式写 Host: log_router。ECS agent 看到：

{
  "firelensConfiguration": { "type": "fluentbit" }
}

和：

{
  "logConfiguration": {
    "logDriver": "awsfirelens"
  }
}

就会在同一个 task 内自动建立日志通道。

完整链路：

app process
    -> stdout/stderr
    -> awsfirelens log driver
    -> log_router / Fluent Bit
    -> HTTP json_lines
    -> VictoriaLogs /insert/jsonline

如果一个 task 里有多个业务 container，只要它们都配置 logDriver=awsfirelens，日志都会交给同一个 log_router。

9. Log Router Startup Troubleshooting#

FireLens 最容易踩的坑是：app logs 走 awsfirelens，但 log_router 自己也没有独立日志出口。这样 Fluent Bit 配置错、TLS 错、secret 错、启动失败时，会看不到失败原因。

推荐固定模式：

app container:
    logDriver=awsfirelens
    stdout/stderr -> log_router -> VictoriaLogs

log_router container:
    logDriver=awslogs
    stdout/stderr -> CloudWatch Logs

原因：

log_router is the logging pipeline:
    if it fails, app logs may not reach VictoriaLogs

log_router must have independent logs:
    use awslogs
    inspect Fluent Bit config error / output error / TLS error / auth error

最小配置：

{
  "name": "log_router",
  "essential": true,
  "firelensConfiguration": {
    "type": "fluentbit",
    "options": {
      "enable-ecs-log-metadata": "true",
      "config-file-type": "file",
      "config-file-value": "/fluent-bit/custom/extra.conf"
    }
  },
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-group": "/ecs/firelens/log-router",
      "awslogs-region": "ap-east-1",
      "awslogs-stream-prefix": "firelens",
      "awslogs-create-group": "true",
      "mode": "non-blocking",
      "max-buffer-size": "25m"
    }
  }
}

排查命令：

aws ecs describe-tasks \
  --cluster <cluster> \
  --tasks <task-arn> \
  --query 'tasks[0].containers[*].{name:name,lastStatus:lastStatus,exitCode:exitCode,reason:reason}'

aws logs tail /ecs/firelens/log-router \
  --follow \
  --region ap-east-1

aws ecs describe-tasks \
  --cluster <cluster> \
  --tasks <task-arn> \
  --query 'tasks[0].stoppedReason'

常见原因：

Symptom	Check
`log_router` exits immediately	Fluent Bit config syntax / file path
no log stream in CloudWatch	task execution role logs permissions / log group region
app cannot start because log router stopped	`log_router` is essential and failed first
output returns 401/403	VictoriaLogs / vmauth Basic Auth secret
output TLS error	CA, hostname, `tls.verify`, endpoint certificate
output connection timeout	security group, route, NACL, DNS
OOMKilled	increase `memoryReservation` / task memory
config file not found	custom image did not include `/fluent-bit/custom/extra.conf`

Production baseline:

1. log_router uses awslogs for its own logs
2. app containers use awsfirelens
3. log_router is essential=true
4. CloudWatch log group for log_router is retained long enough for incident review
5. task execution role can create/write log streams, or log group/stream is pre-created
6. VictoriaLogs endpoint auth/TLS failures are visible in log_router CloudWatch logs

10. Verify#

health#

curl -s https://uat-log.imdev.work/health

insert one plain text log record#

echo '{"date":"2026-06-10T08:15:30Z","service":"order-api","env":"uat","ecs_cluster":"ecs-uat-ap-east-1","ecs_task_definition":"order-api:42","image_version":"2026-06-02","log":"hello from ecs"}' \
  | curl -X POST \
      -H 'content-type: application/stream+json' \
      --data-binary @- \
      'https://uat-log.imdev.work/insert/jsonline?_stream_fields=service,env,ecs_cluster,ecs_task_definition,image_version&_time_field=date&_msg_field=log'

Expected stream:

_stream {ecs_cluster="ecs-uat-ap-east-1",ecs_task_definition="order-api:42",env="uat",image_version="2026-06-02",service="order-api"}

query#

curl -G 'https://uat-log.imdev.work/select/logsql/query' \
  --data-urlencode 'query=service:order-api env:uat image_version:2026-06-02'

11. CloudWatch Bridge#

如果你现在已经全量在用 awslogs，可以先不动 ECS task definition，而是走 CloudWatch Logs bridge。

flow:
    ECS app logs -> CloudWatch Logs -> subscription filter -> Lambda / processor -> VictoriaLogs

good for:
    migration
    compliance path that needs CloudWatch copy
    teams that cannot touch task definition immediately

tradeoff:
    extra CloudWatch cost
    extra latency
    more moving parts than FireLens

12. Operational Checklist#

1. app logs write to stdout/stderr
2. app uses logDriver=awsfirelens
3. log_router has enable-ecs-log-metadata=true
4. log_router uses awslogs for its own startup/runtime logs
5. service/env/image_version are added by Fluent Bit filter or CI/CD-generated config
6. _msg_field=log for plain text logs
7. _stream_fields only use low-cardinality fields
8. request_id/trace_id/user_id/order_id are not stream fields
9. VictoriaLogs ingest endpoint is protected by auth/TLS
10. query and ingest paths are validated separately