VictoriaLogs ECS Logs

验证：未记录
适用版本
未记录
验证范围
尚未登记验证记录。

本文缩写（英文全称与中文含义）
缩写 英文全称／名称 中文含义
AWS Amazon Web Services 亚马逊云服务
CA Certificate Authority 证书颁发机构
CD Continuous Delivery / Continuous Deployment 持续交付／持续部署；是否自动上线取决于流程
CI Continuous Integration 持续集成
DNS Domain Name System 域名系统
EC2 Elastic Compute Cloud AWS 云服务器服务
ECR Elastic Container Registry AWS 容器镜像仓库
ECS Elastic Container Service AWS 容器编排服务
HTTP Hypertext Transfer Protocol 超文本传输协议
IAM Identity and Access Management 身份与访问管理
JSON JavaScript Object Notation 一种结构化数据格式
NACL Network Access Control List 网络访问控制列表
S3 Simple Storage Service AWS 对象存储服务
TLS Transport Layer Security 传输层安全协议
URI Uniform Resource Identifier 统一资源标识符 
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/firelens-taskdef.html
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html
https://docs.fluentbit.io/manual/pipeline/outputs/http
https://docs.victoriametrics.com/victorialogs/data-ingestion/fluentbit/
https://docs.victoriametrics.com/victorialogs/data-ingestion/
https://docs.victoriametrics.com/victorialogs/querying/

1. Important Points#

ECS 日志进 VictoriaLogs，优先推荐 FireLens / Fluent Bit 直连。

recommended:
    app stdout/stderr -> FireLens / Fluent Bit -> VictoriaLogs /insert/jsonline

transition:
    awslogs -> CloudWatch Logs -> subscription -> VictoriaLogs

当前假设：

app logs:
    普通文本日志
    不改业务代码
    app 继续只写 stdout/stderr

VictoriaLogs:
    使用 /insert/jsonline
    使用 _msg_field=log
    使用稳定低基数字段作为 _stream_fields

2. Plain Text Logs#

Format=json_lines 说的是 Fluent Bit 发给 VictoriaLogs 的 HTTP body 格式，不要求 app 自己打印 JSON。

app stdout:
    order submitted request_id=req-123

FireLens / Fluent Bit record:
    {
      "date": "2026-06-10T08:15:30.000000000Z",
      "log": "order submitted request_id=req-123",
      "ecs_cluster": "ecs-uat-ap-east-1",
      "ecs_task_arn": "arn:aws:ecs:ap-east-1:123456789012:task/...",
      "ecs_task_definition": "order-api:42"
    }

普通文本日志要用：

_msg_field=log
Format=json_lines

不要用：

_msg_field=log.message

log.message 只适合已经被解析成嵌套 JSON 的日志。

这个示例适合普通文本日志，并且不修改业务代码。service、env、image_version 由 log_router 注入。

{
  "family": "order-api",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "512",
  "memory": "1024",
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecs-task-execution-role",
  "containerDefinitions": [
    {
      "name": "log_router",
      "image": "123456789012.dkr.ecr.ap-east-1.amazonaws.com/aws-for-fluent-bit-custom:2026-06-10",
      "essential": true,
      "environment": [
        { "name": "SERVICE_NAME", "value": "order-api" },
        { "name": "ENV_NAME", "value": "uat" },
        { "name": "IMAGE_VERSION", "value": "2026-06-02" }
      ],
      "firelensConfiguration": {
        "type": "fluentbit",
        "options": {
          "enable-ecs-log-metadata": "true",
          "config-file-type": "file",
          "config-file-value": "/fluent-bit/custom/extra.conf"
        }
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/firelens/log-router",
          "awslogs-region": "ap-east-1",
          "awslogs-stream-prefix": "firelens",
          "awslogs-create-group": "true",
          "mode": "non-blocking",
          "max-buffer-size": "25m"
        }
      }
    },
    {
      "name": "app",
      "image": "123456789012.dkr.ecr.ap-east-1.amazonaws.com/order-api:2026-06-02",
      "essential": true,
      "logConfiguration": {
        "logDriver": "awsfirelens",
        "options": {
          "Name": "http",
          "Host": "192.168.0.1",
          "Port": "9428",
          "URI": "/insert/jsonline?_stream_fields=service,env,ecs_cluster,ecs_task_definition,image_version&_msg_field=log&_time_field=date",
          "Format": "json_lines",
          "json_date_format": "iso8601",
          "json_date_key": "date",
          "tls": "On",
          "tls.verify": "On"
        },
        "secretOptions": []
      }
    }
  ]
}

关键点：

log_router:
    运行 Fluent Bit
    加载 /fluent-bit/custom/extra.conf
    给每条日志补 service/env/image_version
    自己的 stdout/stderr 使用 awslogs 进入 CloudWatch Logs

app:
    不需要改代码
    只需要 logDriver=awsfirelens
    stdout/stderr 会交给 log_router

4. Custom Fluent Bit Config#

/fluent-bit/custom/extra.conf 是 log_router 容器内的文件路径，不是 app 容器里的路径，也不是宿主机路径。

提供这个文件有几种方式：

Method Works On Recommendation
custom aws-for-fluent-bit image Fargate / EC2 simplest and recommended
mounted volume path Fargate / EC2 possible, but more moving parts
config-file-type=s3 EC2 direct; Fargate needs init container pattern useful for central config, more operational complexity

推荐先用自定义镜像。原因是配置跟随镜像版本发布，task 启动时文件一定在 log_router 容器内。

extra.conf:

[FILTER]
    Name record_modifier
    Match *
    Record service ${SERVICE_NAME}
    Record environment ${ENV_NAME}
    Record platform ecs
    Record source cloudwatch   # 如果值里有空格，才建议加引号，例如：Record service_name "order api"
    Record version ${IMAGE_VERSION}

[FILTER]
    Name modify
    Match *
    Copy ecs_cluster cluster
    Copy ecs_task_definition task_definition

这样最终 record 里会同时有原始字段和短字段：

ecs_cluster
cluster
ecs_task_definition
task_definition

如果想只保留短字段，可以用 Rename，但不建议一开始就删原始 metadata。保留原始字段更方便排查 FireLens 行为。

自定义 log_router 镜像：

FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:stable

COPY extra.conf /fluent-bit/custom/extra.conf

本地文件：

aws-for-fluent-bit-custom/
├── Dockerfile
└── extra.conf

build image：

cd aws-for-fluent-bit-custom

docker build --platform linux/amd64 -t aws-for-fluent-bit-custom:2026-06-10 .

push to ECR：

AWS_ACCOUNT_ID="123456789012"
AWS_REGION="ap-east-1"
ECR_REPO="aws-for-fluent-bit-custom"
IMAGE_TAG="2026-06-10"

aws ecr create-repository --repository-name "$ECR_REPO" --region "$AWS_REGION"
aws ecr get-login-password --region "$AWS_REGION" | docker login --username AWS --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com"
docker tag "aws-for-fluent-bit-custom:$IMAGE_TAG" "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO:$IMAGE_TAG"
docker push "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPO:$IMAGE_TAG"

然后 task definition 里使用容器内路径：

"firelensConfiguration": {
  "type": "fluentbit",
  "options": {
    "enable-ecs-log-metadata": "true",
    "config-file-type": "file",
    "config-file-value": "/fluent-bit/custom/extra.conf"
  }
}

如果用 volume，config-file-value 仍然写容器内挂载后的路径：

"mountPoints": [
  {
    "sourceVolume": "fluent-bit-config",
    "containerPath": "/fluent-bit/custom",
    "readOnly": true
  }
]
"config-file-value": "/fluent-bit/custom/extra.conf"

如果用 S3：

"firelensConfiguration": {
  "type": "fluentbit",
  "options": {
    "enable-ecs-log-metadata": "true",
    "config-file-type": "s3",
    "config-file-value": "arn:aws:s3:::company-ecs-config/fluent-bit/order-api-extra.conf"
  }
}

注意：

Fargate:
    FireLens custom config 直接只支持 config-file-type=file
    如果要从 S3 来，需要 aws-for-fluent-bit init container / multi-config pattern

EC2 launch type:
    可以直接用 config-file-type=s3

IAM:
    S3 config 需要 task execution role 有 s3:GetObject 权限

如果不想依赖 Fluent Bit 环境变量展开，也可以由 CI/CD 发布时直接生成固定值配置：

[FILTER]
    Name record_modifier
    Match *
    Record service order-api
    Record env uat
    Record image_version 2026-06-02

5. Stream Fields#

VictoriaLogs 的 _stream_fields 是从最终 JSON record 里取字段值。字段不会因为写在 URI 里就自动出现。

推荐：

_stream_fields=service,env,ecs_cluster,ecs_task_definition,image_version

原因：

service:
    服务名，低基数

env:
    uat/prod/dev，低基数

ecs_cluster:
    FireLens 默认 ECS metadata

ecs_task_definition:
    FireLens 默认 ECS metadata，例如 order-api:42

image_version:
    发布版本，例如 2026-06-02

不要作为 stream field：

request_id
trace_id
user_id
order_id
ecs_task_arn

这些字段基数太高，会让 stream 数量膨胀。

字段名必须完全匹配：

env is not environment
ecs_cluster is not cluster
ecs_task_definition is not task_definition

如果 extra.conf 里已经把 ecs_cluster 复制成 cluster，把 ecs_task_definition 复制成 task_definition，也可以用短字段：

_stream_fields=service,env,cluster,task_definition,image_version

6. ECS Metadata#

当 enable-ecs-log-metadata=true 时，AWS FireLens 默认加入的 ECS metadata 字段是：

Field Example Stream Field
ecs_cluster ecs-uat-ap-east-1 yes
ecs_task_arn arn:aws:ecs:ap-east-1:123456789012:task/... no
ecs_task_definition order-api:42 yes
ec2_instance_id i-0123456789abcdef0 EC2 launch type only

默认 metadata 不包含：

container name
container image
image tag
service name
environment name
ECS service name
launch type

所以 service、env、image_version 需要自己补。不能改业务代码时，就在 log_router 的 Fluent Bit filter 里补。

7. Image Version#

ECS / FireLens 不会自动把 task definition 里的 app image tag 写进日志 record。

推荐做法：

CI/CD release knows:
    app image = 123456789012.dkr.ecr.ap-east-1.amazonaws.com/order-api:2026-06-02

CI/CD injects:
    IMAGE_VERSION=2026-06-02

Fluent Bit adds:
    image_version=2026-06-02

VictoriaLogs stream:
    image_version="2026-06-02"

不要在 app 里硬编码版本。版本应该来自发布流程，和 task definition 里的 image tag 保持一致。

8. How FireLens Containers Work Together#

task definition 里有两个 container：

log_router:
    FireLens sidecar
    运行 aws-for-fluent-bit
    接收 app container 的 stdout/stderr
    发送到 VictoriaLogs

app:
    业务容器
    使用 logDriver=awsfirelens

两者的关联不是靠 dependsOn 或显式写 Host: log_router。ECS agent 看到：

{
  "firelensConfiguration": { "type": "fluentbit" }
}

和：

{
  "logConfiguration": {
    "logDriver": "awsfirelens"
  }
}

就会在同一个 task 内自动建立日志通道。

完整链路：

app process
    -> stdout/stderr
    -> awsfirelens log driver
    -> log_router / Fluent Bit
    -> HTTP json_lines
    -> VictoriaLogs /insert/jsonline

如果一个 task 里有多个业务 container，只要它们都配置 logDriver=awsfirelens，日志都会交给同一个 log_router。

9. Log Router Startup Troubleshooting#

FireLens 最容易踩的坑是：app logs 走 awsfirelens，但 log_router 自己也没有独立日志出口。这样 Fluent Bit 配置错、TLS 错、secret 错、启动失败时，会看不到失败原因。

推荐固定模式：

app container:
    logDriver=awsfirelens
    stdout/stderr -> log_router -> VictoriaLogs

log_router container:
    logDriver=awslogs
    stdout/stderr -> CloudWatch Logs

原因：

log_router is the logging pipeline:
    if it fails, app logs may not reach VictoriaLogs

log_router must have independent logs:
    use awslogs
    inspect Fluent Bit config error / output error / TLS error / auth error

最小配置：

{
  "name": "log_router",
  "essential": true,
  "firelensConfiguration": {
    "type": "fluentbit",
    "options": {
      "enable-ecs-log-metadata": "true",
      "config-file-type": "file",
      "config-file-value": "/fluent-bit/custom/extra.conf"
    }
  },
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-group": "/ecs/firelens/log-router",
      "awslogs-region": "ap-east-1",
      "awslogs-stream-prefix": "firelens",
      "awslogs-create-group": "true",
      "mode": "non-blocking",
      "max-buffer-size": "25m"
    }
  }
}

排查命令：

aws ecs describe-tasks --cluster <cluster> --tasks <task-arn> --query 'tasks[0].containers[*].{name:name,lastStatus:lastStatus,exitCode:exitCode,reason:reason}'
aws logs tail /ecs/firelens/log-router --follow --region ap-east-1
aws ecs describe-tasks --cluster <cluster> --tasks <task-arn> --query 'tasks[0].stoppedReason'

常见原因：

Symptom Check
log_router exits immediately Fluent Bit config syntax / file path
no log stream in CloudWatch task execution role logs permissions / log group region
app cannot start because log router stopped log_router is essential and failed first
output returns 401/403 VictoriaLogs / vmauth Basic Auth secret
output TLS error CA, hostname, tls.verify, endpoint certificate
output connection timeout security group, route, NACL, DNS
OOMKilled increase memoryReservation / task memory
config file not found custom image did not include /fluent-bit/custom/extra.conf

Production baseline:

1. log_router uses awslogs for its own logs
2. app containers use awsfirelens
3. log_router is essential=true
4. CloudWatch log group for log_router is retained long enough for incident review
5. task execution role can create/write log streams, or log group/stream is pre-created
6. VictoriaLogs endpoint auth/TLS failures are visible in log_router CloudWatch logs

10. Verify#

health#

curl -s https://uat-log.imdev.work/health

insert one plain text log record#

echo '{"date":"2026-06-10T08:15:30Z","service":"order-api","env":"uat","ecs_cluster":"ecs-uat-ap-east-1","ecs_task_definition":"order-api:42","image_version":"2026-06-02","log":"hello from ecs"}' | curl -X POST -H 'content-type: application/stream+json' --data-binary @- 'https://uat-log.imdev.work/insert/jsonline?_stream_fields=service,env,ecs_cluster,ecs_task_definition,image_version&_time_field=date&_msg_field=log'

Expected stream:

_stream {ecs_cluster="ecs-uat-ap-east-1",ecs_task_definition="order-api:42",env="uat",image_version="2026-06-02",service="order-api"}

query#

curl -G 'https://uat-log.imdev.work/select/logsql/query' --data-urlencode 'query=service:order-api env:uat image_version:2026-06-02'

11. CloudWatch Bridge#

如果你现在已经全量在用 awslogs，可以先不动 ECS task definition，而是走 CloudWatch Logs bridge。

flow:
    ECS app logs -> CloudWatch Logs -> subscription filter -> Lambda / processor -> VictoriaLogs

good for:
    migration
    compliance path that needs CloudWatch copy
    teams that cannot touch task definition immediately

tradeoff:
    extra CloudWatch cost
    extra latency
    more moving parts than FireLens

12. Operational Checklist#

1. app logs write to stdout/stderr
2. app uses logDriver=awsfirelens
3. log_router has enable-ecs-log-metadata=true
4. log_router uses awslogs for its own startup/runtime logs
5. service/env/image_version are added by Fluent Bit filter or CI/CD-generated config
6. _msg_field=log for plain text logs
7. _stream_fields only use low-cardinality fields
8. request_id/trace_id/user_id/order_id are not stream fields
9. VictoriaLogs ingest endpoint is protected by auth/TLS
10. query and ingest paths are validated separately
Backward Setup
TLS Forward

搜索文档

输入关键词，搜索所有文档。

↑ ↓ 选择 · Enter 打开 · Esc 关闭