AWS S3

1. Important Points#

Amazon S3 是 object storage，用 bucket 承载 namespace，用 object key 定位对象。它适合保存文件、日志、备份、静态资源、数据湖对象、Terraform state；不适合当作低延迟文件系统、强事务数据库、POSIX shared volume。

S3 的核心对象是 object，不是 file。Object key 可以写成 logs/2026/07/app.log，但 / 只是 key 里的字符，folder 是 console 和工具展示出来的视图。

Bucket name 是全局 namespace 的一部分，创建后不能改名；object key 是 bucket 内的完整名字，prefix 设计会影响 IAM boundary、lifecycle、list/debug、数据湖分区和清理策略。

S3 的 access control 比很多 AWS service 更特殊：它同时支持 identity-based policy 和 resource-based policy。Bucket policy / access point policy 可以直接写 Principal，所以跨账号共享、public access、CloudFront OAC、VPC endpoint 限制通常都在 bucket policy 里表达。

S3 权限不是只看 IAM role。一次请求可能同时被 IAM policy、bucket policy、access point policy、ACL、Block Public Access、VPC endpoint policy、SCP/RCP、KMS key policy 影响；explicit deny 优先，SSE-KMS 对象还需要 KMS 权限。

现代 S3 应默认使用 Object Ownership = Bucket owner enforced，让 ACL disabled，权限主要回到 IAM policy + bucket/access point policy。ACL 是 legacy 机制，除非兼容旧系统，不要把 ACL 当主权限模型。

S3 默认不是 public storage。生产 bucket 应默认 Block Public Access、Bucket owner enforced、SSE、deny insecure transport。真正需要 public read 时，也要让它成为显式设计，而不是某个 ACL 或 bucket policy 漏出来的结果。

Versioning 解决误删、误覆盖、程序 bug 写坏对象后的恢复问题；Object Lock 是 WORM retention，用于审计、合规、证据保全，不是 Terraform state lock。

Lifecycle 是 S3 成本控制的一部分。开启 versioning 后如果没有 noncurrent version 清理策略，长期成本会被历史版本和未完成 multipart upload 放大。

Bucket name must be boring, unique, and non-sensitive. Object key should be generated by backend, not trusted from user input. Enable versioning for state/log/backup/critical config buckets, pair versioning with lifecycle policy, and use CloudTrail data events for sensitive buckets.

2. S3 Naming Convention#

S3 naming 分两层：bucket name 负责全局唯一和 ownership signal，object key 负责业务路径、权限边界、生命周期和查询习惯。

bucket rules:
    3-63 chars
    lowercase letters / numbers / hyphen / period
    begin and end with letter or number
    globally unique within AWS partition
    cannot be renamed
    do not include sensitive data
    avoid dots unless static website hosting really needs them

bucket pattern:
    <org>-<app>-<env>-<purpose>-<region>
    <org>-<app>-<env>-<purpose>-<account-id>-<region>

examples:
    acme-order-prod-uploads-ap-east-1
    acme-platform-prod-terraform-state-333333333333-ap-east-1

object key rules:
    key is case-sensitive
    / is delimiter convention, not real directory
    use stable prefixes for IAM, lifecycle, analytics, and cleanup
    use UUID/ULID/hash suffix for user uploads
    avoid raw user filename as the full key
    avoid standalone . or .. path segments

key pattern:
    <domain>/<entity>/<partition>/<object-name>

examples:
    uploads/user-123/2026/07/02/9f3c2a1e.png
    reports/daily/dt=2026-07-02/orders.csv
    events/source=checkout/dt=2026-07-02/hour=13/part-0001.jsonl
    terraform/prod/network/terraform.tfstate

Browser upload 的 key 应由 backend 生成：

good:
    uploads/<user-id>/<uuid>.<ext>

bad:
    frontend sends arbitrary full S3 key
    original filename is trusted as storage path

3. S3 Versioning#

S3 Versioning 在同一个 bucket 里保留同一个 object key 的多个版本。它主要解决 accidental overwrite、accidental delete、application bug 覆盖对象后的恢复问题。

适合:
    Terraform state bucket
    audit / compliance logs
    backup artifacts
    release artifacts
    user uploaded file recovery
    critical configuration files

不适合单独解决:
    ransomware full recovery without backup isolation
    regulatory WORM retention
    lifecycle cost control
    cross-region disaster recovery

states:
    Unversioned:
        default state for new bucket

    Enabled:
        new objects get version IDs
        overwrite creates a new version
        delete current object creates a delete marker

    Suspended:
        bucket does not return to unversioned
        old versions remain
        new objects use null version ID

Enable versioning:

aws s3api put-bucket-versioning \
  --bucket acme-app-prod-data-ap-east-1 \
  --versioning-configuration Status=Enabled

Check status:

aws s3api get-bucket-versioning \
  --bucket acme-app-prod-data-ap-east-1

List versions:

aws s3api list-object-versions \
  --bucket acme-app-prod-data-ap-east-1 \
  --prefix configs/app.yaml

Restore a previous version:

aws s3api copy-object \
  --bucket acme-app-prod-data-ap-east-1 \
  --copy-source "acme-app-prod-data-ap-east-1/configs/app.yaml?versionId=3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY" \
  --key configs/app.yaml

Remove a delete marker:

aws s3api delete-object \
  --bucket acme-app-prod-data-ap-east-1 \
  --key configs/app.yaml \
  --version-id "delete-marker-version-id"

Lifecycle for versioned bucket:

{
  "Rules": [
    {
      "ID": "expire-noncurrent-versions",
      "Status": "Enabled",
      "Filter": {
        "Prefix": ""
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 90
      },
      "AbortIncompleteMultipartUpload": {
        "DaysAfterInitiation": 7
      }
    }
  ]
}

aws s3api put-bucket-lifecycle-configuration \
  --bucket acme-app-prod-data-ap-east-1 \
  --lifecycle-configuration file://lifecycle.json

For audit / legal / finance logs, validate retention requirements before expiring noncurrent versions.

4. S3 Object Lock#

S3 Object Lock 是 WORM protection：write once, read many。它保护 object versions 在 retention window 内不被删除或覆盖。

Object Lock is for:
    regulatory retention
    audit log immutability
    security evidence
    compliance archives

Object Lock is not for:
    Terraform S3 backend state locking
    normal application concurrency control
    short-lived lock files
    cache objects

Mode	Meaning	Use
`GOVERNANCE`	privileged user can bypass with explicit permission/header	internal protection, first rollout
`COMPLIANCE`	retention cannot be shortened or bypassed	regulatory WORM only

retention period:
    fixed until RetainUntilDate
    can be default bucket retention or object-level retention

legal hold:
    no expiration date
    independent of retention period
    must be explicitly removed

Terraform state:
    use_lockfile creates a normal .tflock object
    do not Object-Lock .tflock objects

Create a new Object Lock bucket:

aws s3api create-bucket \
  --bucket acme-audit-log-333333333333-ap-east-1 \
  --region ap-east-1 \
  --create-bucket-configuration LocationConstraint=ap-east-1 \
  --object-lock-enabled-for-bucket

Apply default retention:

aws s3api put-object-lock-configuration \
  --bucket acme-audit-log-333333333333-ap-east-1 \
  --object-lock-configuration '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":30}}}'

Set retention on one object version:

aws s3api put-object-retention \
  --bucket acme-audit-log-333333333333-ap-east-1 \
  --key cloudtrail/AWSLogs/333333333333/example.json.gz \
  --version-id "3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY" \
  --retention '{"Mode":"GOVERNANCE","RetainUntilDate":"2026-12-31T00:00:00Z"}'

Legal hold:

aws s3api put-object-legal-hold \
  --bucket acme-audit-log-333333333333-ap-east-1 \
  --key cloudtrail/AWSLogs/333333333333/example.json.gz \
  --version-id "3HL4kqtJlcpXroDTDmJ+rmSpXd3dIbrHY" \
  --legal-hold Status=ON

extra permissions:
    s3:GetObjectRetention / s3:PutObjectRetention
    s3:GetObjectLegalHold / s3:PutObjectLegalHold
    s3:GetBucketObjectLockConfiguration
    s3:PutBucketObjectLockConfiguration

dangerous:
    s3:BypassGovernanceRetention
        only for tightly controlled break-glass role

5. Bucket Policy#

Bucket policy 是 bucket 上的 resource-based permission。它适合做 bucket-level guardrail、cross-account access、explicit deny。

S3 policy 特殊点:
    bucket policy / access point policy 是 resource-based policy，必须写 Principal
    identity policy attached to IAM role/user answers "who can call what"
    bucket policy answers "who can access this bucket/object under what condition"
    bucket-level action uses bucket ARN: arn:aws:s3:::bucket
    object-level action uses object ARN: arn:aws:s3:::bucket/prefix/*
    cross-account access often needs both sides to be correct: caller identity permissions + bucket resource policy
    Block Public Access can override public bucket policy / ACL
    Object Ownership = Bucket owner enforced disables ACL authorization
    SSE-KMS object access also needs KMS key policy / grant / IAM permission
    VPC endpoint policy can still deny a request even when IAM and bucket policy allow it

Recommended baseline:

normal private bucket:
    keep Block Public Access enabled
    keep Object Ownership = Bucket owner enforced
    use bucket policy for explicit deny guardrails
    use IAM identity policy for application role permissions
    avoid ACL-based sharing

cross-account sharing:
    prefer bucket policy or access point policy
    name exact role/account principals
    use prefix-scoped resources where possible
    test with the real caller role, not only bucket owner credentials

deny insecure transport#

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::amzn-s3-demo-bucket",
        "arn:aws:s3:::amzn-s3-demo-bucket/*"
      ],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    }
  ]
}

restrict access to one VPC endpoint#

Only use this when all legitimate access goes through that S3 VPC endpoint. It can block console, CI, and cross-account automation.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyAccessOutsideSpecificVpce",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::amzn-s3-demo-bucket",
        "arn:aws:s3:::amzn-s3-demo-bucket/*"
      ],
      "Condition": {
        "StringNotEquals": {
          "aws:SourceVpce": "vpce-0abcdef1234567890"
        }
      }
    }
  ]
}

allow specific account principals#

Use aws:PrincipalAccount as a guardrail. It does not replace identity IAM permissions.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyPrincipalOutsideAccount",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::amzn-s3-demo-bucket",
        "arn:aws:s3:::amzn-s3-demo-bucket/*"
      ],
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalAccount": "333333333333"
        }
      }
    }
  ]
}

6. CORS And Presigned Upload#

Browser direct upload 通常是 backend 生成 short-lived presigned PUT URL，browser 直接上传到 S3。CORS 只决定 browser 是否允许发请求；IAM / bucket policy / signature 才决定 S3 是否接受请求。

good fit:
    app server should not proxy large file body
    backend controls bucket, key prefix, content type, expiry
    frontend uploads directly to S3

not enough:
    user can choose arbitrary object key
    object must be virus-scanned before readable
    size/content validation only exists in frontend

Temporary download URL:

aws s3 presign s3://acme-order-prod-uploads-ap-east-1/reports/2026-06-summary.csv \
  --expires-in 300 \
  --region ap-east-1

Production browser upload usually needs SDK-generated presigned PUT URL, not aws s3 presign.

Bucket CORS:

[
  {
    "AllowedOrigins": [
      "https://app.example.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET",
      "HEAD"
    ],
    "AllowedHeaders": [
      "content-type",
      "x-amz-meta-*",
      "x-amz-server-side-encryption"
    ],
    "ExposeHeaders": [
      "ETag"
    ],
    "MaxAgeSeconds": 3000
  }
]

aws s3api put-bucket-cors \
  --bucket acme-order-prod-uploads-ap-east-1 \
  --cors-configuration file://cors.json

CORS rules:
    AllowedOrigins should be exact production origins
    AllowedMethods needs PUT for upload
    AllowedHeaders must include headers the browser sends
    if presigned URL signs Content-Type, browser must send the same Content-Type
    upload works with curl but not browser usually means CORS, not IAM

IAM for presign service:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPresignUploadPrefix",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::acme-order-prod-uploads-ap-east-1/uploads/*"
    }
  ]
}

Backend presign code:

npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

import { randomUUID } from "node:crypto";
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const s3 = new S3Client({ region: "ap-east-1" });
const bucket = "acme-order-prod-uploads-ap-east-1";

export async function createUploadUrl({ userId, fileName, contentType }) {
  if (!contentType.startsWith("image/")) {
    throw new Error("only image upload is allowed");
  }

  const ext = fileName.split(".").pop()?.toLowerCase() || "bin";
  const key = `uploads/${userId}/${randomUUID()}.${ext}`;

  const command = new PutObjectCommand({
    Bucket: bucket,
    Key: key,
    ContentType: contentType,
    ServerSideEncryption: "AES256",
    Metadata: {
      "uploaded-by": userId
    }
  });

  const uploadUrl = await getSignedUrl(s3, command, {
    expiresIn: 300
  });

  return { bucket, key, uploadUrl, expiresIn: 300 };
}

Browser PUT must send headers that match signed headers:

await fetch(uploadUrl, {
  method: "PUT",
  headers: {
    "content-type": file.type,
    "x-amz-server-side-encryption": "AES256"
  },
  body: file
});

Common mistakes:

Symptom	Check
browser preflight fails	bucket CORS `AllowedOrigins`, `AllowedMethods`, `AllowedHeaders`
`SignatureDoesNotMatch`	browser headers do not match signed headers, especially `content-type`
`AccessDenied`	presign role lacks `s3:PutObject` or bucket policy denies request
upload works with curl but not browser	CORS issue
object uploaded to wrong prefix	backend trusted user-provided key
public read unexpectedly works	bucket policy / ACL / public access block

7. VPC Endpoint#

Private subnet access to S3 normally uses S3 Gateway VPC Endpoint. It avoids NAT Gateway data processing cost and keeps traffic on AWS network paths.

Gateway endpoint:
    route table target for S3 prefix list
    no hourly charge
    common choice for EC2/ECS/EKS private subnet access to S3

Interface endpoint:
    PrivateLink ENI
    useful for some specific S3 APIs / on-prem private connectivity cases
    has hourly and data processing cost

Check endpoint:

aws ec2 describe-vpc-endpoints \
  --filters Name=service-name,Values=com.amazonaws.ap-east-1.s3

Check prefix list:

aws ec2 describe-prefix-lists \
  --filters Name=prefix-list-name,Values=com.amazonaws.ap-east-1.s3

8. Operational Checklist#

normal private application bucket:
    block public access enabled
    Object Ownership = Bucket owner enforced
    versioning enabled if recovery matters
    lifecycle for noncurrent versions and incomplete multipart uploads
    deny insecure transport
    SSE enabled
    least-privilege IAM policy
    CloudTrail data events if sensitive

Terraform state bucket:
    versioning enabled
    S3 Object Lock not required for use_lockfile
    do not Object-Lock .tflock objects
    state access limited to Terraform operators/roles

audit/compliance log bucket:
    versioning enabled
    consider Object Lock
    validate Governance vs Compliance with compliance owner
    tightly control s3:BypassGovernanceRetention
    lifecycle only after retention review

browser upload bucket:
    backend generates key and presigned URL
    CORS origin/header/method list is narrow
    upload prefix is isolated by IAM
    public read is disabled unless explicitly required