Export S3

Links#

1. Important Points#

DynamoDB Export to S3 适合离线分析、备份副本、数据迁移、审计和大批量查询，不适合在线 API。

good for:
    export full table snapshot
    export incremental changes when feature is enabled
    query with Athena / Glue / Spark
    pull files from S3 and run offline script

not for:
    millisecond online lookup
    replacing Query / GetItem
    near-real-time CDC

要求：

PITR:
    full export uses point-in-time recovery backup data
    enable PITR before relying on exports

S3:
    export writes manifest and data files
    choose DynamoDB JSON or Amazon Ion
    use SSE-S3 or KMS encryption

2. Permissions#

Export to S3 涉及 3 个 principal，不能只给 reader 权限：

export requester:
    人 / CI role / ops role that calls export-table-to-point-in-time
    needs DynamoDB export permission
    needs S3 write permission to target prefix
    if using KMS CMK, needs KMS encrypt/data-key permission

destination bucket:
    same account and no restrictive bucket policy:
        IAM permission on requester may be enough
    cross-account or restricted bucket:
        bucket policy must allow export requester role to write objects

export reader:
    app / analyst / batch role that reads manifest and data files later
    needs S3 read permission
    if using KMS CMK, needs KMS decrypt permission

Important:

bucket policy principal:
    use the IAM role/user that triggers export
    do not assume the bucket policy should allow dynamodb.amazonaws.com directly

revocation:
    do not remove S3 write permission while export is running
    it can leave partial export files

export requester iam#

Attach this to the role that runs aws dynamodb export-table-to-point-in-time, for example prod-dynamodb-export-role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ExportCustomerOrderStatusTable",
      "Effect": "Allow",
      "Action": [
        "dynamodb:ExportTableToPointInTime",
        "dynamodb:DescribeTable",
        "dynamodb:DescribeContinuousBackups",
        "dynamodb:UpdateContinuousBackups"
      ],
      "Resource": "arn:aws:dynamodb:ap-northeast-1:111122223333:table/customer_order_status"
    },
    {
      "Sid": "ReadExportStatus",
      "Effect": "Allow",
      "Action": [
        "dynamodb:ListExports",
        "dynamodb:DescribeExport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "WriteExportObjects",
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": "arn:aws:s3:::company-prod-dynamodb-export/customer-order-status/full/*"
    }
  ]
}

If PITR is managed by another platform role, remove dynamodb:UpdateContinuousBackups from this export requester role.

If export uses a customer managed KMS key, also add this to the export requester identity policy:

{
  "Sid": "UseExportKmsKey",
  "Effect": "Allow",
  "Action": [
    "kms:Encrypt",
    "kms:GenerateDataKey",
    "kms:DescribeKey"
  ],
  "Resource": "arn:aws:kms:ap-northeast-1:111122223333:key/abcd-1234-example"
}

bucket policy#

Use this when the bucket is cross-account or the bucket policy is restrictive. The principal is the export requester role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowDynamoDBExportRequesterWrite",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/prod-dynamodb-export-role"
      },
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": "arn:aws:s3:::company-prod-dynamodb-export/customer-order-status/full/*"
    },
    {
      "Sid": "AllowExportReaderList",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/prod-dynamodb-export-reader-role"
      },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::company-prod-dynamodb-export",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "customer-order-status/full/*"
          ]
        }
      }
    },
    {
      "Sid": "AllowExportReaderRead",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/prod-dynamodb-export-reader-role"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::company-prod-dynamodb-export/customer-order-status/full/*"
    }
  ]
}

Baseline bucket security:

aws s3api put-public-access-block \
  --bucket company-prod-dynamodb-export \
  --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true \
  --region ap-northeast-1

aws s3api put-bucket-encryption \
  --bucket company-prod-dynamodb-export \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256"
        }
      }
    ]
  }' \
  --region ap-northeast-1

kms key policy#

If export uses SSE-S3 (AES256), KMS permission is not needed.

If export uses a customer managed KMS key, the requester role needs encrypt/data-key permissions in both IAM and the KMS key policy. The KMS key must be in the same Region as the destination bucket.

{
  "Sid": "AllowDynamoDBExportRequesterUseKey",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::111122223333:role/prod-dynamodb-export-role"
  },
  "Action": [
    "kms:Encrypt",
    "kms:GenerateDataKey",
    "kms:DescribeKey"
  ],
  "Resource": "*"
}

Export reader role needs decrypt:

{
  "Sid": "AllowExportReaderDecrypt",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::111122223333:role/prod-dynamodb-export-reader-role"
  },
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "*"
}

3. Export#

Enable PITR:

aws dynamodb update-continuous-backups \
  --table-name customer_order_status \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true \
  --region ap-northeast-1

Start export:

TABLE_ARN="arn:aws:dynamodb:ap-northeast-1:111122223333:table/customer_order_status"
BUCKET="company-prod-dynamodb-export"
PREFIX="customer-order-status/full/$(date +%Y%m%d-%H%M%S)"

aws dynamodb export-table-to-point-in-time \
  --table-arn "${TABLE_ARN}" \
  --s3-bucket "${BUCKET}" \
  --s3-prefix "${PREFIX}" \
  --export-format DYNAMODB_JSON \
  --s3-sse-algorithm AES256 \
  --region ap-northeast-1

If using KMS:

aws dynamodb export-table-to-point-in-time \
  --table-arn "${TABLE_ARN}" \
  --s3-bucket "${BUCKET}" \
  --s3-prefix "${PREFIX}" \
  --export-format DYNAMODB_JSON \
  --s3-sse-algorithm KMS \
  --s3-sse-kms-key-id arn:aws:kms:ap-northeast-1:111122223333:key/abcd-1234-example \
  --region ap-northeast-1

Check export:

aws dynamodb list-exports \
  --table-arn "${TABLE_ARN}" \
  --region ap-northeast-1

aws s3 ls "s3://${BUCKET}/${PREFIX}/" --recursive

Expected layout:

s3://bucket/prefix/AWSDynamoDB/<export-id>/
    manifest-summary.json
    manifest-files.json
    data/<file>.json.gz

4. Pull And Query With Python#

这个脚本从 S3 读取 manifest-files.json，下载每个 data file，反序列化 DynamoDB JSON，然后在本地过滤。

pip install boto3

export AWS_REGION=ap-northeast-1
export EXPORT_BUCKET=company-prod-dynamodb-export
export EXPORT_PREFIX='customer-order-status/full/20260618-120000/AWSDynamoDB/01600000000000-example'
export CUSTOMER_ID='cus_8f3a91'
export ORDER_ID='ord_20260618_0001'

This script matches the table created in Python CRUD:

table:
    customer_order_status

key:
    customer_id
    order_id

expected exported item fields:
    customer_id = cus_8f3a91
    order_id = ord_20260618_0001
    status = PAID or SHIPPED
    total_amount = 129.90
    currency = HKD
    payment_id = pay_20260618_0001
    warehouse_id = hk-east-1
    shipping_city = Hong Kong

Reader role permissions:

used by:
    the role that runs query_export.py

needs:
    s3:GetObject for manifest and data files
    s3:ListBucket only if script / operator lists prefixes
    kms:Decrypt only when export objects use customer managed KMS key

query_export.py:

import gzip
import json
import os
from decimal import Decimal
from io import BytesIO

import boto3
from boto3.dynamodb.types import TypeDeserializer


REGION = os.getenv("AWS_REGION", "ap-northeast-1")
BUCKET = os.environ["EXPORT_BUCKET"]
EXPORT_PREFIX = os.environ["EXPORT_PREFIX"].rstrip("/")
CUSTOMER_ID = os.getenv("CUSTOMER_ID", "cus_8f3a91")
ORDER_ID = os.getenv("ORDER_ID", "ord_20260618_0001")

s3 = boto3.client("s3", region_name=REGION)
deserializer = TypeDeserializer()


def to_plain(value):
    if isinstance(value, Decimal):
        return int(value) if value % 1 == 0 else float(value)
    if isinstance(value, dict):
        return {k: to_plain(v) for k, v in value.items()}
    if isinstance(value, list):
        return [to_plain(v) for v in value]
    return value


def read_manifest_files():
    obj = s3.get_object(Bucket=BUCKET, Key=f"{EXPORT_PREFIX}/manifest-files.json")
    body = obj["Body"].read().decode("utf-8")
    for line in body.splitlines():
        if line.strip():
            yield json.loads(line)


def read_export_items():
    for entry in read_manifest_files():
        data_key = entry["dataFileS3Key"]
        obj = s3.get_object(Bucket=BUCKET, Key=data_key)
        raw = obj["Body"].read()
        with gzip.GzipFile(fileobj=BytesIO(raw)) as gz:
            for line in gz:
                record = json.loads(line)
                item = record.get("Item", record)
                yield to_plain({k: deserializer.deserialize(v) for k, v in item.items()})


def find_orders_by_customer(customer_id: str):
    for item in read_export_items():
        if item.get("customer_id") == customer_id:
            yield item


def find_order(customer_id: str, order_id: str):
    for item in find_orders_by_customer(customer_id):
        if item.get("order_id") == order_id:
            return item
    return None


def find_orders_by_status(status: str):
    for item in read_export_items():
        if item.get("status") == status:
            yield item


if __name__ == "__main__":
    order = find_order(CUSTOMER_ID, ORDER_ID)
    print("one order:")
    print(json.dumps(order, ensure_ascii=False, indent=2))

    print("customer orders:")
    for item in find_orders_by_customer(CUSTOMER_ID):
        print(json.dumps(item, ensure_ascii=False, indent=2))

    print("shipped orders:")
    for item in find_orders_by_status("SHIPPED"):
        print(json.dumps(item, ensure_ascii=False, indent=2))

Run:

python query_export.py

5. Production Notes#

large export:
    do not load all rows into memory
    stream gzip files line by line
    push query to Athena / Glue / Spark when data is large

security:
    export bucket blocks public access
    bucket policy only allows approved roles
    use KMS when data is sensitive
    set lifecycle expiration for temporary exports

cost:
    export cost
    S3 storage cost
    Athena / Glue / Spark scan cost

Minimal IAM for export reader:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::company-prod-dynamodb-export",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "customer-order-status/full/*"
          ]
        }
      }
    },
    {
      "Sid": "ReadExportObjects",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::company-prod-dynamodb-export/customer-order-status/full/*"
    }
  ]
}

If export objects use a customer managed KMS key, add to the reader role:

{
  "Effect": "Allow",
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "arn:aws:kms:ap-northeast-1:111122223333:key/abcd-1234-example"
}