AWS DynamoDB Export To S3

验证：未记录

适用版本: 未记录
验证范围: 尚未登记验证记录。

本文缩写（英文全称与中文含义）

缩写	英文全称／名称	中文含义
AES256	Advanced Encryption Standard, 256-bit key	使用 256 位密钥的高级加密标准
API	Application Programming Interface	应用程序编程接口
AWS	Amazon Web Services	亚马逊云服务
CDC	Change Data Capture	变更数据捕获
CI	Continuous Integration	持续集成
CMK	Customer Master Key	AWS KMS 旧称，现称 KMS key；其他云须按产品定义理解
CRUD	Create, Read, Update, Delete	创建、读取、更新、删除
HKD	Hong Kong Dollar	港元货币代码
IAM	Identity and Access Management	身份与访问管理
JSON	JavaScript Object Notation	一种结构化数据格式
KMS	Key Management Service	密钥管理服务
PITR	Point-In-Time Recovery	时间点恢复
S3	Simple Storage Service	AWS 对象存储服务
SSE	Server-Side Encryption	服务端加密

Links#

1. Important Points#

DynamoDB Export to S3 适合离线分析、备份副本、数据迁移、审计和大批量查询，不适合在线 API。

good for:
    export full table snapshot
    export incremental changes when feature is enabled
    query with Athena / Glue / Spark
    pull files from S3 and run offline script

not for:
    millisecond online lookup
    replacing Query / GetItem
    near-real-time CDC

要求：

PITR:
    full export uses point-in-time recovery backup data
    enable PITR before relying on exports

S3:
    export writes manifest and data files
    choose DynamoDB JSON or Amazon Ion
    use SSE-S3 or KMS encryption

2. Permissions#

Export to S3 涉及 3 个 principal，不能只给 reader 权限：

export requester:
    人 / CI role / ops role that calls export-table-to-point-in-time
    needs DynamoDB export permission
    needs S3 write permission to target prefix
    if using KMS CMK, needs KMS encrypt/data-key permission

destination bucket:
    same account and no restrictive bucket policy:
        IAM permission on requester may be enough
    cross-account or restricted bucket:
        bucket policy must allow export requester role to write objects

export reader:
    app / analyst / batch role that reads manifest and data files later
    needs S3 read permission
    if using KMS CMK, needs KMS decrypt permission

Important:

bucket policy principal:
    use the IAM role/user that triggers export
    do not assume the bucket policy should allow dynamodb.amazonaws.com directly

revocation:
    do not remove S3 write permission while export is running
    it can leave partial export files

export requester iam#

Attach this to the role that runs aws dynamodb export-table-to-point-in-time, for example prod-dynamodb-export-role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ExportCustomerOrderStatusTable",
      "Effect": "Allow",
      "Action": [
        "dynamodb:ExportTableToPointInTime",
        "dynamodb:DescribeTable",
        "dynamodb:DescribeContinuousBackups",
        "dynamodb:UpdateContinuousBackups"
      ],
      "Resource": "arn:aws:dynamodb:ap-northeast-1:111122223333:table/customer_order_status"
    },
    {
      "Sid": "ReadExportStatus",
      "Effect": "Allow",
      "Action": [
        "dynamodb:ListExports",
        "dynamodb:DescribeExport"
      ],
      "Resource": "*"
    },
    {
      "Sid": "WriteExportObjects",
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": "arn:aws:s3:::company-prod-dynamodb-export/customer-order-status/full/*"
    }
  ]
}

If PITR is managed by another platform role, remove dynamodb:UpdateContinuousBackups from this export requester role.

If export uses a customer managed KMS key, also add this to the export requester identity policy:

{
  "Sid": "UseExportKmsKey",
  "Effect": "Allow",
  "Action": [
    "kms:Encrypt",
    "kms:GenerateDataKey",
    "kms:DescribeKey"
  ],
  "Resource": "arn:aws:kms:ap-northeast-1:111122223333:key/abcd-1234-example"
}

bucket policy#

Use this when the bucket is cross-account or the bucket policy is restrictive. The principal is the export requester role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowDynamoDBExportRequesterWrite",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/prod-dynamodb-export-role"
      },
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": "arn:aws:s3:::company-prod-dynamodb-export/customer-order-status/full/*"
    },
    {
      "Sid": "AllowExportReaderList",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/prod-dynamodb-export-reader-role"
      },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::company-prod-dynamodb-export",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "customer-order-status/full/*"
          ]
        }
      }
    },
    {
      "Sid": "AllowExportReaderRead",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/prod-dynamodb-export-reader-role"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::company-prod-dynamodb-export/customer-order-status/full/*"
    }
  ]
}

Baseline bucket security:

aws s3api put-public-access-block --bucket company-prod-dynamodb-export --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true --region ap-northeast-1

aws s3api put-bucket-encryption --bucket company-prod-dynamodb-export --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256"
        }
      }
    ]
  }' --region ap-northeast-1

kms key policy#

If export uses SSE-S3 (AES256), KMS permission is not needed.

If export uses a customer managed KMS key, the requester role needs encrypt/data-key permissions in both IAM and the KMS key policy. The KMS key must be in the same Region as the destination bucket.

{
  "Sid": "AllowDynamoDBExportRequesterUseKey",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::111122223333:role/prod-dynamodb-export-role"
  },
  "Action": [
    "kms:Encrypt",
    "kms:GenerateDataKey",
    "kms:DescribeKey"
  ],
  "Resource": "*"
}

Export reader role needs decrypt:

{
  "Sid": "AllowExportReaderDecrypt",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::111122223333:role/prod-dynamodb-export-reader-role"
  },
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "*"
}

3. Export#

Enable PITR:

aws dynamodb update-continuous-backups --table-name customer_order_status --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true --region ap-northeast-1

Start export:

TABLE_ARN="arn:aws:dynamodb:ap-northeast-1:111122223333:table/customer_order_status"
BUCKET="company-prod-dynamodb-export"
PREFIX="customer-order-status/full/$(date +%Y%m%d-%H%M%S)"

aws dynamodb export-table-to-point-in-time --table-arn "${TABLE_ARN}" --s3-bucket "${BUCKET}" --s3-prefix "${PREFIX}" --export-format DYNAMODB_JSON --s3-sse-algorithm AES256 --region ap-northeast-1

If using KMS:

aws dynamodb export-table-to-point-in-time --table-arn "${TABLE_ARN}" --s3-bucket "${BUCKET}" --s3-prefix "${PREFIX}" --export-format DYNAMODB_JSON --s3-sse-algorithm KMS --s3-sse-kms-key-id arn:aws:kms:ap-northeast-1:111122223333:key/abcd-1234-example --region ap-northeast-1

Check export:

aws dynamodb list-exports --table-arn "${TABLE_ARN}" --region ap-northeast-1

aws s3 ls "s3://${BUCKET}/${PREFIX}/" --recursive

Expected layout:

s3://bucket/prefix/AWSDynamoDB/<export-id>/
    manifest-summary.json
    manifest-files.json
    data/<file>.json.gz

4. Pull And Query With Python#

这个脚本从 S3 读取 manifest-files.json，下载每个 data file，反序列化 DynamoDB JSON，然后在本地过滤。

pip install boto3

export AWS_REGION=ap-northeast-1
export EXPORT_BUCKET=company-prod-dynamodb-export
export EXPORT_PREFIX='customer-order-status/full/20260618-120000/AWSDynamoDB/01600000000000-example'
export CUSTOMER_ID='cus_8f3a91'
export ORDER_ID='ord_20260618_0001'

This script matches the table created in Python CRUD:

table:
    customer_order_status

key:
    customer_id
    order_id

expected exported item fields:
    customer_id = cus_8f3a91
    order_id = ord_20260618_0001
    status = PAID or SHIPPED
    total_amount = 129.90
    currency = HKD
    payment_id = pay_20260618_0001
    warehouse_id = hk-east-1
    shipping_city = Hong Kong

Reader role permissions:

used by:
    the role that runs query_export.py

needs:
    s3:GetObject for manifest and data files
    s3:ListBucket only if script / operator lists prefixes
    kms:Decrypt only when export objects use customer managed KMS key

query_export.py:

import gzip
import json
import os
from decimal import Decimal
from io import BytesIO

import boto3
from boto3.dynamodb.types import TypeDeserializer


REGION = os.getenv("AWS_REGION", "ap-northeast-1")
BUCKET = os.environ["EXPORT_BUCKET"]
EXPORT_PREFIX = os.environ["EXPORT_PREFIX"].rstrip("/")
CUSTOMER_ID = os.getenv("CUSTOMER_ID", "cus_8f3a91")
ORDER_ID = os.getenv("ORDER_ID", "ord_20260618_0001")

s3 = boto3.client("s3", region_name=REGION)
deserializer = TypeDeserializer()


def to_plain(value):
    if isinstance(value, Decimal):
        return int(value) if value % 1 == 0 else float(value)
    if isinstance(value, dict):
        return {k: to_plain(v) for k, v in value.items()}
    if isinstance(value, list):
        return [to_plain(v) for v in value]
    return value


def read_manifest_files():
    obj = s3.get_object(Bucket=BUCKET, Key=f"{EXPORT_PREFIX}/manifest-files.json")
    body = obj["Body"].read().decode("utf-8")
    for line in body.splitlines():
        if line.strip():
            yield json.loads(line)


def read_export_items():
    for entry in read_manifest_files():
        data_key = entry["dataFileS3Key"]
        obj = s3.get_object(Bucket=BUCKET, Key=data_key)
        raw = obj["Body"].read()
        with gzip.GzipFile(fileobj=BytesIO(raw)) as gz:
            for line in gz:
                record = json.loads(line)
                item = record.get("Item", record)
                yield to_plain({k: deserializer.deserialize(v) for k, v in item.items()})


def find_orders_by_customer(customer_id: str):
    for item in read_export_items():
        if item.get("customer_id") == customer_id:
            yield item


def find_order(customer_id: str, order_id: str):
    for item in find_orders_by_customer(customer_id):
        if item.get("order_id") == order_id:
            return item
    return None


def find_orders_by_status(status: str):
    for item in read_export_items():
        if item.get("status") == status:
            yield item


if __name__ == "__main__":
    order = find_order(CUSTOMER_ID, ORDER_ID)
    print("one order:")
    print(json.dumps(order, ensure_ascii=False, indent=2))

    print("customer orders:")
    for item in find_orders_by_customer(CUSTOMER_ID):
        print(json.dumps(item, ensure_ascii=False, indent=2))

    print("shipped orders:")
    for item in find_orders_by_status("SHIPPED"):
        print(json.dumps(item, ensure_ascii=False, indent=2))

Run:

python query_export.py

5. Production Notes#

large export:
    do not load all rows into memory
    stream gzip files line by line
    push query to Athena / Glue / Spark when data is large

security:
    export bucket blocks public access
    bucket policy only allows approved roles
    use KMS when data is sensitive
    set lifecycle expiration for temporary exports

cost:
    export cost
    S3 storage cost
    Athena / Glue / Spark scan cost

Minimal IAM for export reader:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::company-prod-dynamodb-export",
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "customer-order-status/full/*"
          ]
        }
      }
    },
    {
      "Sid": "ReadExportObjects",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::company-prod-dynamodb-export/customer-order-status/full/*"
    }
  ]
}

If export objects use a customer managed KMS key, add to the reader role:

{
  "Effect": "Allow",
  "Action": [
    "kms:Decrypt",
    "kms:DescribeKey"
  ],
  "Resource": "arn:aws:kms:ap-northeast-1:111122223333:key/abcd-1234-example"
}

Export S3