AWS DynamoDB 跨账号与跨 Region 复制
验证:未记录
- 适用版本
- 未记录
- 验证范围
- 尚未登记验证记录。
本文缩写(英文全称与中文含义)
| 缩写 | 英文全称/名称 | 中文含义 |
|---|---|---|
| ACL | Access Control List | 访问控制列表 |
| AES256 | Advanced Encryption Standard, 256-bit key | 使用 256 位密钥的高级加密标准 |
| API | Application Programming Interface | 应用程序编程接口 |
| ARN | Amazon Resource Name | AWS 资源名称,用于唯一标识资源 |
| AWS | Amazon Web Services | 亚马逊云服务 |
| CDC | Change Data Capture | 变更数据捕获 |
| DLQ | Dead-Letter Queue | 死信队列 |
| ECS | Elastic Container Service | AWS 容器编排服务 |
| EMR | Amazon EMR, originally Elastic MapReduce | AWS 大数据处理服务 |
| ETL | Extract, Transform, Load | 数据抽取、转换与加载 |
| IAM | Identity and Access Management | 身份与访问管理 |
| JSON | JavaScript Object Notation | 一种结构化数据格式 |
| KMS | Key Management Service | 密钥管理服务 |
| LSI | Local Secondary Index | DynamoDB 本地二级索引 |
| MB | Megabyte | 十进制兆字节,1 MB = 10⁶ 字节 |
| RPO | Recovery Point Objective | 恢复点目标,可接受的数据丢失时间范围 |
| S3 | Simple Storage Service | AWS 对象存储服务 |
| SCP | Service Control Policy | AWS Organizations 服务控制策略 |
| SSE | Server-Side Encryption | 服务端加密 |
| TLS | Transport Layer Security | 传输层安全协议 |
| TTL | Time To Live | 存活时间;具体指缓存有效期或记录过期时间 |
| VPC | Virtual Private Cloud | 虚拟私有云网络 |
Links#
- DynamoDB global tables
- DynamoDB Streams
- DynamoDB change data capture for Kinesis Data Streams
- DynamoDB export to Amazon S3
- DynamoDB import from Amazon S3
- DynamoDB resource-based policies
1. 能力边界#
DynamoDB 可以原生跨 Region 复制,但 DynamoDB Global Tables 的 replica tables 必须属于同一个 AWS account。Account A 的 table 不能直接把 Account B 中的 table 加为 Global Tables replica。
same account + cross Region:
use DynamoDB Global Tables
cross account + shared access only:
use DynamoDB resource-based policy
no independent copy is created
cross account + independent table copy:
historical data = PITR Export to S3 + Import from S3
ongoing changes = DynamoDB Streams or Kinesis Data Streams CDC + consumer
cross account backup / restore:
use AWS Backup cross-account copy
not continuous replication| Requirement | Recommended solution | Important limitation |
|---|---|---|
| 同账号多 Region、多活读写 | Global Tables | replica 仍在同一个 account |
| Account B 直接访问 Account A 的原表 | Resource-based policy | 只是授权访问,不是复制或灾备 |
| 跨账号一次性迁移 | Export to S3 + Import from S3 | import 创建新 table,不持续同步 |
| 跨账号持续单向复制 | Streams/Kinesis + Lambda/consumer | 自己负责幂等、删除、积压、告警和恢复 |
| 跨账号备份 | AWS Backup cross-account copy | 恢复时创建 table,不是 near-real-time CDC |
2. 同账号跨 Region:Global Tables#
Global Tables 是 DynamoDB 原生的多 Region replication。它管理 replica、故障恢复和跨 Region 写入复制,不需要自己消费 Streams。
当前需要区分两种一致性模式:
MREC - Multi-Region Eventual Consistency:
multi-active
cross-Region replication is asynchronous
concurrent conflicts use last-writer-wins reconciliation
lower write latency than synchronous cross-Region confirmation
MRSC - Multi-Region Strong Consistency:
strongly consistent multi-Region reads and writes
designed for zero RPO
Region combinations, replica/witness topology, latency, quotas,
unsupported features, and migration constraints must be checked firstGlobal Tables 不等于跨账号方案:
Account A / ap-east-1 table
-> can add replica in Account A / ap-southeast-1
-> cannot add replica in Account B / ap-southeast-1如果组织隔离要求数据必须归 Account B 所有,不能用 resource-based policy 或同账号 Global Tables 替代独立副本。
3. 跨账号只共享访问#
DynamoDB table 和 index 支持 resource-based policy,可以授权另一个 account 的 IAM principal 直接调用 Account A 的 table。
Account B application role
-> cross-account DynamoDB API request
-> Account A table
result:
one physical table
no data copy
no independent retention or failure isolation
request latency depends on table Region适合:
central platform table shared by several accounts
temporary migration access
organization wants one source of truth
Account B does not require an independent recovery copy不适合:
Account A outage or policy error must not affect Account B
Account B needs its own table lifecycle and encryption ownership
data residency requires another Region copy
Account B must continue after Account A is decommissionedResource-based policy 只解决访问授权。Account B 的 identity policy、Account A 的 table policy、Organizations SCP、permissions boundary 和 KMS configuration 仍需共同允许请求。
4. 跨账号独立副本架构#
目标:将 Account A / Region A 的 source table A1 历史数据和后续变更复制到 Account B / Region B 的 destination table B2。
推荐架构:
Account A / Region A
DynamoDB A1
|
| point-in-time snapshot at T0
+----------------> Export to S3 ----------------+
| |
| CDC after T0 v
+----------------> Kinesis Data Streams Import from S3
| |
v v
Lambda / consumer DynamoDB B2
| ^
| AssumeRole |
+-----------------------+
Account B / Region B为什么要同时使用 snapshot 和 CDC:
snapshot only:
exports historical state at T0
misses writes after T0
CDC only:
captures changes after it was enabled
does not reconstruct all older unchanged items
snapshot + CDC:
snapshot establishes baseline
CDC catches changes after snapshot time5. 停机一次性迁移#
如果业务允许从停止 Account A 写入开始,一直到 Account B 验证完成后才恢复服务,可以只迁移一个静止时间点的数据,不需要 Streams、Kinesis 或 CDC consumer。
推荐方案:
stop all writers to A1
-> wait until snapshot time T0 is restorable(停机后,等待Point-in-time recovery (PITR)自动做一次时间是在你停机之后的最新备份 )
-> PITR Export A1 at T0 to S3
-> Import from S3 creates B2
-> configure and validate B2
-> switch application to Account B
-> resume writes on B2这种方案结构简单、不会发生 snapshot 与 CDC 的边界冲突,但停机窗口包含 export、import、索引创建、配置、对账和应用切换,可能持续数小时。先用接近生产规模的数据演练并测量耗时;如果停机窗口不可接受,应使用下一节的 snapshot + CDC 方案。
前置盘点#
迁移前记录:
source table:
table ARN and Region
partition key / sort key name and type
item count and table size
GSI and LSI definitions
billing mode, capacity, maximum throughput, table class
PITR, deletion protection, TTL attribute
stream / Kinesis destination
encryption key, tags, resource policy
destination:
Account B Region and target table name
S3 bucket owner, prefix, bucket policy, encryption key
destination DynamoDB KMS key
import quota and expected write/read capacity after cutoverImport from S3 创建新 table,不会继承 A1 的 PITR、TTL、deletion protection、auto scaling、alarms、resource policy 和 application IAM policy。GSI 需要在 import table creation parameters 中明确创建。
LSI / GSI 限制与迁移选择#
Import from S3 的 table creation parameters 不能定义 LSI,而 LSI 又只能在 table 创建时配置,不能在 import 完成后补加。这里的限制不是“有 LSI 就不能使用 S3”,而是“不能直接使用 DynamoDB 原生 ImportTable 创建一张保留 LSI 的 B2”。
先检查 A1 是否存在 LSI:
aws dynamodb describe-table --profile account-a --region ap-east-1 --table-name A1 --query 'Table.LocalSecondaryIndexes'null or []:
A1 没有 LSI
可以继续使用原生 Export to S3 + ImportTable
returns index definitions:
A1 使用 LSI
必须确认 B2 是否仍需要这些 LSI access patternsLSI 和 GSI 都是 secondary index,但原生 ImportTable 对它们的支持不同:
A1 索引情况 |
能否使用原生 ImportTable | B2 创建方式 |
|---|---|---|
| 无 LSI、无 GSI | 可以 | 在 TableCreationParameters 中只定义 primary key |
| 无 LSI、有 GSI | 可以 | 在 TableCreationParameters 中定义 primary key 和 GlobalSecondaryIndexes |
| 有 LSI、无 GSI | 不能用原生 ImportTable 保留 LSI | 先创建带 LSI 的 B2,再使用 Scan 或 S3 自定义导入写入 |
| 同时有 LSI 和 GSI | 不能用原生 ImportTable 保留 LSI | 先创建带 LSI/GSI 的 B2,再使用自定义导入写入 |
GSI:
ImportTable supports GlobalSecondaryIndexes in TableCreationParameters
DynamoDB creates the GSI while creating B2
LSI:
ImportTable does not support LocalSecondaryIndexes
LSI cannot be added after table creationS3 export 保存的是 item 数据,不会单独导出一个可直接恢复的 index 文件。Item 中作为 GSI/LSI key 的 attributes 仍在数据中;导入时 DynamoDB 根据 TableCreationParameters.GlobalSecondaryIndexes 从这些 attributes 构建 GSI。LSI 因为无法写入 ImportTable 创建参数,所以不能通过这条原生路径重建。
一次性查看 A1 的 LSI 和 GSI:
aws dynamodb describe-table --profile account-a --region ap-east-1 --table-name A1 --query 'Table.{LSI:LocalSecondaryIndexes,GSI:GlobalSecondaryIndexes}'如果检查结果中的 LSI 为 null 或 [],结论是:可以直接使用 DynamoDB 原生 Import from S3,由 Account B 的 ImportTable 从 export data 创建 B2,不需要先 Scan 再逐项写入。此时如果 GSI 非空,必须把所需 GSI 定义加入 TableCreationParameters.GlobalSecondaryIndexes。
Account A:
stop A1 writes
-> PITR Export at T0
-> S3 AWSDynamoDB/<export-id>/data/
Account B preparation:
do NOT create B2 beforehand
prepare import IAM permissions
prepare S3 bucket policy and KMS key policy
prepare TableCreationParameters:
TableName = B2
AttributeDefinitions
KeySchema
GlobalSecondaryIndexes, if required
BillingMode / ProvisionedThroughput
SSESpecification
Account B execution:
call ImportTable with S3 data prefix + TableCreationParameters
-> DynamoDB creates B2
-> DynamoDB loads S3 data into B2
-> wait until ImportStatus = COMPLETED
-> configure PITR / TTL / deletion protection / alarms / IAM
-> validate and cut over applicationAccount B 不需要也不能先执行 CreateTable B2。ImportTable 同时完成“创建 table”和“导入数据”两个动作;如果同名 B2 已存在,原生 import 不能向它追加、合并或覆盖数据,必须换一个未使用的 table name,或者改用 BatchWriteItem 等自定义导入方案。
这里的“直接导入”仍需满足:
B2 does not already exist:
ImportTable always creates a new table
it cannot merge into or overwrite an existing B2
schema is explicit:
partition key and sort key name/type are correct
required GSI definitions are included in TableCreationParameters
input is correct:
S3KeyPrefix points to the export data/ directory, not manifest files
InputFormat and InputCompressionType match the exported files
permissions are complete:
Account B import caller can read the S3 prefix
cross-account bucket policy allows access
source/destination KMS key policies allow required decrypt/encrypt operations
post-import settings are rebuilt:
PITR, TTL, deletion protection, auto scaling, alarms, tags,
resource policy and application IAM are configured separatelyImport 完成后还必须确认 import status 为 COMPLETED、ErrorCount=0,并对账 item count、抽样 item、primary key Query 和 GSI Query。没有 LSI 只代表避开了 LSI 创建限制,不代表其他 schema、权限和数据校验可以省略。
如果 B2 不再需要 LSI,可以继续使用原生 ImportTable。LSI key attributes 仍会作为普通 item attributes 被导入,但目标表没有对应 index,应用不能再执行原来的 LSI Query。
如果 B2 必须保留相同 LSI,有三种主要路径:
| 方案 | 流程 | 适用情况 | 注意项 |
|---|---|---|---|
| 停写后直接复制 | 预建带 LSI 的 B2 -> 停止 A1 写入 -> Scan A1 -> BatchWriteItem B2 |
中小型表、停机窗口充足 | Scan 分页、目标容量、25 items/16 MB batch 限制、UnprocessedItems 重试 |
| S3 快照加自定义导入 | A1 PITR Export -> S3 -> 预建带 LSI 的 B2 -> Glue/EMR/ECS/脚本读取 export files -> BatchWriteItem B2 |
大表、需要固定 T0 快照或并行导入 |
仍可使用 S3,但不能调用原生 ImportTable;需要自行实现反序列化、并发、重试和对账 |
| AWS Backup copy/restore | 跨账号复制 backup -> Account B restore | 希望由备份恢复保留更多 table schema | 上线前必须实测 LSI、GSI、KMS、Region、vault policy 和恢复后 table settings,不要只根据 backup 成功判断可切换 |
直接 Scan -> BatchWriteItem 时,BatchWriteItem 不支持整批事务和条件表达式,并可能返回 UnprocessedItems;程序必须指数退避并持续重试,直到所有 item 写入完成。即使使用 parallel Scan,也要限制并发,避免打满 A1 read capacity 或 B2 write capacity。
对于停机迁移,所有业务 writer 停止后再 Scan 可以避免普通应用更新造成数据漂移,但 TTL 后台删除仍需单独处理。对于数据量较大或要求明确快照边界的迁移,优先选择 PITR Export 到 S3,再由自定义导入程序写入预建 B2。
停写与快照时间#
停止的不只是 Web API,还包括所有可能修改 A1 的 writer:
application services and background workers
Lambda / Step Functions / scheduled jobs
ETL, admin scripts, repair jobs, console writes
stream consumers that write back to A1
TTL background deletion, if exact item set must remain unchanged先确认 PITR:
aws dynamodb describe-continuous-backups --profile account-a --region ap-east-1 --table-name A1关注:
PointInTimeRecoveryStatus = ENABLED
EarliestRestorableDateTime
LatestRestorableDateTime停止写入后记录时间 T0。DynamoDB 的 LatestRestorableDateTime 通常落后当前时间几分钟,必须等它达到或晚于 T0,再导出 T0。如果 PITR 此前没有启用,应先启用并等待可恢复时间覆盖停写时刻,不能导出启用 PITR 之前的数据状态。
停机期间若继续发生 TTL 删除,table 仍可能变化。最简单的做法是提前禁用 TTL 并等待状态稳定,或者明确以选定 T0 的 PITR 快照为唯一迁移基线,不再用停机后的 live Scan 结果与它混合。
导出与导入#
创建 T0 的 full export。完整 export 权限和 S3/KMS policy 见 DynamoDB Export To S3:
aws dynamodb export-table-to-point-in-time --profile account-a --region ap-east-1 --table-arn arn:aws:dynamodb:ap-east-1:111111111111:table/A1 --export-time 2026-08-22T10:00:00Z --s3-bucket account-b-dynamodb-migration --s3-bucket-owner 222222222222 --s3-prefix A1/T0 --s3-sse-algorithm AES256 --export-format DYNAMODB_JSONAccount B destination S3 policy#
这个跨账号 export 涉及两个授权面,两边都必须允许:
Account A export requester role identity policy:
allows the role to call DynamoDB export
allows the role to write the Account B S3 prefix
Account B destination bucket policy:
trusts the exact Account A export requester role
allows that role to write the migration prefix
principal:
use the Account A IAM role that calls export-table-to-point-in-time
do not set Principal to dynamodb.amazonaws.com假设 Account A 使用 arn:aws:iam::111111111111:role/dynamodb-a1-export-role 执行命令,Account A role 至少需要:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ExportA1",
"Effect": "Allow",
"Action": [
"dynamodb:ExportTableToPointInTime",
"dynamodb:DescribeTable",
"dynamodb:DescribeContinuousBackups"
],
"Resource": "arn:aws:dynamodb:ap-east-1:111111111111:table/A1"
},
{
"Sid": "ReadExportStatus",
"Effect": "Allow",
"Action": [
"dynamodb:DescribeExport",
"dynamodb:ListExports"
],
"Resource": "*"
},
{
"Sid": "WriteAccountBMigrationPrefix",
"Effect": "Allow",
"Action": [
"s3:AbortMultipartUpload",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::account-b-dynamodb-migration/A1/T0/*"
}
]
}Account B 的 account-b-dynamodb-migration bucket policy 加入以下 statement。若 bucket 已有 policy,应合并 Statement,不要覆盖已有 TLS、Organization 或其他 guardrail:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAccountAExportRoleWriteA1Prefix",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111111111:role/dynamodb-a1-export-role"
},
"Action": [
"s3:AbortMultipartUpload",
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::account-b-dynamodb-migration/A1/T0/*"
}
]
}推荐让 Account B bucket 使用 BucketOwnerEnforced。这样 ACL 被禁用,Account B 自动拥有写入 bucket 的 export objects;Block Public Access 应继续保持启用:
aws s3api put-bucket-ownership-controls --profile account-b --region ap-southeast-1 --bucket account-b-dynamodb-migration --ownership-controls 'Rules=[{ObjectOwnership=BucketOwnerEnforced}]'s3:PutObjectAcl 出现在 AWS 的 export permission 基线中;授权它不等于启用 ACL。BucketOwnerEnforced 下 ACL authorization 仍然关闭,不要依赖 object ACL 做 ownership 或访问控制。
上面的 export 命令显式使用 AES256,因此 S3 objects 使用 SSE-S3,不需要跨账号 KMS key policy。如果改用 Account B 的 customer managed KMS key:
aws dynamodb export-table-to-point-in-time --profile account-a --region ap-east-1 --table-arn arn:aws:dynamodb:ap-east-1:111111111111:table/A1 --export-time 2026-08-22T10:00:00Z --s3-bucket account-b-dynamodb-migration --s3-bucket-owner 222222222222 --s3-prefix A1/T0 --s3-sse-algorithm KMS --s3-sse-kms-key-id arn:aws:kms:ap-southeast-1:222222222222:key/MIGRATION-KEY-ID --export-format DYNAMODB_JSON此时 Account A export role 的 identity policy 和 Account B KMS key policy 都需要允许:
{
"Sid": "AllowAccountAExportRoleEncryptMigrationObjects",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111111111:role/dynamodb-a1-export-role"
},
"Action": [
"kms:Encrypt",
"kms:GenerateDataKey",
"kms:DescribeKey"
],
"Resource": "*"
}不要使用 Account B 的 AWS managed key aws/s3 做这种跨账号授权,因为它的 key policy 不能由客户修改。后续 Account B 执行 ImportTable 时,import caller 还需要对 export prefix 的 s3:GetObject,以及使用该 KMS key 的 kms:Decrypt 权限。
常见阻断项:
bucket policy Principal 写成 dynamodb.amazonaws.com,而不是 Account A requester role
Account A role 有 S3 identity permission,但 Account B bucket policy 没有 cross-account allow
bucket policy 的 prefix 与 --s3-prefix 不一致
bucket policy/SCP/VPC endpoint policy 中存在 explicit deny
bucket 强制 KMS,但 export command 没有指定允许跨账号使用的 customer managed key
KMS identity policy 已允许,但 Account B key policy 没有信任 Account A role等待 export 为 COMPLETED,记录 export ARN、ItemCount、BilledSizeBytes 和实际 data prefix。不要在 export 完成前启动 import。
Export 输出中的可导入数据位于 AWSDynamoDB/<export-id>/data/;manifest 文件不应作为 import data。创建 import-table-parameters.json:
{
"TableName": "B2",
"AttributeDefinitions": [
{
"AttributeName": "pk",
"AttributeType": "S"
},
{
"AttributeName": "sk",
"AttributeType": "S"
}
],
"KeySchema": [
{
"AttributeName": "pk",
"KeyType": "HASH"
},
{
"AttributeName": "sk",
"KeyType": "RANGE"
}
],
"BillingMode": "PAY_PER_REQUEST",
"SSESpecification": {
"Enabled": true,
"SSEType": "KMS",
"KMSMasterKeyId": "arn:aws:kms:ap-southeast-1:222222222222:key/DESTINATION-KEY-ID"
}
}上面的示例只包含 primary key。存在 GSI 时,需要同时修改两个位置:
AttributeDefinitions:
include every table primary key and GSI key attribute
do not list ordinary non-key attributes
GlobalSecondaryIndexes:
define IndexName
define GSI KeySchema
define Projection
define GSI ProvisionedThroughput only when BillingMode=PROVISIONED例如 A1 使用:
table primary key:
pk HASH String
sk RANGE String
GSI status-created-at-index:
gsi_status HASH String
created_at RANGE String
projection includes total_amount and currency对应的 on-demand import-table-parameters.json:
{
"TableName": "B2",
"AttributeDefinitions": [
{
"AttributeName": "pk",
"AttributeType": "S"
},
{
"AttributeName": "sk",
"AttributeType": "S"
},
{
"AttributeName": "gsi_status",
"AttributeType": "S"
},
{
"AttributeName": "created_at",
"AttributeType": "S"
}
],
"KeySchema": [
{
"AttributeName": "pk",
"KeyType": "HASH"
},
{
"AttributeName": "sk",
"KeyType": "RANGE"
}
],
"GlobalSecondaryIndexes": [
{
"IndexName": "status-created-at-index",
"KeySchema": [
{
"AttributeName": "gsi_status",
"KeyType": "HASH"
},
{
"AttributeName": "created_at",
"KeyType": "RANGE"
}
],
"Projection": {
"ProjectionType": "INCLUDE",
"NonKeyAttributes": [
"total_amount",
"currency"
]
}
}
],
"BillingMode": "PAY_PER_REQUEST",
"SSESpecification": {
"Enabled": true,
"SSEType": "KMS",
"KMSMasterKeyId": "arn:aws:kms:ap-southeast-1:222222222222:key/DESTINATION-KEY-ID"
}
}PAY_PER_REQUEST 模式下不要为 table 或 GSI 添加 ProvisionedThroughput。如果使用 PROVISIONED,table 和每个 GSI 都需要指定容量,例如:
{
"BillingMode": "PROVISIONED",
"ProvisionedThroughput": {
"ReadCapacityUnits": 100,
"WriteCapacityUnits": 100
},
"GlobalSecondaryIndexes": [
{
"IndexName": "status-created-at-index",
"KeySchema": [
{
"AttributeName": "gsi_status",
"KeyType": "HASH"
},
{
"AttributeName": "created_at",
"KeyType": "RANGE"
}
],
"Projection": {
"ProjectionType": "INCLUDE",
"NonKeyAttributes": [
"total_amount",
"currency"
]
},
"ProvisionedThroughput": {
"ReadCapacityUnits": 100,
"WriteCapacityUnits": 100
}
}
]
}第二个 JSON 只展示 capacity 差异,不是完整配置文件;实际文件仍需包含 TableName、AttributeDefinitions、KeySchema 和 SSE 配置。
从 A1 读取 key/index 定义作为准备材料:
aws dynamodb describe-table --profile account-a --region ap-east-1 --table-name A1 --query 'Table.{AttributeDefinitions:AttributeDefinitions,KeySchema:KeySchema,GlobalSecondaryIndexes:GlobalSecondaryIndexes[].{IndexName:IndexName,KeySchema:KeySchema,Projection:Projection}}'不要把 describe-table 返回的整个 GlobalSecondaryIndexes 对象原样粘贴到 ImportTable 参数,因为其中可能包含 IndexArn、IndexStatus、ItemCount 等只读状态字段。只保留 ImportTable 接受的 IndexName、KeySchema、Projection,以及 provisioned 模式需要的 ProvisionedThroughput。
可以通过 DescribeTable + jq 从 A1 生成大部分 import-table-parameters.json。这不是 AWS 提供的原生 export 命令,而是把 source table description 转换成 ImportTable 接受的结构。完整的查询、生成和检查流程见 从 DynamoDB Table 生成 ImportTable 参数。
TARGET_TABLE=B2
TARGET_KMS_KEY_ARN=arn:aws:kms:ap-southeast-1:222222222222:key/DESTINATION-KEY-ID
aws dynamodb describe-table --profile account-a --region ap-east-1 --table-name A1 --output json | jq --arg target_table "${TARGET_TABLE}" --arg target_kms_key_arn "${TARGET_KMS_KEY_ARN}" '
.Table as $source
| if (($source.LocalSecondaryIndexes // []) | length) > 0 then
error("A1 has LSI; native ImportTable cannot preserve it")
else
($source.BillingModeSummary.BillingMode // "PROVISIONED") as $billing_mode
| (
{
TableName: $target_table,
AttributeDefinitions: $source.AttributeDefinitions,
KeySchema: $source.KeySchema,
BillingMode: $billing_mode,
SSESpecification: {
Enabled: true,
SSEType: "KMS",
KMSMasterKeyId: $target_kms_key_arn
}
}
+ (
if $billing_mode == "PROVISIONED" then
{
ProvisionedThroughput: {
ReadCapacityUnits: $source.ProvisionedThroughput.ReadCapacityUnits,
WriteCapacityUnits: $source.ProvisionedThroughput.WriteCapacityUnits
}
}
else
{}
end
)
+ (
if (($source.GlobalSecondaryIndexes // []) | length) > 0 then
{
GlobalSecondaryIndexes: [
$source.GlobalSecondaryIndexes[]
| (
{
IndexName: .IndexName,
KeySchema: .KeySchema,
Projection: .Projection
}
+ (
if $billing_mode == "PROVISIONED" then
{
ProvisionedThroughput: {
ReadCapacityUnits: .ProvisionedThroughput.ReadCapacityUnits,
WriteCapacityUnits: .ProvisionedThroughput.WriteCapacityUnits
}
}
else
{}
end
)
)
]
}
else
{}
end
)
)
end
' > import-table-parameters.json生成后必须人工 review:
jq . import-table-parameters.jsonmust verify:
TableName is the new Account B table name
AttributeDefinitions contains only table/GSI key attributes
KeySchema matches A1
every required GSI has correct key schema and projection
BillingMode and capacity are appropriate for cutover traffic
KMSMasterKeyId belongs to Account B and the destination Region
not automatically carried by this generator:
on-demand maximum throughput and warm throughput
table class
TTL, PITR, deletion protection
auto scaling, alarms, tags
resource policy and application IAM如果 A1 和 B2 逻辑结构相同,这个生成文件可以作为起点,但不能把“相同”理解为复用 Account A 的 KMS key、ARN 或运行时状态。跨账号迁移仍要替换 destination table name/KMS,并重新确认容量和安全配置。
如果某个 item 缺少 GSI partition key 或 sort key attribute,该 item 不会进入该 GSI,这就是 sparse GSI 行为。导入后必须使用代表性 Query 验证 GSI 数据和 Projection 是否符合应用需求。
这里不用提前运行 create-table,而是在 Account B 调用 ImportTable,由 DynamoDB 根据完整的 TableCreationParameters 创建 B2、构建 GSI 并装载数据:
aws dynamodb import-table --profile account-b --region ap-southeast-1 --s3-bucket-source S3BucketOwner=222222222222,S3Bucket=account-b-dynamodb-migration,S3KeyPrefix=A1/T0/AWSDynamoDB/EXPORT-ID/data/ --input-format DYNAMODB_JSON --input-compression-type GZIP --table-creation-parameters file://import-table-parameters.json命令返回 ImportArn 后,持续检查状态:
aws dynamodb describe-import --profile account-b --region ap-southeast-1 --import-arn arn:aws:dynamodb:ap-southeast-1:222222222222:table/B2/import/IMPORT-ID --query 'ImportTableDescription.{Status:ImportStatus,Processed:ProcessedItemCount,Imported:ImportedItemCount,Errors:ErrorCount,FailureCode:FailureCode,FailureMessage:FailureMessage}'IN_PROGRESS:
B2 is being created and data is being loaded
application must not cut over yet
COMPLETED:
import service finished
continue schema/configuration/data validation before cutover
FAILED or ErrorCount > 0:
inspect FailureCode / FailureMessage and import error logs
do not use B2 as a completed migration调用者需要 DynamoDB import 权限、source S3 prefix 读取权限,以及使用 source/destination customer managed KMS key 所需的权限。跨账号 S3 bucket policy 和 KMS key policy 中的 explicit deny 仍会阻止 import。
验证与切换#
Import 变为 COMPLETED 后不要立刻恢复业务,先完成:
data:
export ItemCount vs import ImportedItemCount
import ErrorCount = 0
sample GetItem for known keys
critical migration uses partitioned Scan + deterministic item hash
schema and query:
partition/sort key types match
every required GSI is ACTIVE
representative Query and GSI Query return expected results
table configuration:
TTL decision applied
PITR and deletion protection enabled
billing/capacity/auto scaling configured
KMS, tags, resource policy, application IAM, alarms configured
application:
credentials and table Region/name changed
read/write/delete smoke test passes on B2
no application component still writes A1TTL 设置本身不会随数据导入。导出的 item 仍可能包含已经过期的 TTL attribute;在 B2 重新启用 TTL 后,这些 item 可能被异步删除,需要确认这是预期行为。
切换后 B2 成为唯一 writer。不要同时恢复 A1 写入,否则两张表立即分叉。可以保留 A1 只读作为短期回滚基线,但一旦 B2 接受了新写入,直接切回 A1 会丢失这些变更;此时回滚必须先停写并执行反向增量或重新迁移。
停机迁移 Checklist#
[ ] 停机时长已按生产规模演练
[ ] 所有 A1 writers 和 TTL 行为已盘点
[ ] PITR 已启用且 LatestRestorableDateTime 覆盖 T0
[ ] export status = COMPLETED
[ ] import source prefix 只包含 export data files
[ ] A1 没有无法由 Import from S3 重建的 LSI 要求
[ ] B2 key schema、GSI、KMS 和 billing mode 正确
[ ] import status = COMPLETED and ErrorCount = 0
[ ] item count、抽样 item、hash 和 query 已对账
[ ] B2 PITR、TTL、deletion protection、IAM 和 alarms 已配置
[ ] 应用 smoke test 通过,且 A1 保持无写入
[ ] 已明确 B2 接受新写入后的回滚办法6. 无数据缺口的在线迁移顺序#
推荐使用 Kinesis Data Streams for DynamoDB 承接较长迁移,因为 DynamoDB Streams 的记录只保留约 24 小时,而大型 export/import 可能超过这个时间。
- 盘点
A1key schema、GSI/LSI、billing mode、capacity、TTL、PITR、encryption、tags 和 item 数量。 - 创建或选择 Kinesis Data Stream,设置足够覆盖 export、import、验证和故障重试的 retention。
- 启用
A1 -> Kinesisstreaming destination,并确认状态为 Active。 - 记录 snapshot 时间
T0,创建A1的 PITR export。 - 在 Account B 从 export data 创建
B2;Import from S3 会创建新 table,不能覆盖已有 table。 - 为 CDC consumer 配置 Account B writer role,并让 consumer 从
T0之后开始回放。 - CDC 追平后,对账 item count、抽样 item、删除、TTL、GSI 查询和 consumer lag。
- 停止 Account A 写入或进入短暂 maintenance window,等待 CDC lag 归零。
- 将应用切到
B2,并保留回滚窗口。 - 确认不再回滚后,停止旧 CDC pipeline,并按 retention policy 处理
A1和 export files。
T0 边界可能出现少量重复事件。不要依赖“刚好只消费一次”,consumer 必须幂等,并以 item key 和业务版本/更新时间处理重复或过期变更。
小表替代方案#
如果数据量小、迁移可在 24 小时内完成,也可以使用 DynamoDB Streams:
A1 Streams NEW_AND_OLD_IMAGES
-> Lambda in Account A
-> AssumeRole into Account B
-> PutItem / DeleteItem on B2但如果 target 要通过 Import from S3 创建,在 import 完成前 consumer 没有可写目标。需要先把 stream records 持久化到 Kinesis/S3,或确认整个流程不会超过 Streams retention。不要让 Lambda 错误重试 24 小时来充当持久缓冲区。
7. 历史数据回填#
PITR Export 不消耗 source table 的 read capacity,适合大表迁移。要求 A1 已启用 PITR,并选择明确的 export time T0。
历史数据流程:
A1 PITR at T0
-> ExportTableToPointInTime
-> S3 DynamoDB JSON or Ion files
-> ImportTable
-> new B2 table需要注意:
Import creates a new table:
target table name and settings are chosen during import
cannot merge into or overwrite an existing B2 table
table configuration:
verify partition key and sort key types
recreate GSI definitions intentionally
review billing mode, capacity, table class, deletion protection, PITR, TTL, tags
verify customer managed KMS key in Account B
data validation:
compare export item count with imported item count
sample keys from different partitions
validate number, binary, list, map, empty value, and large item handlingExport、跨账号 S3 bucket policy、KMS 和输出格式见 DynamoDB Export To S3。
8. 增量 CDC 映射#
CDC 使用 NEW_AND_OLD_IMAGES 时,consumer 可以明确处理创建、修改和删除:
| Source event | Destination action | Notes |
|---|---|---|
INSERT |
PutItem |
写入 NewImage |
MODIFY |
PutItem 或受控 UpdateItem |
用 NewImage 替换可以避免 source 已删除属性残留在 B2 |
REMOVE |
DeleteItem |
使用 event 中的 key;包括用户删除和 TTL 删除 |
如果使用 low-level DynamoDB client,Streams/Kinesis record 中的 NewImage 已是 DynamoDB AttributeValue map,可以直接作为 PutItem.Item。使用 high-level resource client 时需要先反序列化。
def apply_record(record, destination_client, table_name):
event_name = record["eventName"]
keys = record["dynamodb"]["Keys"]
if event_name in ("INSERT", "MODIFY"):
destination_client.put_item(
TableName=table_name,
Item=record["dynamodb"]["NewImage"],
)
elif event_name == "REMOVE":
destination_client.delete_item(
TableName=table_name,
Key=keys,
)这是语义示例,不是完整生产 consumer。生产代码还需要处理 batch partial failure、重试、DLQ、checkpoint、限流、日志、指标和幂等。
TTL 删除#
TTL expiration 最终表现为 source item 的 REMOVE 事件。是否同步到 B2 必须明确:
sync REMOVE:
B2 current state follows A1
recommended for migration / read replica
ignore TTL REMOVE:
B2 keeps historical items
only appropriate for archive use cases
B2 needs independent retention and access controls不要同时依赖“复制 source TTL 删除”和“B2 自己用不同 TTL 删除”而不定义优先级,否则两边保留时间不可预测。
9. 跨账号写入权限#
推荐让 CDC consumer 使用 Account B 中的专用 writer role。Account B role trust policy 只信任 Account A 的 consumer execution role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111111111111:role/dynamodb-a1-cdc-consumer"
},
"Action": "sts:AssumeRole"
}
]
}Account B writer role 的 identity policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "WriteReplicatedItems",
"Effect": "Allow",
"Action": [
"dynamodb:DescribeTable",
"dynamodb:PutItem",
"dynamodb:UpdateItem",
"dynamodb:DeleteItem",
"dynamodb:BatchWriteItem"
],
"Resource": [
"arn:aws:dynamodb:ap-southeast-1:222222222222:table/B2",
"arn:aws:dynamodb:ap-southeast-1:222222222222:table/B2/index/*"
]
}
]
}Account A consumer role 还需要 sts:AssumeRole 指向上述 Account B role。实际生产中,如果 consumer 不调用 UpdateItem、BatchWriteItem 或 index ARN,不要授予对应权限。
也可以直接在 B2 使用 resource-based policy 授权 Account A consumer role,但 AssumeRole 方案通常更容易让 Account B 集中控制凭证、CloudTrail principal 和撤权边界。
10. 一致性与冲突#
自建跨账号 CDC 不是 Global Tables,不会自动提供 conflict reconciliation。
recommended migration mode:
A1 is the only writer before cutover
B2 is replication-writer-only
after CDC catches up, stop A1 writes and cut over once
avoid:
applications write A1 and B2 concurrently
B2 application writes race with CDC PutItem
two-way CDC without loop prevention and conflict ownership至少需要定义:
idempotency:
repeated INSERT/MODIFY/REMOVE produces the same target state
ordering:
preserve order for changes to the same item key
do not assume global order across partition keys
stale event protection:
include application version or updated_at when old events might arrive late
use conditional writes when target can receive another writer
delete protection:
an old MODIFY must not resurrect an item after a newer REMOVE
loop prevention:
never send B2 changes back to A1 unless events carry origin and version controls11. 监控与对账#
持续监控:
source:
DynamoDB write throttles
Kinesis streaming destination status
stream:
IncomingRecords / IncomingBytes
GetRecords.IteratorAgeMilliseconds
read/write throttles
consumer:
Lambda Errors / Throttles / Duration
IteratorAge
partial batch failures
DLQ or failure destination depth
destination:
UserErrors / SystemErrors
write throttle events
consumed write capacity迁移对账不能只比较 table item count:count 是近似值,并且无法发现相同 key 的内容差异。至少执行:
export/import item count reconciliation
sample GetItem by known keys
partitioned Scan with deterministic item hash for critical migrations
INSERT / MODIFY / REMOVE / TTL canary tests
GSI query result validation
consumer lag reaches zero before cutover
CloudTrail confirms only approved replication principal writes B212. 方案选择#
same account, multi-Region active-active:
Global Tables
cross account, no independent copy required:
resource-based policy
cross account, one-time migration:
PITR Export to S3 + Import from S3
cross account, historical + ongoing replication:
PITR Export/Import + Kinesis CDC consumer
cross account, small table and migration completes within 24h:
Export/Import or controlled backfill + DynamoDB Streams
cross account backup for restore only:
AWS Backup cross-account copy13. 在线迁移上线 Checklist#
[ ] 已确认是共享访问、一次迁移、持续单向复制还是多活
[ ] 已确认 Global Tables 不能跨 account 创建 replica
[ ] A1 PITR 已启用,snapshot time T0 已记录
[ ] Kinesis retention 覆盖 export/import/验证/重试时间
[ ] B2 key schema、GSI、billing、capacity、TTL、PITR、KMS 已确认
[ ] Account B writer role 只授权 B2 所需 actions
[ ] CDC consumer 处理 INSERT/MODIFY/REMOVE 和 TTL
[ ] consumer 支持幂等、partial failure、DLQ 和 checkpoint
[ ] B2 在 cutover 前没有未受控业务写入
[ ] item count、抽样 hash、GSI query 和 delete behavior 已对账
[ ] consumer lag 归零后才执行 cutover
[ ] rollback window 和旧表下线策略已确认