跨 AWS 账号同步 S3 存量与新增对象

新增对象:
    S3 Replication
    持续、异步复制启用规则后产生的新 object version

存量对象:
    S3 Batch Replication
    一次性回填启用规则前已经存在的 object version

Account A                                      Account B

producer -> A1 source bucket                  B2 destination bucket
               |                                          ^
               | S3 Replication: new versions             |
               +------------------------------------------+
               |                                          |
               | S3 Batch Replication: existing versions  |
               +------------------------------------------+

               replication IAM role in Account A
               destination bucket policy trusts that role

SOURCE_ACCOUNT_ID=111111111111
DESTINATION_ACCOUNT_ID=222222222222
SOURCE_BUCKET=account-a-bucket-a1
DESTINATION_BUCKET=account-b-bucket-b2
SOURCE_REGION=ap-east-1
DESTINATION_REGION=ap-southeast-1
REPLICATION_ROLE_NAME=s3-a1-to-b2-replication

A1 和 B2 都启用 S3 Versioning
B2 已存在，bucket name 和 Region 已确认
Account A 有权限创建 IAM role 和修改 A1 replication configuration
Account B 有权限修改 B2 bucket policy
两边的 SCP、permissions boundary、VPC endpoint policy 没有显式拒绝相关操作
使用 SSE-KMS 时，两边都使用 customer managed KMS key，并配置跨账号 key policy

aws s3api get-bucket-versioning --bucket A1
aws s3api get-bucket-encryption --bucket A1
aws s3api get-bucket-ownership-controls --bucket A1
aws s3api get-object-lock-configuration --bucket A1
aws s3api get-bucket-lifecycle-configuration --bucket A1
aws s3api get-bucket-replication --bucket A1
aws s3api get-bucket-policy --bucket A1

aws s3api put-bucket-versioning --profile account-a --region ap-east-1 --bucket account-a-bucket-a1 --versioning-configuration Status=Enabled

aws s3api put-bucket-versioning --profile account-b --region ap-southeast-1 --bucket account-b-bucket-b2 --versioning-configuration Status=Enabled

existing objects:
    remain as null versions

new PUT or overwrite:
    creates a new version ID
    previous current/null version becomes noncurrent

DELETE without version ID:
    creates a delete marker
    previous data version remains stored

permanent delete/cleanup jobs:
    must handle object versions and delete markers

event consumers:
    may need versionId for idempotency

cost:
    configure Lifecycle for noncurrent versions and incomplete multipart uploads

aws s3api put-bucket-ownership-controls --profile account-b --region ap-southeast-1 --bucket account-b-bucket-b2 --ownership-controls 'Rules=[{ObjectOwnership=BucketOwnerEnforced}]'

Lifecycle filter:
    tag=archive -> transition to Glacier

IAM condition:
    s3:ExistingObjectTag/classification=confidential

operations:
    backup=true
    retention-class=finance-7y
    owner=team-a

aws s3api get-object-tagging --profile account-a --bucket account-a-bucket-a1 --key path/to/object --version-id SOURCE-VERSION-ID

business/HTTP metadata:
    Content-Type
    Cache-Control
    Content-Disposition
    Content-Encoding
    x-amz-meta-* custom metadata

storage/security metadata:
    storage class
    encryption algorithm and KMS key
    replication status

application / operator calls CopyObject on A1
    -> A1 creates a new object version for the same key
    -> continuous replication detects the new version
    -> S3 asynchronously creates its replica in B2

aws s3api copy-object --profile account-a --bucket account-a-bucket-a1 --copy-source account-a-bucket-a1/path/to/object --key path/to/object --metadata-directive REPLACE --content-type application/json --metadata owner=team-a,classification=internal

aws s3api head-object --profile account-a --bucket account-a-bucket-a1 --key path/to/object --version-id SOURCE-VERSION-ID

GOVERNANCE:
    默认不能提前删除
    具有 s3:BypassGovernanceRetention 的受控角色可以显式绕过

COMPLIANCE:
    retention 到期前不能删除
    包括 root user 在内也不能提前缩短或绕过

retention 已到期 + legal hold ON:
    仍然不能删除

retention 未到期 + legal hold OFF:
    仍然不能删除

retention 已到期 + legal hold OFF:
    可以按 IAM 和 Lifecycle 规则删除

aws s3api get-object-lock-configuration --profile account-a --bucket account-a-bucket-a1

aws s3api get-object-retention --profile account-a --bucket account-a-bucket-a1 --key path/to/object --version-id SOURCE-VERSION-ID

aws s3api get-object-legal-hold --profile account-a --bucket account-a-bucket-a1 --key path/to/object --version-id SOURCE-VERSION-ID

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

aws iam create-role --profile account-a --role-name s3-a1-to-b2-replication --assume-role-policy-document file://replication-role-trust.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadSourceBucketConfiguration",
      "Effect": "Allow",
      "Action": [
        "s3:GetReplicationConfiguration",
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::account-a-bucket-a1"
    },
    {
      "Sid": "ReadSourceObjectVersions",
      "Effect": "Allow",
      "Action": [
        "s3:GetObjectVersionForReplication",
        "s3:GetObjectVersionAcl",
        "s3:GetObjectVersionTagging",
        "s3:GetObjectRetention",
        "s3:GetObjectLegalHold"
      ],
      "Resource": "arn:aws:s3:::account-a-bucket-a1/*"
    },
    {
      "Sid": "WriteDestinationReplicas",
      "Effect": "Allow",
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateDelete",
        "s3:ReplicateTags"
      ],
      "Resource": "arn:aws:s3:::account-b-bucket-b2/*"
    }
  ]
}

aws iam put-role-policy --profile account-a --role-name s3-a1-to-b2-replication --policy-name replicate-a1-to-b2 --policy-document file://replication-role-policy.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowReplicationFromAccountA",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:role/s3-a1-to-b2-replication"
      },
      "Action": [
        "s3:ReplicateObject",
        "s3:ReplicateDelete",
        "s3:ReplicateTags"
      ],
      "Resource": "arn:aws:s3:::account-b-bucket-b2/*"
    }
  ]
}

aws s3api put-bucket-policy --profile account-b --region ap-southeast-1 --bucket account-b-bucket-b2 --policy file://destination-bucket-policy.json

{
  "Role": "arn:aws:iam::111111111111:role/s3-a1-to-b2-replication",
  "Rules": [
    {
      "ID": "replicate-all-to-account-b",
      "Priority": 1,
      "Status": "Enabled",
      "Filter": {
        "Prefix": ""
      },
      "DeleteMarkerReplication": {
        "Status": "Enabled"
      },
      "Destination": {
        "Bucket": "arn:aws:s3:::account-b-bucket-b2"
      }
    }
  ]
}

aws s3api put-bucket-replication --profile account-a --region ap-east-1 --bucket account-a-bucket-a1 --replication-configuration file://replication.json

aws s3api get-bucket-replication --profile account-a --region ap-east-1 --bucket account-a-bucket-a1

source KMS key:
    replication role 可以 kms:Decrypt

destination KMS key:
    replication role 可以 kms:Encrypt 和 kms:GenerateDataKey

replication rule:
    启用 SseKmsEncryptedObjects
    指定 destination ReplicaKmsKeyID

{
  "SourceSelectionCriteria": {
    "SseKmsEncryptedObjects": {
      "Status": "Enabled"
    }
  },
  "Destination": {
    "Bucket": "arn:aws:s3:::account-b-bucket-b2",
    "EncryptionConfiguration": {
      "ReplicaKmsKeyID": "arn:aws:kms:ap-southeast-1:222222222222:key/DESTINATION-KEY-ID"
    }
  }
}

continuous replication role:
    trusted service = s3.amazonaws.com
    reads source versions and writes destination replicas

Batch Operations job role:
    trusted service = batchoperations.s3.amazonaws.com
    calls s3:InitiateReplication for each source object in the manifest

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "batchoperations.s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InitiateReplicationForSourceObjects",
      "Effect": "Allow",
      "Action": "s3:InitiateReplication",
      "Resource": "arn:aws:s3:::account-a-bucket-a1/*"
    }
  ]
}

before DELETE:
    reports/a.csv version v2  <- current and visible
    reports/a.csv version v1

after DELETE without version-id:
    reports/a.csv delete marker d1  <- current
    reports/a.csv version v2        <- data still exists
    reports/a.csv version v1

GET reports/a.csv:
    returns 404 because d1 is current

GET reports/a.csv?versionId=v2:
    can still read v2 if the caller has permission

aws s3api list-object-versions --profile account-a --bucket account-a-bucket-a1 --prefix path/to/object

delete current key without version ID:
    creates a delete marker
    can be replicated when DeleteMarkerReplication is enabled

permanently delete a specific source version:
    is not propagated as a permanent deletion of the corresponding replica

lifecycle expires source objects or versions:
    do not assume destination lifecycle follows it
    destination needs its own lifecycle policy

printf 'replication-check\n' > /tmp/replication-check.txt

aws s3api put-object --profile account-a --region ap-east-1 --bucket account-a-bucket-a1 --key replication-canary/replication-check.txt --body /tmp/replication-check.txt

aws s3api head-object --profile account-a --region ap-east-1 --bucket account-a-bucket-a1 --key replication-canary/replication-check.txt --query '{VersionId:VersionId,ReplicationStatus:ReplicationStatus}'

PENDING     等待或正在复制
COMPLETED   已复制成功
FAILED      复制失败，需要检查权限、KMS、bucket policy 和 replication metrics
REPLICA     当前对象本身是 replica

aws s3api head-object --profile account-b --region ap-southeast-1 --bucket account-b-bucket-b2 --key replication-canary/replication-check.txt --query '{VersionId:VersionId,ReplicationStatus:ReplicationStatus,SSE:ServerSideEncryption,KMSKey:SSEKMSKeyId}'

OperationsFailedReplication
OperationsPendingReplication
BytesPendingReplication
ReplicationLatency
s3:Replication:OperationFailedReplication EventBridge event
Batch Operations job failed task count and completion report

新对象可以自动复制
覆盖同一个 key 后，新 version 可以复制
普通 DELETE 的 delete marker 行为符合预期
对象 tag 已复制
SSE-KMS destination key 正确
Account B 对 replica 有读取和 lifecycle 管理权限
存量 batch job 的 manifest 数量、成功数、失败数可以对账

[ ] A1/B2 Versioning 都是 Enabled
[ ] B2 Block Public Access 已启用
[ ] B2 Object Ownership 是 Bucket owner enforced
[ ] replication role trust 只允许 s3.amazonaws.com
[ ] role policy 只覆盖 A1 source 和 B2 destination
[ ] B2 bucket policy 精确授权 replication role
[ ] SSE-KMS 两边 key policy 和 role policy 已配置
[ ] 先启用持续 replication，再启动存量 Batch Replication
[ ] Batch manifest 明确是当前版本还是全部历史版本
[ ] completion report 有独立保存位置并完成失败项复核
[ ] canary 的 create/overwrite/delete/tag/encryption 已验证
[ ] replication metrics、EventBridge failure event 和告警已配置
[ ] B2 lifecycle、retention、Object Lock 需求已独立确认