AWS DynamoDB Kinesis Data Streams

验证：未记录

适用版本: 未记录
验证范围: 尚未登记验证记录。

本文缩写（英文全称与中文含义）

缩写	英文全称／名称	中文含义
API	Application Programming Interface	应用程序编程接口
AWS	Amazon Web Services	亚马逊云服务
CDC	Change Data Capture	变更数据捕获
IAM	Identity and Access Management	身份与访问管理
KCL	Kinesis Client Library	Kinesis 客户端库
KMS	Key Management Service	密钥管理服务
P1	Priority 1	优先级编号；具体严重程度按团队规范定义

Links#

1. Important Points#

DynamoDB 可以把 table item-level changes 写入 Amazon Kinesis Data Streams。它更像 CDC pipeline，不是 Lambda trigger 的轻量版本。

good for:
    many independent consumers
    analytics / lakehouse ingestion
    custom stream processing
    fanout to multiple applications
    longer retention than DynamoDB Streams
    consumer reads with checkpoint / replay control

not for:
    simplest one Lambda trigger
    synchronous business transaction
    low-volume app where Streams is enough

2. Streams vs Kinesis#

Area	DynamoDB Streams	Kinesis Data Streams For DynamoDB
Primary use	table-local change trigger	shared CDC stream
Consumer model	Lambda / Streams API	Kinesis consumers, Lambda, KCL, Firehose-like pipeline
Fanout	limited/simple	better for multiple consumers
Replay	short operational replay	retention configurable in Kinesis
Operational surface	simpler	stream capacity, shard/on-demand, retention, consumers
Best fit	update index/cache/workflow	analytics, data platform, multi-consumer CDC

Decision:

choose DynamoDB Streams:
    one or two consumers
    Lambda trigger is enough
    short retention acceptable

choose Kinesis Data Streams:
    data platform needs CDC
    consumer teams are independent
    stream retention and replay matter
    throughput needs explicit management

3. Enable Kinesis Destination#

Create stream:

aws kinesis create-stream --stream-name orders-ddb-cdc --stream-mode-details StreamMode=ON_DEMAND --region ap-east-1

Enable DynamoDB Kinesis streaming destination:

TABLE_ARN="arn:aws:dynamodb:ap-east-1:111122223333:table/orders"
STREAM_ARN="arn:aws:kinesis:ap-east-1:111122223333:stream/orders-ddb-cdc"

aws dynamodb enable-kinesis-streaming-destination --table-name orders --stream-arn "${STREAM_ARN}" --region ap-east-1

Check status:

aws dynamodb describe-kinesis-streaming-destination --table-name orders --region ap-east-1

4. Read Records With Python#

pip install boto3
export AWS_REGION=ap-east-1
export STREAM_NAME=orders-ddb-cdc

read_kinesis.py:

import json
import os
import time

import boto3


REGION = os.getenv("AWS_REGION", "ap-east-1")
STREAM_NAME = os.getenv("STREAM_NAME", "orders-ddb-cdc")

kinesis = boto3.client("kinesis", region_name=REGION)


def first_shard_id() -> str:
    resp = kinesis.describe_stream_summary(StreamName=STREAM_NAME)
    stream_arn = resp["StreamDescriptionSummary"]["StreamARN"]
    shards = kinesis.list_shards(StreamARN=stream_arn)["Shards"]
    return shards[0]["ShardId"]


def read_latest_records():
    shard_id = first_shard_id()
    iterator = kinesis.get_shard_iterator(
        StreamName=STREAM_NAME,
        ShardId=shard_id,
        ShardIteratorType="LATEST",
    )["ShardIterator"]

    while iterator:
        resp = kinesis.get_records(ShardIterator=iterator, Limit=100)
        for record in resp["Records"]:
            payload = json.loads(record["Data"].decode("utf-8"))
            print(json.dumps(payload, ensure_ascii=False, indent=2))
        iterator = resp.get("NextShardIterator")
        time.sleep(1)


if __name__ == "__main__":
    read_latest_records()

Run:

python read_kinesis.py

For production consumers, prefer Lambda event source mapping or KCL-style checkpointing instead of this simple polling script.

5. Best Practices#

不同 partition key 之间没有全局顺序保证；消费端按业务键处理顺序，并为重复投递和重试实现幂等。

capacity:
    use on-demand stream mode unless workload is predictable
    monitor write/read throttles
    check downstream consumers before increasing producer volume

consumer design:
    one app per consumer group / checkpoint namespace
    make processing idempotent
    store checkpoint outside business table when using custom consumer

schema:
    version event payload
    keep consumer tolerant to new attributes
    do not assume global order across different partition keys

security:
    table role can write to stream through AWS-managed integration
    consumers get kinesis:DescribeStream, ListShards, GetShardIterator, GetRecords
    encrypt stream with KMS for sensitive data

Minimal consumer IAM:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStream",
        "kinesis:DescribeStreamSummary",
        "kinesis:ListShards",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords"
      ],
      "Resource": "arn:aws:kinesis:ap-east-1:111122223333:stream/orders-ddb-cdc"
    }
  ]
}

6. Monitoring#

Signal	Why
`WriteProvisionedThroughputExceeded`	producer write throttle
`ReadProvisionedThroughputExceeded`	consumer read throttle
`GetRecords.IteratorAgeMilliseconds`	consumer lag
`IncomingRecords` / `IncomingBytes`	CDC input volume
Lambda errors / throttles	event source consumer failure

Alert defaults:

IteratorAge:
    P1 when p95 >= 300s for 10m

Read/Write throttle:
    P1 when Sum > 0 for 5m

Consumer errors:
    P1 when errors > 0 for 5m