Kinesis Streams Gotcha - Scaling On-Demand Streams

date

Feb 23, 2023

slug

kinesis-streams-gotcha-scaling-on-demand-streams

author

status

Public

Kinesis Data Stream On-Demand Gotcha

ON_DEMAND Kinesis data streams are originally created with 4 shards and help scale data streams when the traffic load is unknown. By design, ON_DEMAND streams will look at network traffic from 15 days ago to determine peak operations that were experienced. ON_DEMAND streams are capable of scaling up one level. For example, 4 Shards scales up to 8 Shards, and 8 shards scales up to 16 Shards, etc…

The issue occurs whenever traffic/write operations are doubled compared to what was seen in the last 15 days. This causes AWS Kinesis Streams that are ON_DEMAND to get throttled returning a message similar to the following: Rate exceeded for shard shardId-XXXX in stream <STREAM NAME> under account <ACCOUNT NUMBER>.

In order to fix this issue, we have to switch the Kinesis Data Stream from ON_DEMAND to PROVISIONED in order to be able to increase the shard count. After the shard count is increased, we need to switch back to ON_DEMAND to maintain the new shard count.

Even when managing the Kinesis Data Stream through Terraform, we are unable to increase the shard count without first switching over to PROVISIONED. Hence, watch out for this gotcha.

Even with this gotcha, using ON_DEMAND is still favored since it’s able to handle an increase of up to 49 percent of previous traffic/operations seen.