Hi, I have a question regarding the KinesisStreams...
# troubleshooting
h
Hi, I have a question regarding the KinesisStreamsSource. I’m currently working on upgrading an application from Flink 1.18.1 to Flink 1.20, using flink-connector-aws with AWS Kinesis Data Stream as the source. I couldn’t find the SHARD_GETRECORDS_INTERVAL_MILLIS configuration for the new KinesisStreamsSource within the flink-connector-aws-kinesis-streams for Flink 1.20. Was this intentional, and if so, what is the recommended way to manage the GetRecords call frequency with the new KinesisStreamsSource? --- According to the AWS documentation, the GetRecords API has a call limit of 5 TPS per shard. The 200ms interval (1000ms / 5 TPS = 200ms) configurable in FlinkKinesisConsumer seemed appropriate to respect this limit. In fact, when testing with KinesisStreamsSource, I’ve observed through AWS CloudWatch Metrics that approximately 10-20% of the GetRecords API calls are failing. I suspect this might be due to hitting the API limits, as there’s no explicit interval control. --- Thanks
a
We're face the same issue. We're planning to increase the backoff period by setting
setRetryStrategyMinDelay
and
setRetryStrategyMaxDelay
. I'm hoping this will make the Kinesis consumer to wait longer between pulls after getting rate limited by AWS. I don't know how well it works in practice yet.
h
Thanks for the information! If you happen to find out what effects the configuration changes had, could you please let me know? In my environment, I have two Flink applications that need to subscribe to a common Kinesis Data Stream, so the configuration you shared might not work correctly for my setup. I’m aware I should probably be using EFO, but I encountered an issue similar to the one reported here: https://issues.apache.org/jira/browse/FLINK-37648, so I’m holding off on that for now. I’ve forked the repository and am trying to work on a modification to add a configurable ‘Interval’ setting. However, I’m still undecided on whether the library should fundamentally allow this interval to be configured in the first place. Any thoughts or advice?