Hey folks
I am seeing mindblowing difference in resource usage when using flink 1.19 KinesisStreamsConsumer, between ELO & POLLING.
Kinesis stream has 220MB/s data rate w/ 900 shards.
EFO_CONSUMER is maxing out 16 CPU, 30 instances, parallelism=450
POLLING is stable at 2.2 CPU, 18 instances, parallelism=90
EFO has 10x better record retrival latency, 150ms vs 1500ms, but I would have expected most of the difference to come from IO, not manifesting itself in such difference in CPU usage.
Does anyone have a theory on whats up?