Hey folks I am seeing mindblowing difference in r...
# troubleshooting
l
Hey folks I am seeing mindblowing difference in resource usage when using flink 1.19 KinesisStreamsConsumer, between ELO & POLLING. Kinesis stream has 220MB/s data rate w/ 900 shards. EFO_CONSUMER is maxing out 16 CPU, 30 instances, parallelism=450 POLLING is stable at 2.2 CPU, 18 instances, parallelism=90 EFO has 10x better record retrival latency, 150ms vs 1500ms, but I would have expected most of the difference to come from IO, not manifesting itself in such difference in CPU usage. Does anyone have a theory on whats up?
App does some light filtering, converts json to avro and writes out to kafka