This message was deleted.

This most likely comes down to increased parallelism, that would be my guess

@Ian Roberts where would the parallelism benefit come from, in the reading of the records, or in the processing of them after they are read?

I would think both in the the instance of peak times. The recommendation is to have a task count equal to the number of shards/partitions or as close to that as possible for this reason.

Thanks @Ian Roberts. What is the general intuition behind that advice?

more tasks than shards/partitions will mean some tasks are not reading anything and less means they will be reading from multiple shards/partitions. It is not always feasible to have a 1:1 ratio but that is the goal if possible for number of tasks (taskCount)

What is the issue if tasks read from multiple tasks? I.e what issues could that cause?

depending on the number of events, there could be lag, I would think that there could be skew if events are not balanced in the streaming service, if there is a variation of throughput it could cause issues during peak times, etc... It is not really an issue to have multiple partitions/shards being read but just something to consider for your particular use case

thanks @Ian Roberts 👍