Hi, I have a data skew issue with kinesis source T...
# troubleshooting
f
Hi, I have a data skew issue with kinesis source The uniform shard assigner assigns the shards uniformly across subtasks, most of the time it does a good job and I get subtasks with either N or N+1 shards assigned (e.g. 10 shards, 3 subtasks I get 3,3,4 shards per subtask Now the problem is that the subtasks are distributed to task managers randomly. Sometimes I get even distribution, but sometimes one task manager gets all the subtasks with N+1 shards and other task manager gets all the tasks with N subtasks I tried to implement my own KinesisShardAssigner that takes into account the location in the ReaderInfo This works as I want it to, but only when the job starts fresh, when the job is restored from a checkpoint, it doesn't re-assign the shards, but restores them to the subtasks as they were in the last job