This message was deleted.

Not corrupted. Are you talking about streaming or batch? In streaming, partition offsets are updated only when a segment has been successfully published. If the task fails before that, the supervisor starts the task again and it picks up consumption from the last successful partition offsets. The only caveat is that the retention in kafka must be enough to store the messages between the time of the failure and when it continues to re-consume. For higher availability you can use task replicas. Which means there will be pairs of tasks that manage the same kafka partitions such that if one in the pair fails, the other one just continues.

Thank you very much