Question on TwoPhaseCommittingSink. 1. Is there any open jira issue or existing requirement for supporting single parallelism for Committer, and preCommitTopology? My understanding is that existing Sink Comittter can only support same parallelism to that of Sink writer, and that there are additional checks in CommitterOperator/CommittableCollector for committables summary and committables received by each subtask commitable manager. I believe that single parallelism is potentially supported in postCommitTopology and that's how Global committer is translated for new sink. But we had requirement to have single parallelism before committer as the committer/postCommitTopology is invoked post snapshot/checkpoint.

My assumption would be that you should be able to include a PreCommit Topology since that's been added since https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction

Does the small file compaction combines files across multiple parallel subtasks of Sink Writer? Or only single parallel tasks across different checkpoints? I saw the FileSink compactor interfaces but let me check more on how does it implement internally.

Small file compaction is just one of the example use cases for pre and postcommit topologies (but arguably the most requested one)

Sure. Thanks, I will check that out. I was trying to have single parallelism in preCommitTopology but it needed reshuffle post Writer step. But at the same time, committer not accepting shuffled input records as the committables messages are reshuffled to different committer. Maybe, there is something I am missing and need to see how file compactor implements it internally.

Please do, else I can ask around for more insights