Hi, I am testing a compaction functionality in Fil...
# troubleshooting
g
Hi, I am testing a compaction functionality in FileSink https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/filesystem/#compaction I have enabled it on FileSink like this:
Copy code
FileCompactStrategy.Builder.newBuilder()
                            .enableCompactionOnCheckpoint(numCheckpointsBeforeCompaction)
                            .build(),
                    new RecordWiseFileCompactor<>(new RecordWiseFileCompactorSpecificAvroReaderFactory<>(Record.class)
But what I noticed that betwen Writer and a Commiter it adds CompactorCoordinator with parallelism of only 1. Does it have to be the case that this won't scale? I have tested setting parallelism explicit on the sink and on the function before the sink. I have tried also using
.setNumCompactThreads(24)
thinking that maybe it somehow affects parallelism but no luck.
Nevermind. I have found that in the code it is explicit set to 1. So I understand the actual writing happens in the Writing operator that have parallelism of 24 but where happens the compactification? In the coordinator with parallelism of 1?
πŸ‘€ 1