Hey everyone, need some help with FileSink, S3 and compaction. I've got the following FileSink configured:

FileSink.forRowFormat(new Path(outputBasePath),
                    new Encoder<Object>() {
                          @Override
                          public void encode(Object o, OutputStream outputStream) throws IOException {
                            outputStream.write(mapper.writeValueAsBytes(o));
                            outputStream.write(newline);
                          }
                    })
        .withBucketAssigner(new CustomerBucketAssigner<>(subPath))
        .withOutputFileConfig(config)
        .withRollingPolicy(OnCheckpointRollingPolicy.build())
        .enableCompact(FileCompactStrategy.Builder.newBuilder()
                .enableCompactionOnCheckpoint(1)
                .setSizeThreshold(-1)
                .setNumCompactThreads(10)
                .build(), new GzipFileCompactor())
        .build();
  }

TL;DR: • On each checkpoint it moves a part file from in-progress to finished. These files are uncompressed, thus large in size. • Once in 10 checkpoints, it takes the 10 finished part files and compacts them into one gzip compressed, small in size. The outputBasePath is set to s3://bucket. As a result, the large part files are being uploaded to S3.. only to be re-downloaded for compaction. I wish for this not to happen because it's redundant. I think the end result should be that the part files would be stored on the hdfs, and only the compacted files uploaded to s3. One solution I came up with is to use s3distcp. But is this somehow achievable within Flink and the FileSink?