Hi, we are using Flink 1.20.1 in HA mode with on-p...
# troubleshooting
s
Hi, we are using Flink 1.20.1 in HA mode with on-premise S3 as FS. We noticed that we are getting MultiObjectDeleteExceptions nowadays, with following debug logs
Copy code
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem                       [] - flink-internal/: "ERROR_INTERNAL" - ERROR_INTERNAL
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem                       [] - flink-internal/production/: "ERROR_INTERNAL" - ERROR_INTERNAL
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem                       [] - flink-internal/production/ha-recovery/: "ERROR_INTERNAL" - ERROR_INTERNAL
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem                       [] - flink-internal/production/ha-recovery/production/: "ERROR_INTERNAL" - ERROR_INTERNAL
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.statistics.impl.IOStatisticsStoreImpl   [] - Incrementing counter files_delete_rejected by 4 with final value 16
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem                       [] - Deleting 4 keys: duration 2:24.768s
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem                       [] - Partial delete failure
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.statistics.impl.IOStatisticsStoreImpl   [] - Incrementing counter ignored_errors by 1 with final value 32
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem                       [] - While deleting keys flink-internal/production/ha-recovery/production/,flink-internal/production/ha-recovery/,flink-internal/production/,flink-internal/, 
com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: xxx; S3 Extended Request ID: xxx; Proxy: null)
My question is that why does it try to remove all levels of the directory structure (i.e. separately
flink-internal/
,
flink-internal/production/
,
flink-internal/production/ha-recovery/
and ``flink-internal/production/ha-recovery/production/` ?
Root cause was found. Flink is expecting that the Flink process solely controls the path from the root as it recursively deletes it. We had other stuff also in our
flink-internal/
path and that caused this very odd situation with our HA. This prequisite/behaviour seems to be changed between 1.19 and 1.20