Sami Airaksinen
03/13/2025, 10:47 AM2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - flink-internal/: "ERROR_INTERNAL" - ERROR_INTERNAL
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - flink-internal/production/: "ERROR_INTERNAL" - ERROR_INTERNAL
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - flink-internal/production/ha-recovery/: "ERROR_INTERNAL" - ERROR_INTERNAL
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - flink-internal/production/ha-recovery/production/: "ERROR_INTERNAL" - ERROR_INTERNAL
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.statistics.impl.IOStatisticsStoreImpl [] - Incrementing counter files_delete_rejected by 4 with final value 16
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - Deleting 4 keys: duration 2:24.768s
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - Partial delete failure
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.statistics.impl.IOStatisticsStoreImpl [] - Incrementing counter ignored_errors by 1 with final value 32
2025-03-13 12:38:35,757 DEBUG org.apache.hadoop.fs.s3a.S3AFileSystem [] - While deleting keys flink-internal/production/ha-recovery/production/,flink-internal/production/ha-recovery/,flink-internal/production/,flink-internal/,
com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects could not be deleted (Service: null; Status Code: 200; Error Code: null; Request ID: xxx; S3 Extended Request ID: xxx; Proxy: null)
My question is that why does it try to remove all levels of the directory structure (i.e. separately flink-internal/ , flink-internal/production/ , flink-internal/production/ha-recovery/ and ``flink-internal/production/ha-recovery/production/` ?Sami Airaksinen
03/17/2025, 10:19 AMflink-internal/ path and that caused this very odd situation with our HA. This prequisite/behaviour seems to be changed between 1.19 and 1.20