Hi team, I see BE node logs this `vacuum.cpp:139] Fail to delete: Internal error: starlet err [RequestID=][StatusCode=-1]Delete objects error: XML_ERROR_MISMATCHED_ELEMENT`` I think there is some issue when starrocks try delete file during compaction. Is there a way I verify this? StarRocks version: 3.3.18 Deploy shared-data on K8S + S3 (Ceph)

looks like a malformed xml response produced by Ceph.

Hi, I check metrics from S3 provider, there is no delete request on dashboard 🥲

ceph doesn't support batch delete s3 api?

My service provider say their service support batch delete s3 api

turn on aws sdk debug log by adding the following line in cn.conf, and restart the cn service

aws_sdk_logging_trace_enabled = true

it will generate a log file under the cn home directory, with filename pattern as

aws_sdk_<date>.log

, check the detailed info of the errors in this logfile, and possibly get to your SP with the detailed info.

Hi, here is the log from aws sdk

[TRACE] 2025-12-04 04:00:05.122 CurlHttpClient [139871321712192] HTTP/1.1 100 CONTINUE

[TRACE] 2025-12-04 04:00:05.122 CurlHttpClient [139871321712192]

[TRACE] 2025-12-04 04:00:05.122 CurlHttpClient [139871321712192] HTTP/1.1 200 OK

[TRACE] 2025-12-04 04:00:05.122 CurlHttpClient [139871321712192] transfer-encoding: chunked

[TRACE] 2025-12-04 04:00:05.123 CurlHttpClient [139871321712192] x-amz-request-id: tx00000dd62ec429d8207f1-0069310744-e639955c-sgn10-1

[TRACE] 2025-12-04 04:00:05.123 CurlHttpClient [139871321712192] content-type: application/xml

[TRACE] 2025-12-04 04:00:05.123 CurlHttpClient [139871321712192] date: Thu, 04 Dec 2025 04:00:05 GMT

[TRACE] 2025-12-04 04:00:05.123 CurlHttpClient [139871321712192]

[TRACE] 2025-12-04 04:00:05.123 CurlHttpClient [139871321712192] 100 bytes written to response.
[DEBUG] 2025-12-04 04:00:05.123 CurlHttpClient [139871321712192] Returned http response code 200
[DEBUG] 2025-12-04 04:00:05.123 CurlHttpClient [139871321712192] Returned content type application/xml
[DEBUG] 2025-12-04 04:00:05.123 CurlHttpClient [139871321712192] Releasing curl handle 0x7f3664124000
[DEBUG] 2025-12-04 04:00:05.123 CurlHandleContainer [139871321712192] Releasing curl handle 0x7f3664124000
[DEBUG] 2025-12-04 04:00:05.123 CurlHandleContainer [139871321712192] Notified waiting threads.
[DEBUG] 2025-12-04 04:00:05.123 AWSClient [139871321712192] Request returned successful response.
[TRACE] 2025-12-04 04:00:05.123 AWSClient [139871321712192] Request successful returning.
[ERROR] 2025-12-04 04:00:05.123 AWSXmlClient [139871321712192] Xml parsing for error failed with message XML_ERROR_MISMATCHED_ELEMENT

tx00000dd62ec429d8207f1-0069310744-e639955c-sgn10-1

have this request id to your sp, check what the response looks like. it says a

application/xml

but could be an invalid xml in the response body.