Hello Everyone, We are using

GCS offloader

in pulsar

3.1.0

and frequently get the following gcs/jclouds offload error...

Error in offload
null

Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: org.jclouds.http.HttpResponseException: command: POST <https://www.googleapis.com/storage/v1/b/offload-bucket/o/70131234-d36c-4fcb-98e0-90397013ac3f7-ledger-3124/compose> HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
  "error": {
    "code": 400,
    "message": "The number of source components provided (40) exceeds the maximum (32)",
    "errors": [
      {
        "message": "The number of source components provided (40) exceeds the maximum (32)",
        "domain": "global",
        "reason": "invalid"
      }
    ]
  }
}
]

It seems that a combination of increasing topic offload size threshold

pulsar-admin topics offload --size-threshold 50M

and adjusting

gcsManagedLedgerOffloadMaxBlockSizeInBytes

helps but this does not always work to prevent the offload failure. Are there any other parameters that we are missing that would affect this aspect of offloading? Is this expected behavior for GCS offloader? Any advice is greatly appreciated, thanks!