Hello, everyone, I am testing offloading using the lunastreaming 2.10_4.8 release and following the instructions here: https://pulsar.apache.org/docs/next/tiered-storage-s3/ more or less. I've set up the bucket and a user with RW perms. In my helm chart, the broker is set to the following (values.yaml):

storageOffload:
  driver: "aws-s3"
  bucket: "<my bucket>"
  region: "<my region"
  accessKey: "<user's key>"
  accessSecret: "<user's secret>"
  maxBlockSizeInBytes: "64000000"
  readBufferSizeInBytes: "1000000"
  managedLedgerOffloadDeletionLagMs: "14400000"
  managedLedgerOffloadAutoTriggerSizeThresholdBytes: "-1" # disabled

But I created a namespace with threshold and retention set:

admin namespaces set-retention tenant/testoffload --size 1G --time 3d
admin namespaces set-offload-threshold --size 10M tenant/testoffload

However, when I use pulsar-perf to produce to the topic, nothing happens...

./pulsar-perf produce -r 1000 -s 2048 <persistent://tenant/testoffload/test-topic>
2023-08-24T04:54:51,993+0000 [main] INFO  org.apache.pulsar.testclient.PerformanceProducer - Throughput produced:   99604 msg ---   1000.0 msg/s ---     15.6 Mbit/s  --- failure      0.0 msg/s --- Latency: mean:   3.884 ms - med:   3.747 - 95pct:   4.931 - 99pct:   9.597 - 99.9pct:  19.534 - 99.99pct:  21.662 - Max:  21.718
// etc...

Messages are going, but...

admin topics stats-internal <persistent://tenant/testoffload/test-topic>
{
  "entriesAddedCounter" : 102492,
  "numberOfEntries" : 102492,
  "totalSize" : 230582632,
  "currentLedgerEntries" : 102492,
  "currentLedgerSize" : 230582632, // 230.582632MB, WAY above threshold
  "lastLedgerCreatedTimestamp" : "2023-08-24T04:53:12.359Z",
  "waitingCursorsCount" : 0,
  "pendingAddEntriesCount" : 0,
  "lastConfirmedEntry" : "1178328:102491",
  "state" : "LedgerOpened",
  "ledgers" : [ {
    "ledgerId" : 1178328,
    "entries" : 0,
    "size" : 0,
    "offloaded" : false,
    "underReplicated" : false
  } ],
  "cursors" : { },
  "schemaLedgers" : [ ],
  "compactedLedger" : {
    "ledgerId" : -1,
    "entries" : -1,
    "size" : -1,
    "offloaded" : false,
    "underReplicated" : false
  }
}

Attempting to manually trigger offload does nothing, despite appearing to succeed.

admin topics offload --size-threshold 1M tenant/testoffload/test-topic
Offload triggered for <persistent://tenant/testoffload/test-topic> for messages before 1178328:0:-1"
admin topics offload-status <persistent://tenant/testoffload/test-topic>
Offload was a success

But the S3 bucket is empty. This is driving me nuts, what am I missing here?