Ashish
06/18/2025, 10:42 AM[INPUT]
Name http
port 8080
buffer_max_size 150M
buffer_chunk_size 50M
mem_buf_limit 150M
storage.type filesystem
[OUTPUT]
Name s3
Match *
bucket ${var.s3_logs_bucket}
region ${var.region}
json_date_key timestamp
json_date_format iso8601
total_file_size 150M
upload_timeout 300s
upload_chunk_size 50M
s3_key_format /$TAG/%Y/%m/%d/%H/%M:%S-$UUID.log
retry_limit 5
store_dir /var/fluent-bit/s3
As visible, my upload_chunk_size is 50MB, and total file size is 150MB, Somehow my fluentbit deployment created 6 parts, (while it should only atmax create 3/4 based on the total size and chunk size).. And after that CompleteMultipartUpload started failing with error:
CompleteMultipartUpload API responded with error='EntityTooSmall', message='Your proposed upload is smaller than the minimum allowed size'
and eventually gave up. Here's a paste of the complete logs.
Can someone help take a look here? there seems to be some bug in the multipart upload logic, and since multipart upload is the default for s3 output, this would be leading to log loss across users.