Ashish
06/05/2025, 10:24 AMThe Prometheus success/retry/error metrics values output by the built-in HTTP server in Fluent Bit are meaningless for S3 output. S3 has its own buffering and retry mechanisms. The Fluent Bit AWS S3 maintainers apologize for this feature gap; you can track issue progress on GitHub.
And there's this open github ticket: https://github.com/fluent/fluent-bit/issues/6141 that talks about the gap in monitoring.
In my monitoring setup for fluentbit, i noticed fluenbit_output_dropped_record metric being sent (attaching screenshots) and the time coincides with the time of these logs:
SrBssuo_Xfn8G.7RUOovC0QRRbhLtcAPJ7XxhOxcqbeQKz2BzrnvPGhFO5S._x5gl4Dvq_jL4W3QPXEHNSs9wPf6xlyZ3Cr5eLna_6S9ECEhvmnOR29fT8Agw--
[2025/06/05 08:17:48] [error] [http_client] broken connection to s3.us-east-1.amazonaws.com:443 ?
[2025/06/05 08:17:48] [ info] [output:s3:s3.0] UploadPart http status=0
[2025/06/05 08:17:48] [error] [output:s3:s3.0] UploadPart: Could not parse response
[2025/06/05 08:17:48] [error] [output:s3:s3.0] UploadPart request failed
[2025/06/05 08:18:04] [ info] [output:s3:s3.0] UploadPart http status=200
[2025/06/05 08:18:04] [ info] [output:s3:s3.0] Successfully uploaded part #2 for /document_life_cycle/2025/06/05/08/13:34-7l7AOj6m.log, UploadId=NJ1eR_YQBjHwT1Rsye4wwaeAPPr4Rw5mQJHjtJjNE08_Mll1SuGvImIiY_BAqrK3qVPb0scMLMGRX9.I_fzU5ZyK2S2_0bDETkCuo_u2lnHndan1pQe_p9LE83v0e.hrRa4ewCGFfXfGt38We728sg--, ETag=4bae58e3cdd876f1e406df4ca11d9518
The above github open ticket mentiones:
Thus, it always returns FLB_OK for all flushes (unless there was an error writing to the buffer files).which would mean that the metric gets incremented for S3 output when there was an error writing buffer to files on disk. But the timing coinciding with the UploadPart error makes me think otherwise. There is no metric sent for output_retries. There are no error logs that indicate that the part is not retryable. Does anyone have an idea on how these metrics behave for S3 output?
Ashish
06/05/2025, 10:24 AMAshish
06/05/2025, 11:51 AMAshish
06/05/2025, 1:54 PMs3_upload_queue(config, ctx);
if (ctx->upload_queue_success == FLB_FALSE) {
ctx->upload_queue_success = FLB_TRUE;
FLB_OUTPUT_RETURN(FLB_ERROR);
}
Note that by this time, the chunk is already buffered in the file, so should be on disk.
looking at s3_upload_queue:
it has this code:
https://github.com/fluent/fluent-bit/blob/a8f3f507613883f8224af26a3d5d14f6b18b684f/plugins/out_s3/s3.c#L1719
ret = send_upload_request(ctx, NULL, upload_contents->upload_file,
upload_contents->m_upload_file,
upload_contents->tag, upload_contents->tag_len);
if (ret < 0) {
goto exit;
}
else if (ret == FLB_OK) {
remove_from_queue(upload_contents);
ctx->retry_time = 0;
ctx->upload_queue_success = FLB_TRUE;
}
else {
s3_store_file_lock(upload_contents->upload_file);
ctx->upload_queue_success = FLB_FALSE;
So when the upload request fails, it returns -1 / FLB_FALSE based on the error type. and returns, which leads to an FLB_ERROR.
This FLB_ERROR sent back from the FLUSH leads to the metric being updated, but the actual chunk is already stored in disk, so there wont be any logs droppedAshish
06/05/2025, 1:57 PMAshish
06/05/2025, 3:13 PM