Hi Team, I wanted to know what's the current state...
# fluent-bit
a
Hi Team, I wanted to know what's the current state of fluentbit output metrics integration with S3 Output? The fluentbit S3 page has this caution:
Copy code
The Prometheus success/retry/error metrics values output by the built-in HTTP server in Fluent Bit are meaningless for S3 output. S3 has its own buffering and retry mechanisms. The Fluent Bit AWS S3 maintainers apologize for this feature gap; you can track issue progress on GitHub.
And there's this open github ticket: https://github.com/fluent/fluent-bit/issues/6141 that talks about the gap in monitoring. In my monitoring setup for fluentbit, i noticed fluenbit_output_dropped_record metric being sent (attaching screenshots) and the time coincides with the time of these logs:
Copy code
SrBssuo_Xfn8G.7RUOovC0QRRbhLtcAPJ7XxhOxcqbeQKz2BzrnvPGhFO5S._x5gl4Dvq_jL4W3QPXEHNSs9wPf6xlyZ3Cr5eLna_6S9ECEhvmnOR29fT8Agw--
[2025/06/05 08:17:48] [error] [http_client] broken connection to s3.us-east-1.amazonaws.com:443 ?
[2025/06/05 08:17:48] [ info] [output:s3:s3.0] UploadPart http status=0
[2025/06/05 08:17:48] [error] [output:s3:s3.0] UploadPart: Could not parse response
[2025/06/05 08:17:48] [error] [output:s3:s3.0] UploadPart request failed
[2025/06/05 08:18:04] [ info] [output:s3:s3.0] UploadPart http status=200
[2025/06/05 08:18:04] [ info] [output:s3:s3.0] Successfully uploaded part #2 for /document_life_cycle/2025/06/05/08/13:34-7l7AOj6m.log, UploadId=NJ1eR_YQBjHwT1Rsye4wwaeAPPr4Rw5mQJHjtJjNE08_Mll1SuGvImIiY_BAqrK3qVPb0scMLMGRX9.I_fzU5ZyK2S2_0bDETkCuo_u2lnHndan1pQe_p9LE83v0e.hrRa4ewCGFfXfGt38We728sg--, ETag=4bae58e3cdd876f1e406df4ca11d9518
The above github open ticket mentiones:
Thus, it always returns FLB_OK for all flushes (unless there was an error writing to the buffer files).
which would mean that the metric gets incremented for S3 output when there was an error writing buffer to files on disk. But the timing coinciding with the UploadPart error makes me think otherwise. There is no metric sent for output_retries. There are no error logs that indicate that the part is not retryable. Does anyone have an idea on how these metrics behave for S3 output?
👀 1
cc: @Wesley Pettit who created the github ticket
I now have this being fired many more times, and every time its fired, the time for metric is around the same time as UploadPart error
I think i figured out the code that causes this, would be great if someone who has knowledge can confirm: This is the piece of code that returns FLB_ERROR: https://github.com/fluent/fluent-bit/blob/a8f3f507613883f8224af26a3d5d14f6b18b684f/plugins/out_s3/s3.c#L2223
Copy code
s3_upload_queue(config, ctx);
            if (ctx->upload_queue_success == FLB_FALSE) {
                ctx->upload_queue_success = FLB_TRUE;
                FLB_OUTPUT_RETURN(FLB_ERROR);
            }
Note that by this time, the chunk is already buffered in the file, so should be on disk. looking at s3_upload_queue: it has this code: https://github.com/fluent/fluent-bit/blob/a8f3f507613883f8224af26a3d5d14f6b18b684f/plugins/out_s3/s3.c#L1719
Copy code
ret = send_upload_request(ctx, NULL, upload_contents->upload_file,
                                  upload_contents->m_upload_file,
                                  upload_contents->tag, upload_contents->tag_len);
        if (ret < 0) {
            goto exit;
        }
        else if (ret == FLB_OK) {
            remove_from_queue(upload_contents);
            ctx->retry_time = 0;
            ctx->upload_queue_success = FLB_TRUE;
        }
        else {
            s3_store_file_lock(upload_contents->upload_file);
            ctx->upload_queue_success = FLB_FALSE;
So when the upload request fails, it returns -1 / FLB_FALSE based on the error type. and returns, which leads to an FLB_ERROR. This FLB_ERROR sent back from the FLUSH leads to the metric being updated, but the actual chunk is already stored in disk, so there wont be any logs dropped
I guess we don't want to return FLB_RETRY because we dont want the engine to retry because the chunk has been successfully stored. So we return FLB_ERROR, but that in turn causes the metric to be counted even though no logs are dropped.
From reading the code, its a bug, created this issue to track it if someone gets to fixing it: https://github.com/fluent/fluent-bit/issues/10440 i'm wondering if just returning FLB_OK instead of FLB_ERROR would work.. that should fix the metrics showing false errors alteast (the metrics will show nothing, which is better than showing error). But I'm not sure if there would be other consequences, if someone can comment on this, would be helpful, I can make the fix.