Hey all, I don’t know if this has been discussed y...
# fluent-bit
m
Hey all, I don’t know if this has been discussed yet. We have had many records dropped without any retries happening from investigating the logs we concluded the following : • looks like the issue is coming from a client connection reset with s3. The KA (keep-alive) tcp connection is re-establishing the connection when the connection is broken, a putrequest is attempted and we gett a status 0 error, which means it isn't retrying via the the retry mechanism that fluent_bit metrics tracks. • I believe due to the empty response of the S3 fluent bit seems the records unrecoverable and gives up and drops them without retrying. Is there a way to force fluent bit to retry these kinds of errors? I tried tuning the networking plugin parameters but no luck. Is this a known issue or am I just doing something wrong
a
@Moulid Adan, how do you know logs are being dropped?
m
I look at the
fluentbit_output_retried_records_total
and there is no retries ever happening. I also looked at the logs. I would assume these are being dropped. We also have a way of testing gaps within our records that were outputted to S3. There have been many gaps. Another data point is the
fluentbit_output_dropped_records_total
metric. This metric may not be fully trusted but I looked at it as well.
a
this thread should be helpful: https://fluent-all.slack.com/archives/C0CTQGHKJ/p1749119044940979 TLDR is , fluentbit_output_* metrics are useless for S3 output
you can go through the code here: https://github.com/fluent/fluent-bit/blob/master/plugins/out_s3/s3.c and you'll be able to find the actual scenario where logs are really lost. You can then create a log based metric on those logs when fluentbit actually drops logs. Thats what i'm doing atleast.
m
Got it, do you know if this is true for any other plugin? I want to now use kinesis and then write to S3. We can't have many log loses
a
I'm not sure, will need to read the code. But why do you want to use kinesis? IMO s3 output should be good enough. There is no actual log loss. its just the metrics that are wrong.
the only real log loss scenario is when s3 goes down for a while.
m
I am seeing gaps in my log timestamps. We emit logs every 10s and there has been 60 sec gaps or more sometimes. How could that not be log loss
a
I had run some scale tests on fluentbit. I wrote ~200M logs in about 2 hours, and didnt lose any logs. Can you double check the application code?
you can also use the fluentbit_input_* metrics to see if any logs actually were sent to fluentbit or not at the time when you expect logs to have been sent
you can also check if the logs corresponding to the log loss scenarios here: https://github.com/fluent/fluent-bit/blob/master/plugins/out_s3/s3.c were printed during the time when you expect your logs
m
The screen shot above is always the errors we get. it is always broken connection to s3. Here is the record count for input
a
Its very hard to give any answer with this. If you can come up with a repro-able config and setup that will be useful. I have verified in a test setup that these connection timeouts don't lead to log loss. Code wise, this is where the debug log youre seeing comes from: https://github.com/fluent/fluent-bit/blob/3b7dbbb82bb33c3f4b8407329bc4b38c62599776/plugins/out_s3/s3.c#L1748 it schedules a retry for a specific time in future, The
s3_upload_queue
function picks up the file from the queue again.
m
When it retries in the future fluent bit logs that a retry happend? I don't see any retries after the connection timeouts or broken connections. I have debug logs enabled.
a
No it doesn't. it logs on success, or logs when its going to drop the log. You can check the code out.
IMO if you're worried, just add a debug line in timer callback that retries files and build fluentbit from source and use that.