Hi all I am facing with this following issue that ...
# fluent-bit
h
Hi all I am facing with this following issue that FluentBit cannot handle natively so I am looking for advises. Indeed I am using the OpenSearch output and when there are error in the messages ingestion, OS rejects them with a message error then FluentBit does the retry .. and it does not resolve the issue .. Ideally FluentBit could detect that the error is not a connection technical error (so retry does not help here) but more a functional error so put the malformed messages in a DLQ and keeps continuing with the others messages (instead of retrying the same blocking messages).. So what could I do here?
e
do you have a JSON example for the input that we can use to repro the issue ?
h
hi @eduardo Sorry I don't really have something "shareable" here but this is what users could say to us
Copy code
to reproduce the issue, you need to send structured logs with 2 conflicting fields.
So send
{"msg": "anything, "level": "INFO"}
and after
{"msg": "anything else", "level": 30}
Opensearch rejects the logs and you never see the second log anywhere.
The fluentbit logs will tell you the log id is something but will never show you the content of the log.
here the logs are coming from different applications from other teams so they don't have the control on the logs format
e

https://www.youtube.com/watch?v=vqM3DCbevMs▾

h
thanks @eduardo .. this helps me for the other issue 🙂 but here it's not related to the parsing issue. In fact there are different applications (versions) that generate different format of logs (for instance the field level could be INFO or a number in order versions like in the example shared by the user) and OpenSearch rejects those messages as the mappings are not correct -- the rolling upgrade the apps is in progress but could take long time. That's why the DLQ could be helpful to identify those messages or at least Fluentbit could log them (something for identifiable than the internal id