Hey all,
we're trying to wrap our heads around the DLQ handling in SQL in case the de-serialization fails. As far as we've seen there is no side-output etc. using SQL. We use the kafka connector which just throws an error if it fails to de-serialize the message which causes a job stop. This stop would be OK but re-producing/ consuming all messages would take too long in certain cases and our SLA wouldn't be met.
We're using the kafka connector + exactly-once semantics + rocksdb as the state backend.
Would there be a way for example to skip an offset or do you guys use different approaches to this?
We also though of adding a proxy before kafka so that we can't even produce wrong messages but that will take longer to be set up and used by the whole company.
Thank you π