Hello! I have a question about best practices in a ETL service with flink:
• I created a rabbitmq consumer, that deserializes a json and returns a customs Deserialized class, then I need to store this class using a sink, very simple flow.
• My doubt is , how to handle correctly escenarios where the consumed message is inconsistent, for example, instead of receive a json, a plain text was consumed, so it can generate an exception because of deserialization error.
• Rabbitmq includes some mechanism like deadletterqueue, where after some retries the message is moved to a different queue automatically to prevent eternal loops, but this is how flink has to treat the data? or I have to use another mechanism to handle exceptions and always go forward with the pipeline like sideoutputs?
• Probably the same doubt can match with kafka or pulsar sources
Thanks for your suggestions