👋 Does anyone have experience with handling large messages in Kafka with Flink? We've got some messages that are greater than our Kafka message size limit and we want to sink + subsequently read from a source. Currently we are thinking of storing these messages out of band in object storage, and storing a reference to object storage in the message. Interested in implementing this in a DynamicTableSource/Sink, thinking about custom Format Factories + SerializationSchema + DeserializationSchema. One thing that comes to mind is ideally we upload to object storage async -- any tips or gotchas we should keep in mind?

Interesting problem! Do you need

Exactly_Once

guarantees?

Not in this case