I was hoping to create a custom table connector th...
# troubleshootinga
Adam Augusta
03/30/2023, 10:01 PMI was hoping to create a custom table connector that would allow me to stream field updates.
But it seems like Flink requires the full row, even if only a single field has changed. Is that correct?
m
Martijn Visser
03/31/2023, 11:43 AMYes, that's correct
Martijn Visser
03/31/2023, 11:44 AMThere will be https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081 in Flink 1.18
a
Adam Augusta
03/31/2023, 2:49 PMThat’s brilliant! Thank you.
Adam Augusta
03/31/2023, 2:53 PMAlthough I wonder if this FLIP is comprehensive.
Depending on the operator topology, a partial update may or may not require more state within Flink, in which case requiring the source to do the extra work to populate the some subset of columns could be more performant. Seems like a planner optimization extension?
Adam Augusta
03/31/2023, 3:07 PMOh, I’m being dumb. FLIP-300 deals exclusively with sinks.
m
Martijn Visser
03/31/2023, 3:07 PMYeah that's what I assumed you needed 😅
a
Adam Augusta
03/31/2023, 3:08 PMSorry, yeah, still learning how to think/communicate in the Flink frame. 😅