Madhusudhan Reddy
08/11/2025, 1:42 PMSELECT
t1.ID,
LISTAGG(
CONCAT_WS(
CHR(31), -- Unit Separator
t1.c1,
t1.c2
),
'\u001E' -- Record Separator
) AS t1_concat
FROM table1 t1
JOIN table2 t2 ON t1.ID = t2.ID
GROUP BY t1.ID
Issue:
This intermediate emission leads to duplicate events — one partial and one final — whereas only the final consolidated state should be emitted.
Question:
How can I make sure that I emit only the latest event that have change instead of emitting
1) Event that doesn't have the row that I am updating
2) Event that have the row that i am updating
Update on one of the row in Flink Table- Operation AGGLIST, Join and GroupBY we see ROWKIND -U +U -U +U
How can I filter for only the last +U and emit event?
Please let me know your view
Here is the code
StreamTableEnvironment streamTableEnvironment = StreamTableEnvironment.create(streamExecutionEnvironment);
FlinkConfigurationManager flinkConfigurationManager = new FlinkConfigurationManager(streamExecutionEnvironment);
// Run the query that provides publishable data
Table viewsAggregateQueryResult = streamTableEnvironment.sqlQuery(sqlQuery);
// Process the result rows
DataStream<JsonNode> eventDataStream = streamTableEnvironment
.toChangelogStream(viewsAggregateQueryResult)
.filter(new UpdateBeforeFilter())
.uid(PROC_NAME + "-update-before-filter-" + "-v1")
.name("update before filter:") // filter out -U Rowkind
.map(new RowToJsonNodeMapper(mappings.toString()));
// Output the data after transformation to
eventDataStream
.map(new EventTransformerMapFunction(
configs.getKafkaSchemaHeaders()))
.keyBy(EventData::getKey)
.sinkTo(new KafkaSinkWriter(
getOutboundKafkaSink(configs)));
streamExecutionEnvironment.execute();David Anderson
08/13/2025, 8:44 PMMadhusudhan Reddy
08/13/2025, 9:10 PMMadhusudhan Reddy
08/20/2025, 10:08 PM