Hi community, was wondering if I could get pointed in the right direction implementing what I thought was a pretty common pattern in streaming systems • We’re using Debezium to stream a few Postgres Tables to Kafka to compacted topics • We’re also streaming high volume IoT traffic to Kafka • Using Flink/Zeppelin for prototyping we’re successfully INNER JOIN our Debezium tables with IoT sensor data & seeing the data we want in Zeppelin. • Finally, we’re trying to write the joined/enriched output to another Kafka topic but we’re getting the error:

Table sink doesn't support consuming update and delete changes which is produced by node Join …

[lists my joined tables] I’m trying to diagnose this error. Is this because we’ve chosen compacted topics? Does the order of the JOIN matter in my query? Some other reason? I’ve implemented this pattern in Kafka Streams / KSQL in the past with great results so wondering what we’re doing wrong with Flink to make it work. Thanks so much for any tips!

Just the fact of using a JOIN converts the data stream to a changelog format internally (with updates and deletes). You can either use Upsert Kafka sink that supports changelog data or convert it to an append-only format.

convert it to an append-only format.

What is it in this case? My Kafka Sink topic / corresponding table? I want my output data to append-only so this sounds conceptually correct. A little web searching isn’t revealing how to do this. I did read this documentation and while it sounds like it might work around the error it doesn’t sound like what I want to do. I’d like my output data to be append-only https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/upsert-kafka/

You may want to check https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/dynamic_tables/ to start