Hi guys, I'm working on a statefull pipeline and I have some issues. I want to read three stream from kafka topics and join them for creating a flatten table but kafka topics fill with CDC data so all records may have updates and I want to have last version. I tried use interval join but I can not keep events for long time(huge states) is there better solution for this scenario? I was thinking about store last state of records in external cach db like redis. could you please share your ideas?

I had a similar issue once. What I did was to change my CDC to send the full record instead of the delta record. I achieved this changing the replica identity to full. This way, state was not a concern anymore.