Hey everyone, I'm currently using DBT + Redshift for data processing, and I'd like to understand whether Flink can help me to achieve better performance (I get data into redshift from kinesis stream). So if I'm using dbt which means 100% sql for my processing, does Flink provides advantages? isn't the SQL just pushed down to redshift? I'd appreciate any advice 🙏

Flink would potentially replace the processing that's done in Redshift. Instead of Kinesis -> Redshift with processing as part of the load (ETL) or after the load (ELT), you would read from Kinesis into Flink, transform your data, and then write it to Redshift. Flink provides the potential advantage of lower latencies in your data processing as you can process your data as a stream, not batched. Performance is extremely subjective - it depends where your current bottlenecks are. Depending on your data patterns though, processing it as a stream can help with performance as you smooth out the consumption and processing instead of it all bunching up and needing processing at once (i.e. batch).

Thank you 🙌 I plan to utilize the stream/batch unified API in order to work with streams most of the time, but when I have some changes to my processing logic, I would like to reprocess old data. In batch mode does Flink also provides advantages?

that's a good question - I've not dug into it to that extent to be able to answer, sorry 🙂

thanks anyway 🙂