Peter Esselius
04/19/2023, 1:31 PMDataStream[RowData] from a connector in Scala, but having no idea how to use it. π
I enjoyed the rockthejvm-course a lot, so I'm very motivated like to figure this out!
Previously I've dabbled in Debezium & kafka as input, but my current input is a delta lake table, the connector doesn't support the table api, so I guess I either need to turn the stream into a DataStream[Row] to do enable using tEnv.fromChangelogstream() or figure out how to specify schema with tEnv.fromDataStream() , but how?
Can I avoid specifying schema when the delta lake connector supposedly already knows what the schema is?
I also wanted to turn DataStream[RowData] into DataStream[MyCaseClass] somehow, hopefully without manually pointing out and casting field-by-field in a ds.map()
I've tried is using CaseClassConverter and RowRowConverter , but I couldn't really understand how to use either of them π
Am I missing something? Am I failing to progress because I'm trying too hard to avoid retyping the input schema multiple times?
The DataStream API Integration docs mention RowData -> Row conversion, but I haven't been able to find any examples π
Is using Scala + Case Classes making it worse? I can switch to using the third party scala api implementation if that helps, or switch to java or kotlinGaurav Miglani
04/19/2023, 2:13 PMPeter Esselius
04/19/2023, 3:30 PMDataStream[Row] :s
I'm following the delta connector example and receive RowData: https://github.com/delta-io/connectors/tree/master/flink#6-source-creation-for-delta-table-to-read-on[β¦]-table-version-and-then-monitors-for-changesIshan
05/23/2023, 12:00 AMPeter Esselius
06/02/2023, 2:53 PM