Hi all, I've tried to start using Flink for work m...
# troubleshooting
p
Hi all, I've tried to start using Flink for work multiple times in the last 2 years, but I keep getting stuck on stuff like getting a
DataStream[RowData]
from a connector in Scala, but having no idea how to use it. πŸ˜• I enjoyed the rockthejvm-course a lot, so I'm very motivated like to figure this out! Previously I've dabbled in Debezium & kafka as input, but my current input is a delta lake table, the connector doesn't support the table api, so I guess I either need to turn the stream into a
DataStream[Row]
to do enable using
tEnv.fromChangelogstream()
or figure out how to specify schema with
tEnv.fromDataStream()
, but how? Can I avoid specifying schema when the delta lake connector supposedly already knows what the schema is? I also wanted to turn
DataStream[RowData]
into
DataStream[MyCaseClass]
somehow, hopefully without manually pointing out and casting field-by-field in a
ds.map()
I've tried is using
CaseClassConverter
and
RowRowConverter
, but I couldn't really understand how to use either of them πŸ˜– Am I missing something? Am I failing to progress because I'm trying too hard to avoid retyping the input schema multiple times? The
DataStream API Integration
docs mention
RowData
->
Row
conversion, but I haven't been able to find any examples πŸ˜• Is using Scala + Case Classes making it worse? I can switch to using the third party scala api implementation if that helps, or switch to java or kotlin
g
RowData is internal class, as a user, we shouldn't use it
p
I've read this as well, RowData being internal and Row being the preferred alternative, it's why I want to turn the stream into
DataStream[Row]
:s I'm following the delta connector example and receive RowData: https://github.com/delta-io/connectors/tree/master/flink#6-source-creation-for-delta-table-to-read-on[…]-table-version-and-then-monitors-for-changes
i
@Peter Esselius I have spent a good part of last two days trying to figure out the problem. Most of the examples I see don't cater to this scenario - I usually get answers like above "it's an internal class don't use it" which are not helpful. If you have solved the probelm, I'd love to connect.
p
Will do :)