Hello everyone, I’m considering forking and modif...
# troubleshooting
n
Hello everyone, I’m considering forking and modifying Flink’s KafkaSource so that it can, if needed, read a starting offset for each topic partition from an external state store and override the offset that would normally be chosen based on checkpointing. In other words, I want to force KafkaSource to start from specific offsets I provide, even though checkpoints usually dictate where the source resumes. • Is it feasible to achieve this by customizing components like KafkaSourceEnumerator, KafkaSourceReader, or KafkaSourceSplitReader? • Or do Flink’s source mechanics make this approach impossible, regardless of how much the KafkaSource code is modified? Any insights on whether this kind of override is supported (or fundamentally blocked) by the current KafkaSource architecture would be greatly appreciated. --- UPDATE --- I do not want to use startingOffsets, as I want to achieve that in runtime, not again through DAG recreation (job resubmission, restart, etc.)
l
What about using the kafkasource starting offsets using topic partition offsets option, and having tour own library your app uses to load the offsets from your external state?
n
yes i checked that, but i want to change on runtime, startingoffsets is in DAG creation, so i will need to restart the job, etc.
a
I have no clue whether what you want to achieve is possible, or not. I've never used the KafkaSource. However, in terms of Source architecture I've had good luck with using a decorator pattern to augment existing Source and Sink implementations instead of having to resort to forking them, in order to change behavior and add features. A fork comes with a lot of extra work to keep up-to-date with original implementation (bug fixes, features, flink version compatibility etc.)
🙏 1