Hello everyone,
I’m considering forking and modifying Flink’s KafkaSource so that it can, if needed, read a starting offset for each topic partition from an external state store and override the offset that would normally be chosen based on checkpointing.
In other words, I want to force KafkaSource to start from specific offsets I provide, even though checkpoints usually dictate where the source resumes.
• Is it feasible to achieve this by customizing components like KafkaSourceEnumerator, KafkaSourceReader, or KafkaSourceSplitReader?
• Or do Flink’s source mechanics make this approach impossible, regardless of how much the KafkaSource code is modified?
Any insights on whether this kind of override is supported (or fundamentally blocked) by the current KafkaSource architecture would be greatly appreciated.
--- UPDATE ---
I do not want to use startingOffsets, as I want to achieve that in runtime, not again through DAG recreation (job resubmission, restart, etc.)