I currently have a (plain) Kafka consumer that consumes from a dynamic list of topics (every minute the list is being recalculated, usually it's the same list). Can I do the same with Flink? And how? (Flink to consume from a dynamic list of kafka topics). Every minute there's some method that gets all the topics by some logic (topics can be added, and removed in this logic, but usually it's the same).

Haven't tested it, but I wonder if you could use a combination of topic-partition subscription and dynamic partition discovery. See https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/#topic-partition-subscription and further downstream

@Martijn Visser Thanks! How to change the topics list mid-job? When initializing the job, the topics are set in the

KafkaSource

, but how to change this list mid-job so the dynamic partitions discovery will kick in?