Hi everyone, I have a question regarding Flink sin...
# troubleshooting
s
Hi everyone, I have a question regarding Flink sinks and fault tolerance. My setup is: • Source: Kafka • Sinks: • Kafka Sink • S3 Sink Both sinks consume from the same stream. Goal: I want both sinks to operate independently, meaning: • If the S3 sink fails, the Kafka sink should continue writing. • If the Kafka sink fails, the S3 sink should continue writing. At the same time, I want to maintain exactly-once delivery guarantees. My questions are: Is it possible in Flink to configure multiple sinks so that they fail independently without causing the job to restart? Since Flink checkpointing is global across the job graph, how would state management and checkpointing behave in this case? Would appreciate any guidance Thanks!
a
I don't think so. Try two different job graphs. I.e. two instances of the same source where one connected is to the kafka sink, another to the s3 sink. Or alternatively two entirely separate jobs.
👍 1
s
i think you can explore side output with datastream api .
👍 1