Hey everyone. I am trying to find a way to deal wi...
# troubleshooting
t
Hey everyone. I am trying to find a way to deal with
KafkaSink
exceptions like
Caused by: org.apache.flink.util.FlinkRuntimeException: Failed to send data to Kafka <topic-name>--1@-1 with FlinkKafkaInternalProducer{transactionalId='null', inTransaction=false, closed=false}
which means that the topic doesn't exist. Is there a way I can add a callback/hook to the sink such that it get's called when the sink raises an exception? Or, what would be the best way to prevent a record reach the sink if the destination topic doesn't exists? Open to any idea. Thanks
p
Hey there Tudor! Does this error stops the pipeline? Or it just logs it? I do not know a way to do that with triggers. The way I'm thinking about this is that you have a CustomSink that either utilizes KafkaSink or extends it to add the required capability.
Or, what would be the best way to prevent a record reach the sink if the destination topic doesn't exists?
Do you mean having a mapping of available topics or such? I think a different exception would also happen when the sink has no permission to write to the destination topic.
t
Hey, about the permissions, that's valid, but mitigated already with a loose security policy. What I was thinking about is a process function that could take that mapping as an input and use Kafka admin client and query topic metadata. Not very nice, but with some caching and such, can do the work...
p
But do those errors stop your pipeline? Because if it doesnt I understand the concern of it. In the other hand, if it just crashes, it brings the same outcome as a process doing it, no?
t
It stops the pipeline, yes. The goal is to have it not crash if topic doesn't exists in Kafka.
p
Got it. So you'd be filtering out records are meant to be produced in a non existent topic
t
Exactly!
p
Thinking about the "best way" to achieve this I think abou a chain of: 1. Map (to have the topic already on the record to be sink 2. AsyncIO function to fetch if the topic exists (the caching here would need to be some static property because AsyncIO cannot rely on state) 3. Filter function to remove records that do not have the topic created 4. Sink Those steps could be hidden in a specific method/class and you offer it as a platform tool.