Hello! I'm writing a flink process that writes to...
# troubleshooting
e
Hello! I'm writing a flink process that writes to two Kafka sinks different data - events and CRUD operations. I'm getting an error of found duplicated "transactionalIdPrefix for multiple Kafka sinks: null" on running, BUT - I'm not using any transactions (enable idempotence is false, there is no transaction config and the delivery guarantee is none). It seems like it happend because of the WriteableBackChannel that is used for communication between the commiter and the writer, but it relies on the transactionalIdPrefix even if it's not configured. The even bigger problem is, that even when I'm trying to configure this property, it looks like the same error is occuring, just without the null but with one of the transactionalIdPrefix that I set for the sinks (they are different... 🥴) Can anyone help?
Update: I'm using the two sinks twice. In some point my data stream splits into two event types, and both of the event types are sent to the event's topic, and both of them are also sent to the CRUD operations topic (after modification & filtering etc) It seems that this problem occured only in cases when I reused multiple different sinks. For example, let's name the sinks A and B. I can: • Use sink A once and sink B once • Use sink A twice and sink B once • Use sink A once and sink B twice I cannot: • Use sink A twice and sink B twice I solved this problem by union-ing the CRUD data streams into one data stream, and by that I'm using the CRUD sink only once. It DOES seem like a bug and not like an intentional behavior...
l
So the problem happen in this case: data stream ---> stream 1 -> write to Sink A , write to sink B | |-----------------> stream 2 --> Write to sink A, write to sink B Am I correct?
e
Yes 🥲
Could I get any help here? It seems that the problems also occur in simpler cases, when a task failed for example