I have a flink job with 2 kafka sources, is there ...
# troubleshooting
f
I have a flink job with 2 kafka sources, is there a way I can ensure one source is consumed after the other? Currently, I'm having issues in tests, where I cannot ensure it, but my logic requires it.
d
I am not aware of Flink having inbuilt support for this scenario. However its possible that you could emit a start processing/switch event. Processing from first source would begin after start event. Once all data is processed it would then fire a switch event.
You could implement a Process function that buffers or holds back records events sent from the second source until it receives the Switch event which was only emitted after the the last message from the first source.
f
That’s similar to what I did, but I have issues in the integration tests where I run the whole flink job using mock source and sink. In that scenario, there’s no way of enforcing the order of the message consumption. I have a CoProcessFunction, which receives the messages from Source1, buffers them in a state. When a message from Source2 comes, it does the business logic and outputs the message to the collector. In the tests, sometimes Source2 is processed first. I could implement flags to check if both streams were processed and only then collect to the output, but I wanted to avoid it.