Hey everyone, maybe someone can help me with this ...
# troubleshooting
l
Hey everyone, maybe someone can help me with this hopefully easy question. Is there a way to guarantee the ordering of operators that work on the same stream? What I mean by is that let's imagine we have operators in our job that look something like this:
Copy code
SingleOutputStreamOperator<Event> processedEvents = someStream.process(processFunction);
processedEvents.sinkTo(firstSink);
processedEvents.sinkTo(secondSink);
Now while in the code it looks like the processed events are first sinked to
firstSink
and then to
secondSink
, then I understand there is actually no ordering guarantee here. both of the sinks operate on the same
processedEvents
stream then they can operate in parallel. A message can appear in
secondSink
first and then later in
firstSink
. In case of failures it's also possible that the event reaches
secondSink
but never reaches the
firstSink
etc. My question is: is there a way to force ordering here so that events are always sinked to
firstSink
first and only after the event is successfully sinked to the
firstSink
it would be sinked to
secondSink
.
ChatGPT is recommending a solution like this:
Copy code
processedEvents.sinkTo(firstSink);

SingleOutputStreamOperator<Event> confirmedEvents = processedEvents.map(event -> {
    // Ensures firstSink is completed before proceeding
    return event;
});

confirmedEvents.sinkTo(secondSink);
Basically the map after the first sink ensures that both the first sink and map need to complete before we can move on to the second sink. Does this work?
Idk, to me it would still seem that since both firstSink and map operate on the same data stream then map wouldn't care if firstSink is finished or not
p
I don't think this would be possible. The only way to ensure that would be having a way to callback from the first sink to the second. Perhaps a multi-layer sink may do your trick there. I'm interested in understanding this usecase of "cascade ordered sink" if you're up to discuss it 😄
l
Basically I have a process function that sends a request to S3 to determine how the event should be processed. The result is then sinked to Kafka (first sink) and also to that same S3 bucket (second sink). Now since the output that we write back to S3 changes how the event should be processed, then if the same event was processed for the second time we would get a different result, which is expected and what we want to achieve. The unfortunate thing is that in erroneous cases where the Kafka write fails and the S3 one succeeds then Flink will try to process the event again to achieve exactly once processing guarantee, and during the second processing we would get a different result (which again, is expected), however the first result will forever be missing from Kafka side.
The exact use case is pretty complicated and we've somewhat dug ourselves in a hole with some bad design decisions early on. We're redesigning the solution but we still need to keep what we have working 😄
p
Fair enough. We all have been there. I'd solve this problem like 1. Fisrt pipeline sink to Kafka only 2. Second souce/pipeline reads from the previous sink and write it to S3 You'd be "outboxing" it. Double writes are always a issue indeed