Hi, I have a question regarding the design of a pi...
# random
d
Hi, I have a question regarding the design of a pipeline that involves an enrichment step using AsyncIO (likely to be backpressured) and a buffering operation (like a window for reordering/joining/merging/deduplicating events coming from multiple streams). Details are in the thread. Thanks!
In the past we faced issues when we tried to design a pipeline with a layout like: buffering operator -> enrichment operator. Problem was that the buffering operator disconnects the flow of operations and does not let backpressure to do its job and let many events enter the buffer even if they have no chances of getting processed in a timely manner by the rest of the pipeline. Ultimately this is leading to problems like FLINK-20217 causing checkpoints to constantly grow and always timeout. What we did in the past is to avoid buffering events as much as we can to reconnect the flow between operators to let backpressure propagate and slow down the sources. I'm now in a situation where I can't avoid buffering events and I'm struggling to find a good solution, the ideas I have in mind are: • reverse the order of operators and do the slow operation first: AsyncIO -> window, main drawback is that the AsyncIO cannot benefit from the deduplication done by the buffering operation and the buffering operation have to buffer larger events because they are now enriched. • have two jobs and use kafka as a buffer: flink_job1(reorder/join/merge/dedup) -> kafka -> flink_job2(AsyncIO). Main drawback is that 2 jobs are required. • flow control with a throttling operator in front of the buffering operator: source throttling -> buffering -> AsyncIO. Not tested but I anticipate some tedious tuning of the throttling rules based on the performance of the AsyncIO operator • flow control with a custom process function for the buffering operation with some throttling using some operator state that keeps track a metric reflecting the size of the buffer to mimic backpressure. Not tested but same as above might be tedious to tune and other difficulties because throttling in the process function might also block timers from being fired in this same operator. I'm curious to know if you faced similar design issues and how you approached them?