Hello, I have an aggregator job that experiences b...
# troubleshooting
r
Hello, I have an aggregator job that experiences backpressure after running for a while and completely stops processing. It doesn't take any further input from the source. Here's a bit of context: - There are 3 producer jobs, all of which write data to a common Kafka topic - The aggregator job reads in from that topic, aggregates the data once it finds 3 matching records for the same ID and emits a single record into another Kafka topic - The logic is quite simple: the aggregator stores the data in its state for each ID until all 3 matches arrive and then the state is cleared for that ID - 2 of the producer jobs emit records at an almost constant pace, while the 3rd producer emits data in chunks of variable length. The emission of data is also not periodic for this 3rd producer; meaning it can hold up to 15 seconds worth of data in one instance and 2 minutes of data in another for example As far as I understand, the backpressure can cause the processing to slow down. But how is it possible that the processing stops completely? Can anyone please provide me some helpful explanations here?