Hi everyone:wave:. I am looking for ways to reduce...
# random
l
Hi everyone👋. I am looking for ways to reduce the impact of backpressure events on my undergraduate thesis. But since I've never worked with Flink, I don't know of a real scenario that generates back pressure. Do you have any examples of this? Maybe some material to guide my studies.
d
A few examples: • Throttling from downstream storage, such as Amazon Kinesis Data Streams • High network latency at sink/enrich • Latency from Disk IO, for example RocksDB • Inefficient code burning CPU • Bugs in code leading to deadlocks
d
Example of CPU intensive task: https://github.com/rmetzger/flink-reactive-mode-k8s-demo/blob/main/reactive-mode-demo-jobs/src/main/java/org/apache/flink/DemoJob.java Example of RocksDB task: https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-rocksdb-s[…]flink/streaming/tests/RocksDBStateMemoryControlTestProgram.java You can configure the stress on the CPU with the "iteration" parameters as well as the stress on RocksDB with the "sequence_generator_source.keyspace" and "sequence_generator_source.payload_size" parameters. It is fairly easy to generate backpressure with those tools but they are not quite representative of real workload.