Hi Folks, I'm currently working on a Flink-to-Iceb...
# random
t
Hi Folks, I'm currently working on a Flink-to-Iceberg pipeline, and I'm facing a challenge related to file generation while writing data into a table. To address this, I've decreased the parallelism of my Iceberg sink to 10% of the overall job parallelism. I'm aware that this approach involves shuffling data across the network before sending it to the sink task, potentially causing backpressure on the operator tasks prior to reaching the sink. My question is whether it's feasible to optimize this situation by adjusting the network buffer configurations to maintain a more balanced backpressure.