Hello there, I'm writing here to check if anybody has ever faced this issue.
I'm trying to optimise a batch job to be able to read from a OBT (iceberg) and write to ~3000 single tables (iceberg) applying a flattening operation on a cluster with 20 nodes 1 master (32 vCPUs, 128 GB memory each). 1 table works fine 100 works fine 1000 I can hardly check from the job manager if anything is running 3000 I don't have any way to observe what's happening.
Can someone help me to identify the bottleneck and scale this properly, it's not about throughput but scheduling / planning it seems? Feel free to ask for more details to narrow down the issue. Thanks a lot π