Hello there, I'm writing here to check if anybody ...
# troubleshooting
t
Hello there, I'm writing here to check if anybody has ever faced this issue. I'm trying to optimise a batch job to be able to read from a OBT (iceberg) and write to ~3000 single tables (iceberg) applying a flattening operation on a cluster with 20 nodes 1 master (32 vCPUs, 128 GB memory each). 1 table works fine 100 works fine 1000 I can hardly check from the job manager if anything is running 3000 I don't have any way to observe what's happening. Can someone help me to identify the bottleneck and scale this properly, it's not about throughput but scheduling / planning it seems? Feel free to ask for more details to narrow down the issue. Thanks a lot πŸ™‚