Hi everyone,
We are using Flink to write with fanout DynamicIcebergSink into about 300 Iceberg tables and we need to run TableMaintenance (expireSnapshots, vacuum, etc.) on all of them.
Running a separate maintenance job for each table would consume a huge number of taskSlots.
Is there any best practice, tool, or pattern to efficiently manage maintenance for a large number of tables, without exhausting taskSlots with hundreds of concurrent jobs?
Has anyone solved this problem or implemented scalable TableMaintenance orchestration across many tables?
Thanks a lot!