Hi Team,
We have a 15TB realtime table with upserts enabled, running on 32 Pinot servers. Each server has 10 disks (total 35TB storage) and 1TB RAM. The table uses native upserts. Whenever we make schema changes (e.g., adding a column), Pinot requires a table reload. During this reload, we see a massive spike in heap usage — the servers fail to start with the usual 120GB heap, and we’re forced to temporarily increase it to 600GB. Once the segments load, heap usage drops back down.
Has anyone else encountered this? Any suggestions for managing reloads more sustainably?