Hi Team, We have a 15TB realtime table with upsert...
# troubleshooting
v
Hi Team, We have a 15TB realtime table with upserts enabled, running on 32 Pinot servers. Each server has 10 disks (total 35TB storage) and 1TB RAM. The table uses native upserts. Whenever we make schema changes (e.g., adding a column), Pinot requires a table reload. During this reload, we see a massive spike in heap usage — the servers fail to start with the usual 120GB heap, and we’re forced to temporarily increase it to 600GB. Once the segments load, heap usage drops back down. Has anyone else encountered this? Any suggestions for managing reloads more sustainably?
k
Hi @Vipin Rohilla, We’re also experiencing a similar issue, though on a much smaller scale. We currently have less than 100 GB of data, but whenever we restart a node, it requires a significant amount of RAM and CPU to re-index all segments. After startup, however, most of the resources go unused. We're running Pinot on Google Cloud instances with 32 GB of RAM—mainly just to handle startup—while Pinot operates smoothly using less than 8 GB under normal load. If anyone finds a solution to this, it would be very helpful for us as well. I'm happy to share our configurations or provide more details if someone can point to a potential issue. Note: We are using Google Cloud Storage as Deep Store.
j
Are you folks already using preload and snapshots for upserts --> https://docs.pinot.apache.org/manage-data/data-import/upsert-and-dedup/upsert#enable-validdocids-snapshots-for-upsert-metadata-recovery if not that could help.
🙌 1
k
@Jayesh Asrani Thanks I will give it a try when I get a chance. It seems promising.