Idlan Amran
07/01/2025, 8:46 AMprimaryKeyColumns, based on my experience on upsert table, primary key metadata TTL was stored on heap and as number of records and primary keys grows, memory/RAM usage will increase too. do i need to expect this kind of situation too on dimension table?
and does dimension table can be a realtime table rather than offline so i can push the data through kafka ?
our app architecture is kinda complex right now. we need a table to stores product activity logs, kind of product tracking such as example stock increase, price increment and etc. and in some cases there are ingestion that was duplicated like in the same day it will be pushed more than 1 time to kafka, causing duplicate. by right we do not need full product data, we just need the changes like the example i shared and what are the id of the changes so we can check historically for this particular product what was changed.
i tested using upsert since its the most near to my use case but the memory usage was very huge and our pinot ec2 server was going downtime from time to time bcs of upsert table due to out of memory error.
i really appreciate if any of you guys can share any config that you guys work on / whatever i can do to tune my config / improve our ingestion to pinotBolek Ziobrowski
07/01/2025, 1:02 PMBolek Ziobrowski
07/01/2025, 1:23 PM