<https://docs.pinot.apache.org/manage-data/data-im...
# troubleshooting
i
https://docs.pinot.apache.org/manage-data/data-import/batch-ingestion/dim-table im looking into Dimensions table currently and noticed that it requires
primaryKeyColumns
, based on my experience on upsert table, primary key metadata TTL was stored on heap and as number of records and primary keys grows, memory/RAM usage will increase too. do i need to expect this kind of situation too on dimension table? and does dimension table can be a realtime table rather than offline so i can push the data through kafka ? our app architecture is kinda complex right now. we need a table to stores product activity logs, kind of product tracking such as example
stock increase
,
price increment
and etc. and in some cases there are ingestion that was duplicated like in the same day it will be pushed more than 1 time to kafka, causing duplicate. by right we do not need full product data, we just need the changes like the example i shared and what are the id of the changes so we can check historically for this particular product what was changed. i tested using upsert since its the most near to my use case but the memory usage was very huge and our pinot ec2 server was going downtime from time to time bcs of upsert table due to out of memory error. i really appreciate if any of you guys can share any config that you guys work on / whatever i can do to tune my config / improve our ingestion to pinot
b
Hello. Only offline table can be a dimension table ("Dimension tables are a special kind of offline tables ..." ). If : • "disablePreload": true - pinot keeps primary key values and segment id, doc id in ram • "disablePreload": false - pinot keeps all row values in ram Dimension tables shouldn't be big because they're stored in memory on all servers and actual in-memory size can be many times bigger than (compressed) segment file .
Some improvements are coming in 1.4.0 , e.g. https://github.com/apache/pinot/pull/15130 .