what is the recommended segment size for an offline table with batch ingestion? is it still 300-500 MB or do we increase it to 1 GB? Also for realtime tables we configure offheap memory in servers. is that also needed for batch ingestion?

1GB should be fine. What are your configuring specifically?

I am ingesting several 100s of GB of data hourly into the pinot cluster via spark. I have mostly configured cluster for realtime use case and doing a PoC for offline tables. I have • zookeeper: 3 pods each with 2 cpu and 4G memory • controller: 3 pods each with 1 cpu and 2G memory • broker: 3 pods each with 1 cpu and 2G memory • server: 6 pods each with 3 cpu and 24G memory. tables are configured with replication 2, and server java ops as

-Xms8G -Xmx8G -XX:MaxDirectMemorySize=14G ...

You don't need to explicitly set it for realtime or offline, it defaults to xmx.

does that mean i can remove direct memory jvm option, and reduce server pod memory? i am also setting this server config

pinot.server.instance.realtime.alloc.offheap=true

. do i remove it as well?

I think that already defaults to true.

Any recommendations given that the query volume is super low and usage is basically to power some dashboards

recommendation on what?

for server java settings.

There's a couple of capacity planning blogs you can google

great, thank you!