Hey Team, We have use-case where we ingest upto 8...
# troubleshooting
r
Hey Team, We have use-case where we ingest upto 800k rps. We are observing backlog during ingestion, we have 24 partitions, increasing them to 36 did not help. We also increased controller heap from 32GB to 48GB but it also did not helped. I observed high GC pause times up to 1 sec. We are already on JDK 21 with ZGC. What is your recommendation for this? Is there any way to split traffic between multiple controllers?
m
Controller is not in the critical path for ingestion. What's the sizing for servers in terms on number of cores, memory, jvm args?
r
We have 24 servers each with 50 GB RAM, 32 GB heap, 12 core CPU. 24 partitions for ingestion. 2 replica groups each with 12 servers and deep store S3. jvm args
Copy code
jvmOpts: -Djute.maxbuffer=12000000 -javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml
    -Xms24G -Xmx32G -XX:+UseZGC -XX:+ParallelRefProcEnabled -XX:+DisableExplicitGC
    -Xlog:gc*:file=/opt/pinot/gc-pinot-server.log
Current avg CPU: 200% Current avg heap consumption: 15-20 GB with peaks upto 25-30GB
m
Hmm, then increasing partitions should have helped, because one partition is consumed by 1 thread only (hence the low cpu). You should go to 64 partitions
j
even if this is evenly distributed, 32 partitions is looking at ~22k events/second/partition. 20k is actually where we’ve seen bottlenecks as well where Pinot cannot keep up in realtime. +1 to doubling partitions again
x
Also suggest to use DOWNLOAD mode if you have multiple replicas and one segment build time is longer than 10 secs. It will save cpu and improve the ingestion latency. https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime#controlling-segment-build-vs.-segment-download-on-real-time-servers
r
I have enabled the download and increased partitions to 60. The backlog issue is resolved but pinot server heap has increased now. It is consistently between 25-30GB. I am planning to reduce segment size from 500mb to 300mb for this. Is there any other recommendations for this? Can increasing server heap help here?