Hey all, We have 3 replicas per segment. If one s...
# troubleshooting
p
Hey all, We have 3 replicas per segment. If one server restarts and one replica goes offline, we get error
Copy code
pinotdb.exceptions.DatabaseError: {'errorCode': 235,
 'message': 'ServerSegmentMissing:\n'
            'Found 1 unavailable segment for table table_name: '
            '[table_name__0__2__20250307T0805Z]'}
We use multistage engine and we see this happening frequently. This causes downtime for our tables if one query consumes a lot of memory leading to server restart. Is this a bug in Pinot? We are currently considering setting the replica to 1 instead of 3 as it is not providing us high availability currently as it is supposed to.
x
When you restart server did you give enough time for graceful shutdown?
If not then broker might not update the routing table immediately hence the glitch.
In 1.3 there is a quota feature you might want to try out to rate limit and protect the server from OOM issue
p
Kubernetes kills the pod when it reaches the memory limit automatically and restart the server pod. We do have 100 GB server pods but we still face restarts During helm updates, our server typically takes around 8 mins to restart. But we have sufficient values for kubernetes probs which can wait upto 20 mins for a server to restart. We have just upgraded to 1.3 in our staging environment and will roll out to prod soon. But as far as I can see, we only have query quotas (maxQueriesPerSecond). We don‘t have memory quota afaik. For us, the problem is some developers run a query over a large table without proper query filters and these queries scan entire tables consuming a large amount of server memory thereby causing a server restart. And we have no way to restrict them.
For server pods, our memory requests and limit is set to 100 GB. Here are our JAVA_OPTS
Copy code
JAVA_OPTS:                    -XX:+ExitOnOutOfMemoryError -Xms50G -Xmx50G -Djute.maxbuffer=100000000 -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-server.log  -javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Dlog4j2.configurationFile=/opt/pinot/etc/conf/pinot-server-log4j2.xml -Dplugins.dir=/opt/pinot/plugins
Now that I think of it, maybe we should try removing -XX:+ExitOnOutOfMemoryError ? 🤔
x
Did you set memory limit for the server statefulset?
This exitOnOutOfMemoryError flag tries to proactively restart servers. We used to see the server might be hanging and not recover.
p
Yes we have set both memory request and limit to 100gb for our server statefulset. We have also used 50gb (half of 100gb) for -Xms and -Xmx after reading documentation and some messages here. Do you suggest removing the memory limit on server pods? Or removing this exitOnOutOfMemoryError flag? Or both? Thanks for your reply!
x
suggest to remove memory limit, so that k8s won’t proactively kill the server pod.
one issue we saw from the k8s is that it counts offheap memory which causes server killed
p
Thanks we we try this on our staging environment.
We often see pods immediately capturing a large portion of memory on start Most of our applications are python based where memory consumed is the actual memory being used by the python app. But in Java pods, it seems k8s is calculating memory based on what is configured in JVM and not what is actually being used. I will study more about JVM memory allocation in k8s pods when I get time.
x
yes, that’s the jvm allocation, xms and xmx