Ross Morrow
06/14/2025, 7:59 PMpod/pinot-server-0 1/1 Running 0 43h
pod/pinot-server-1 1/1 Running 0 43h
pod/pinot-server-2 1/1 Running 1 (26s ago) 43h
pod/pinot-server-3 1/1 Running 0 43h
pod/pinot-server-4 1/1 Running 0 43h
pod/pinot-server-5 0/1 OOMKilled 3 (3m56s ago) 43h
pod/pinot-server-6 1/1 Running 1 (8m50s ago) 81m
pod/pinot-server-7 1/1 Running 1 (7m44s ago) 81m
The table data itself is not that large, pretty small in fact (at 10GB currently), there are 8 servers with O(60GB) memory each, 100GB PVCs, and a total of maybe 450 segments over 4 tables.
But this table is (IIUC) trying to do some pretty high cardinality upserts, easily into the many tens of millions of keys. Could this be the cause of the OOMs? Are there specific settings I can review or actions I can take while I'm learning besides larger instances?Ross Morrow
06/14/2025, 8:06 PMcgroup kill not JVM the pods have 60G requests/limits and Xmx is 58G; they're scheduled on dedicated r5.2xlarge instances with 64G which claim to have < 1G left.Mayank
Ross Morrow
06/14/2025, 8:19 PM"realtime.segment.flush.threshold.segment.size": "200M",
"realtime.segment.flush.threshold.time": "24h",
"realtime.segment.flush.threshold.rows": "0"Mayank
Ross Morrow
06/14/2025, 9:01 PM2025/06/14 20:59:32.221 WARN [MultiHttpRequest] [async-task-thread-5] Caught 'java.net.SocketTimeoutException: Read timed out' while executing: GET on URL: http://pinot-server-4.pinot-server-headless.pinot.svc.cluster.local:8097/table/silver_images_REALTIME/sizeRoss Morrow
06/14/2025, 9:02 PMException: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ParticipantHealthReportTimerTask"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpServer-0"Ross Morrow
06/14/2025, 9:08 PMRoss Morrow
06/15/2025, 2:10 PM2025/06/15 13:55:20.173 WARN [ClientCnxn] [Start a Pinot [SERVER]-SendThread(pinot-zookeeper:2181)] Client session timed out, have not heard from server in 40842ms for session id 0x3000028f141007d
2025/06/15 13:55:20.173 WARN [ClientCnxn] [Start a Pinot [SERVER]-SendThread(pinot-zookeeper:2181)] Session 0x3000028f141007d for server pinot-zookeeper/172.20.101.168:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException or SessionTimeoutException.
org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 40842ms for session id 0x3000028f141007d
There are 3 ZK pods running in replicated mode with 4CPU, 14G mem, 20G disk.
Pretty much any query reports unavailable segments. If the REST API is returning accurate results, there still are not that many segments, and much less data (< 200GB) than server space (750GB mem, 3TB disk)?Ross Morrow
06/15/2025, 2:21 PM2025/06/15 14:18:23.312 WARN [StrictReplicaGroupInstanceSelector] [ClusterChangeHandlingThread] Found unavailable instance: Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098 in instance group: [Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098, Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098] for segment: ****__4__11__20250615T0012Z, table: ****_REALTIME (IS: {Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098=ONLINE, Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098=ONLINE}, EV: null)Mayank
Ross Morrow
06/15/2025, 3:23 PMMayank
Ross Morrow
06/15/2025, 3:32 PMMayank