I'm experimenting and trying to load in data into ...
# troubleshooting
r
I'm experimenting and trying to load in data into a realtime table that is very "tall", with only about 10 columns but hundreds of millions of rows. I'm seeing server restarts:
Copy code
pod/pinot-server-0                           1/1     Running     0               43h
pod/pinot-server-1                           1/1     Running     0               43h
pod/pinot-server-2                           1/1     Running     1 (26s ago)     43h
pod/pinot-server-3                           1/1     Running     0               43h
pod/pinot-server-4                           1/1     Running     0               43h
pod/pinot-server-5                           0/1     OOMKilled   3 (3m56s ago)   43h
pod/pinot-server-6                           1/1     Running     1 (8m50s ago)   81m
pod/pinot-server-7                           1/1     Running     1 (7m44s ago)   81m
The table data itself is not that large, pretty small in fact (at 10GB currently), there are 8 servers with O(60GB) memory each, 100GB PVCs, and a total of maybe 450 segments over 4 tables. But this table is (IIUC) trying to do some pretty high cardinality upserts, easily into the many tens of millions of keys. Could this be the cause of the OOMs? Are there specific settings I can review or actions I can take while I'm learning besides larger instances?
I guess I should say because this may be a
cgroup
kill not JVM the pods have 60G requests/limits and
Xmx
is 58G; they're scheduled on dedicated
r5.2xlarge
instances with 64G which claim to have < 1G left.
m
I think you almost all of the available memory to heap, and there's nothing left for MMAP and direct memory. What you really want is to have heap not more than 32 GB, so the rest can be available for direct memory and mmap. Also 450 segments for 10GB seems too many, you may want to increase segment size.
r
That sounds like an easy update thanks; I can turn it down. 450 segments is for all tables probably... above 400GB. But still may not be good, not sure. This is the attempt to size segments in the table configs:
Copy code
"realtime.segment.flush.threshold.segment.size": "200M",
      "realtime.segment.flush.threshold.time": "24h",
      "realtime.segment.flush.threshold.rows": "0"
m
200M seems fine
r
Seems like the controller is now having trouble reaching the servers, lots of logs like
Copy code
2025/06/14 20:59:32.221 WARN [MultiHttpRequest] [async-task-thread-5] Caught 'java.net.SocketTimeoutException: Read timed out' while executing: GET on URL: http://pinot-server-4.pinot-server-headless.pinot.svc.cluster.local:8097/table/silver_images_REALTIME/size
I see JVM OOMs there, maybe underprovisioned now
Copy code
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ParticipantHealthReportTimerTask"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpServer-0"
Ok they're back up after a restart
The servers stay up in the cluster now but they cycle in and out of alive/dead status. Maybe they're not staying in touch with zookeeper?
Copy code
2025/06/15 13:55:20.173 WARN [ClientCnxn] [Start a Pinot [SERVER]-SendThread(pinot-zookeeper:2181)] Client session timed out, have not heard from server in 40842ms for session id 0x3000028f141007d
2025/06/15 13:55:20.173 WARN [ClientCnxn] [Start a Pinot [SERVER]-SendThread(pinot-zookeeper:2181)] Session 0x3000028f141007d for server pinot-zookeeper/172.20.101.168:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException or SessionTimeoutException.
org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 40842ms for session id 0x3000028f141007d
There are 3 ZK pods running in replicated mode with 4CPU, 14G mem, 20G disk. Pretty much any query reports unavailable segments. If the REST API is returning accurate results, there still are not that many segments, and much less data (< 200GB) than server space (750GB mem, 3TB disk)?
I think I see related (?) broker logs about unavailable instances and being unable to find segments
Copy code
2025/06/15 14:18:23.312 WARN [StrictReplicaGroupInstanceSelector] [ClusterChangeHandlingThread] Found unavailable instance: Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098 in instance group: [Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098, Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098] for segment: ****__4__11__20250615T0012Z, table: ****_REALTIME (IS: {Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098=ONLINE, Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098=ONLINE}, EV: null)
m
ZK disconnects are typically due to STW GC, which is typically due to resource constraints or incorrect configs. How many PKs do you have per server, what’s the heap usage?
r
Ok, SG. "Tons". I'm playing with a use case where I "expect" (wonder if?) this to get stressed out. A rough guess is at least 20B total records, 13B (68%) unique primary keys over 4 fields in the schema. This is like a big backfill so I can learn more, in practice it might be more like 10s of millions with updates at any given time but for a table with B's of records. The existing cluster is "stalled" at about 250M records. In retrospect actually there's redundancy in a couple of the PK fields I could remove, not sure if that matters. Honestly I'm not sure how to check directly with the running cluster. I can look around for that. Kubernetes metrics sure suggest the servers are using all their heap space.
m
Are you using multiple columns for PK, are they Strings? That will need a lot of heap. I suspect all heap is being used by Upsert metadata leading to memory pressure and GC
r
Yes and yes, basically IDs. I'm looking at the specs now and I can trim them all down, but this is what I think we're interested in. This is server heap that's needed I assume? If so I can think about how this would actually be planned around instead of YOLO'd.
m
Server heap is needed in Open Source implementation. There's a closed source implementation that removes this limitation