I have one question that doesn't seem like it shou...
# troubleshooting
g
I have one question that doesn't seem like it should happen, but still. We have apache pinot in prod with version 1.3, 21 servers (64RAM, 8CPU, "-XX:ActiveProcessorCount=8 -Xms24G -Xmx24G") deployed in k8s using yamls, not helm. When started consumption of a kafka topic with 21 partitions and replication factor of 3 without upsert or minion tasks, i saw a steady RAM at 16GB for quite a while (only ingesting without queries, at a constant 300-400msg/s), up until 20-30m rows it was stable, but with ingesting more data(currently at 520m rows) and ram continued to grow. We have to ingest 8billion more rows (historical data) and we need to know that the server is not going to just die. We have integrated deepstore as well and see the segments there if it is relevant. Based on server metrics it seems that MMap buffer is just keeps growing, is it expected?
i think part of the problem may be the fact that we have a split commit failing 2025/06/04 105644.349 INFO [LLCSegmentCompletionHandlers] [grizzly-http-server-7] Response to segmentConsumed for segment: purchases__13__13__20250603T1152Z is: {"status":"FAILED","streamPartitionMsgOffset":null,"buildTimeSec":-1,"isSplitCommitType":true} 2025/06/04 105644.349 INFO [ControllerResponseFilter] [grizzly-http-server-7] Handled request from 10.52.0.3 GET http://pinot-controller-1.pinot-controller.pinot.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=9057670&instance=Server_pinot-server-5.pinot-server.pinot.svc.cluster.local_8098&name=purchases__13__13__20250603T1152Z&rowCount=39923&memoryUsedBytes=83171438, content-type null status code 200 OK 2025/06/04 105644.350 ERROR [SegmentCompletionManager] [grizzly-http-server-15] Caught exception in segmentConsumed for segment purchases__13__13__20250603T1152Z java.lang.IllegalStateException: Failed to find segment ZK metadata for segment: purchases__13__13__20250603T1152Z of table: purchases_REALTIME at org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:836) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.getSegmentZKMetadata(PinotLLCRealtimeSegmentManager.java:411) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.createFsm(SegmentCompletionManager.java:128) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.lambda$lookupOrCreateFsm$0(SegmentCompletionManager.java:122) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?] at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.lookupOrCreateFsm(SegmentCompletionManager.java:122) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.segmentConsumed(SegmentCompletionManager.java:175) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlers.segmentConsumed(LLCSegmentCompletionHandlers.java:136) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?] at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
the segment is missing inside deepstore
but i can't see the logs from early morning (during the night) that would say why it failed
my guess is that deep store upload to google cloud storage failed and now the split commit is stuck in retrying
m
I think these may be different issues: 1. RAM usage growing - is it heap or MMAP usage? Latter won't cause the server to crash. If heap, take jstack/jfr to see what is being held in heap. 2. Check server/controller logs on which specific part of the split commit is failing.
g
1. it is MMAP 2. the controller was failing to find the segment inside deepstore and thus the failing response given to the server
once i stopped consuming, restarted the whole cluster with scale down and scale up, and started consuming servers are at stable 16gb of RAM
m
Hmm, did you fix controller config last time to be able to talk to GCS? May be the restart fixed it by picking up that config this time.
g
after each config change i did last time i did total scale down and scale up
m
I see, then I can't think of a reason why simple restart would resolve the two issues you saw, with the info I have. But if it is resolved already, that's good
g
if the MMAP won't cause the server to crash and server is stable at 16GB of ram would that mean that in theory i can descale the servers from 64GB RAM to 24-32GB ? and still run queries fine? As far as i know about MMAP it is virtual memory for faster access to the physical files, this would just mean that if i don't have enough memory to space queries will take a bit more time correct?
m
You can, but then you will trade off performance, because now the sever will do a lot more IO to page in/out MMAP’d pages
g
yes, but also there is a cost associated with it
i'm just using this as information, to know if i need to support something not that demanding (up to 1sec responses) that i can descale and save on money
m
Yep, cost vs performance tradeoff